CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > General Forums > Main CFD Forum

HDD Storage Slow in Parallel Read/Write reconstructPar

Register Blogs Community New Posts Updated Threads Search

Like Tree1Likes
  • 1 Post By mprinkey

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   July 11, 2016, 14:46
Default HDD Storage Slow in Parallel Read/Write reconstructPar
  #1
Member
 
james wilson
Join Date: Aug 2014
Location: Orlando, Fl
Posts: 39
Rep Power: 12
jameswilson620 is on a distinguished road
All,

I searched the forum for something related to my issue but couldn't find much specific to my issue.

So i have a dual boot system and a shared HDD attached to the MOBO that is formatted to NTFS so that I can access data from windows as well as linux.

When i run cases on the HDD, namely parallel but also serial, I notice there is a large hang time when writing larger quantities of data, e.g. reconstructPar. I should say, more hang time than what I would expect, even from large case files with a high cell count.

When executing the reconstructPar command for larger cases 100K-1M or so cells, the utility will hang for anywhere from 10s to minutes before beginning to reconstructing the data. Even the transition from one time step to the next while reconstruction takes a good deal of time. Its so bad that I have restricted the number of parallel processors to use for the simulation because each additional processor drastically increases reconstruction time. When running these larger cases on my SSD, formatted to ext4, I dont see any excessive hang time (other than what would be expected when dealing with a large cell count and number of fields written).

The questions I have are:

1. Has anyone experienced this with an HDD?

2. Does anyone have any suggestion on what file system type I should use for a cross OS compatible storage drive such that linux (ubuntu 14.04) can handle the read/write operations required by openfoam?

3. should I abandon the requirement to have a cross OS compatible file system so that I can maximize performance when running large OF cases on the drive?

I will be purchasing a 2TB drive that has a higher spin speed and read/write speed. I just want to make sure I format everything properly before I make an investment.

Thanks all for the views and any feedback,

James
jameswilson620 is offline   Reply With Quote

Old   July 12, 2016, 10:33
Default
  #2
Senior Member
 
Michael Prinkey
Join Date: Mar 2009
Location: Pittsburgh PA
Posts: 363
Rep Power: 25
mprinkey will become famous soon enough
reconstructPar is pretty I/O intensive. I would try copying the data over from NTFS to a native Linux filesystem and see if performance improves. NTFS support in Linux has been focused on "making it work" versus "making it fast." You will likely have better results with ext3/4 or xfs filesystem serving up the data.

and if you need speed, buy a 128 or 256 GB SSD will make more differences than anything else. Use it as your storage for the current run and archive to the spin disk when you are done.
pgh likes this.
mprinkey is offline   Reply With Quote

Old   July 24, 2017, 13:54
Default
  #3
Senior Member
 
sandy
Join Date: Feb 2016
Location: .
Posts: 117
Rep Power: 10
saddy is on a distinguished road
reconstructPar can be done in parallel as well. it reduces time immensly. i have the utility. in case you guys need it
saddy is offline   Reply With Quote

Reply

Tags
file system type, hdd, openfoam, reconstructpar, slow write speed


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
The udf.h headers are unable to open- in VISUAL STUDIO 13 sanjeetlimbu Fluent UDF and Scheme Programming 4 May 2, 2016 06:38
Explicitly filtered LES saeedi Main CFD Forum 16 October 14, 2015 12:58
simpleFoam parallel AndrewMortimer OpenFOAM Running, Solving & CFD 12 August 7, 2015 19:45
the speed of parallel computation for over 3 million grid is very slow liujmljm SU2 7 June 27, 2014 17:45
Coal combustion, big node, slow calculation, parallel Whitebear CFX 1 April 8, 2014 04:27


All times are GMT -4. The time now is 23:26.