Memory bandwidth vs number of GB

BrainPop · August 19, 2011, 09:29

Hi everyone, thanks for a great forum. I've looked through the hardware threads and thought it pertinent to post this question as 8GB non ECC RAM DIMMS are becoming more affordable and prevalent which poses the question:

Is it better to go with 16GB of say DDR3 2000Mhz gaming ram (4 x 4Gb) , or 32Gb of 1333Mhz RAM (4 x 8Gb)?

It probably depends on what you are doing, so here is my situation:
I am running a series of simulations with about 4 to 7m unstructured/structured hexahedral cells for each model using Fluent - parallel across 4 cores. I have built a new PC (i7 2500k, SSD, 32GB capable motherboard etc.) and also have another 'work' PC that is an off the shelf i7 2.8Ghz HP with 16Gb 1333 valueRam. Both PCs are 64bit Win7.

The reason for the new PC is I need to meet a deadline in a few months and are being held up mainly by mesh generation, which is taking anywhere from 2 to 7 days per model at the moment on the i7. I wish I had Gambit to work with, but instead are using the ANSYS meshing program (I don't have any experience with Gambit - so its hard to know if ANSYS mesh is really limited for CFD meshing and inefficient in using comp resources).
My understanding is that ANSYS meshing doesn't utilise all the cores even when you set multiple CPUs in the options menu.

My plan is to mesh and refine on the new PC while running sims on the other, then eventually run sims on both, followed by DPM calculations (Runge Kutta/Implicit). I don't plan to refine past more than about 7m cells. Checking the resource monitor in Win7 while generating the mesh shows it is almost bouncing off 15.5Gb. I have had some instances of 'running out of usable memory' (meshing error message) but I think this has more to do with how clean the CAD is.

Is the there a relationship between amount of memory and mesh generation? eg. 2Gb : 1m cells - tet or hex?

Also, is the mesh gen. speed and DPM particle tracking calcs more likely throttled by memory bandwidth or the amount of available memory?

Any advice offered would be greatly appreciated, and I hope this thread will be of use to those in the market for a new CFD machine.

kyle · August 20, 2011, 16:45

I went through this a few weeks ago. I wasn't all that scientific about my tests on memory for meshing, but there is definitely some improvement as you scale up the memory frequency. As for solving on unstructured meshes, memory bandwidth is very clearly the bottleneck on modern CPUs. Going from 1333mhz to 2000mhz you might see as much as a 20% speedup.

Are you meshing a transient case with a completely new mesh for each timestep? Otherwise, I have no idea how it can take multiple days to create at 7 million cell mesh. That should take two hours on decent hardware, maximum.

If you aren't doing some exotic meshing, then your insanely long meshing times are most likely due to not having enough RAM. I am not familiar with the Windows resource monitor, but 15.5gb might just be the actual physical memory used by CFX, and it could also using a boatload of virtual memory. Virtual memory is extremely slow, and would explain why your meshes are taking so long. If this is the case, going to 32gb would obviously be the way to go.

BrainPop · August 22, 2011, 02:40

Thanks for the pointers kyle. I checked the memory tab and sure enough it was thundering away on the HD using virtual memory which is definitely the reason for the long times. My mesh isn't time stepped/transient or anything like that. It threw me as the HD access LED must have stopped working or blown and wasn't showing up. So a good lesson is to always check the resource monitor. Another thing I found was that it was only dipping into the virtual memory after about an hour into the mesh gen once it had created the unstructured hex bodies.

I have opted to go with the faster memory (2133 Mhz) as it is much cheaper than 32GB and try and be more conservative with the number of cells to fit it to the 16GB. Another important thing to consider after checking out what overclocking gamers tend to post about is that running 8GB on a single DIMM can lead to bottle necks in a desktop motherboard's architecture as opposed to using a proper server mobo which is specifically designed to handle more than 16GB. So while my motherboard is specified as being able to handle 32GB the system would probably be fairly slow from the get go.

August 19, 2011, 09:29	Memory bandwidth vs number of GB	#1
BrainPop New Member Steve Join Date: Aug 2011 Posts: 2 Rep Power: 0	Hi everyone, thanks for a great forum. I've looked through the hardware threads and thought it pertinent to post this question as 8GB non ECC RAM DIMMS are becoming more affordable and prevalent which poses the question: Is it better to go with 16GB of say DDR3 2000Mhz gaming ram (4 x 4Gb) , or 32Gb of 1333Mhz RAM (4 x 8Gb)? It probably depends on what you are doing, so here is my situation: I am running a series of simulations with about 4 to 7m unstructured/structured hexahedral cells for each model using Fluent - parallel across 4 cores. I have built a new PC (i7 2500k, SSD, 32GB capable motherboard etc.) and also have another 'work' PC that is an off the shelf i7 2.8Ghz HP with 16Gb 1333 valueRam. Both PCs are 64bit Win7. The reason for the new PC is I need to meet a deadline in a few months and are being held up mainly by mesh generation, which is taking anywhere from 2 to 7 days per model at the moment on the i7. I wish I had Gambit to work with, but instead are using the ANSYS meshing program (I don't have any experience with Gambit - so its hard to know if ANSYS mesh is really limited for CFD meshing and inefficient in using comp resources). My understanding is that ANSYS meshing doesn't utilise all the cores even when you set multiple CPUs in the options menu. My plan is to mesh and refine on the new PC while running sims on the other, then eventually run sims on both, followed by DPM calculations (Runge Kutta/Implicit). I don't plan to refine past more than about 7m cells. Checking the resource monitor in Win7 while generating the mesh shows it is almost bouncing off 15.5Gb. I have had some instances of 'running out of usable memory' (meshing error message) but I think this has more to do with how clean the CAD is. Is the there a relationship between amount of memory and mesh generation? eg. 2Gb : 1m cells - tet or hex? Also, is the mesh gen. speed and DPM particle tracking calcs more likely throttled by memory bandwidth or the amount of available memory? Any advice offered would be greatly appreciated, and I hope this thread will be of use to those in the market for a new CFD machine.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[snappyHexMesh] external flow with snappyHexMesh	chelvistero	OpenFOAM Meshing & Mesh Conversion	11	January 15, 2010 20:43
RPM in Wind Turbine	Pankaj	CFX	9	November 23, 2009 05:05
air bubble is disappear increasing time using vof	xujjun	CFX	9	June 9, 2009 08:59
Smooth Grid Error! Help	seasoul	FLUENT	1	March 24, 2008 11:56
Number of User-Defined Memory Locations to 12.	Michael	FLUENT	1	February 6, 2008 05:48

August 20, 2011, 16:45		#2
kyle Senior Member Join Date: Mar 2009 Location: Austin, TX Posts: 160 Rep Power: 18	I went through this a few weeks ago. I wasn't all that scientific about my tests on memory for meshing, but there is definitely some improvement as you scale up the memory frequency. As for solving on unstructured meshes, memory bandwidth is very clearly the bottleneck on modern CPUs. Going from 1333mhz to 2000mhz you might see as much as a 20% speedup. Are you meshing a transient case with a completely new mesh for each timestep? Otherwise, I have no idea how it can take multiple days to create at 7 million cell mesh. That should take two hours on decent hardware, maximum. If you aren't doing some exotic meshing, then your insanely long meshing times are most likely due to not having enough RAM. I am not familiar with the Windows resource monitor, but 15.5gb might just be the actual physical memory used by CFX, and it could also using a boatload of virtual memory. Virtual memory is extremely slow, and would explain why your meshes are taking so long. If this is the case, going to 32gb would obviously be the way to go.

August 22, 2011, 02:40		#3
BrainPop New Member Steve Join Date: Aug 2011 Posts: 2 Rep Power: 0	Thanks for the pointers kyle. I checked the memory tab and sure enough it was thundering away on the HD using virtual memory which is definitely the reason for the long times. My mesh isn't time stepped/transient or anything like that. It threw me as the HD access LED must have stopped working or blown and wasn't showing up. So a good lesson is to always check the resource monitor. Another thing I found was that it was only dipping into the virtual memory after about an hour into the mesh gen once it had created the unstructured hex bodies. I have opted to go with the faster memory (2133 Mhz) as it is much cheaper than 32GB and try and be more conservative with the number of cells to fit it to the 16GB. Another important thing to consider after checking out what overclocking gamers tend to post about is that running 8GB on a single DIMM can lead to bottle necks in a desktop motherboard's architecture as opposed to using a proper server mobo which is specifically designed to handle more than 16GB. So while my motherboard is specified as being able to handle 32GB the system would probably be fairly slow from the get go.