[havref]

First a brief description:
We are looking to expand our computing capabilities by investing in another CFD-specific computer, mainly used for interFoam and interDyMFoam in marine applications. Today we have a dual processor rack with two 16-core Intel E5-2697A V4, which is in more or less constant use. In addition to having an increasing amount of projects lining up, simulations with complex geometries and/or dynamic meshes are often taking too long.

A typical mesh consists of 3-8 million cells, depending on the size and accuracy of the turbulence modelling. Our current memory usage for such a typical simulation with 6 million cells is in the range of 10-15 GB.

Potential workstation number 2
We don't really have a specified budget, but are aiming towards the $ 15k-30k region. A setup we've looked quite a lot at is the following:

Poweredge R940-server
4 x Intel Xeon Gold 6154 3.0GHz
8 x 16GB RDIMM 2666MT/s Dual Rank
1 x 960GB SSD SATA
Price ~ $26k

Questions:
1)
Do you have any comments regarding this setup? I'm thinking of potential bottlenecks or better computing per dollar alternatives.
2)
How much "better" will a 4 x Intel Gold 6154 perform than a dual Intel Gold 6154 setup? Can we expect the 80-90% increase in computing power that we are hoping for?
3)
I've wondered how much memory is really needed for simulations where we are rarely expecting to use more than 20 GB. Is it the memory channels we should pay most attention to? These processors have 6 memory channels. Does that mean we should use multiples of 6 when buying memory? Thus decrease to 6 or increase to 12?
4)
Is there any speed difference between 16x 8GB RAM, 8x 16GB RAM or 4 x 32? I've noticed it does not affect the cost much, so is it purely a reliability and space issue?

Thank you!

[flotus1]

For this setup, you MUST use 24 DIMMs, 6 DIMMs per CPU. If memory channels are not populated, the CPU can not use their additional memory bandwidth. With only 8 or 12 DIMMs you will waste more than half of the performance this system can have for CFD.
With that fixed, the setup should be more than twice as fast compared to your old setup. But of course only if your simulations are not i/o-bound and scale well on such a large amount of cores.
If it does not scale so well, you can settle for a lower core count processor model and save a few bucks. Or if money is not that much of an issue use less cores per CPU but spread evenly among the sockets.

Edit: if you haven't done it yet, you can see how your jobs scale on your old system by running the same simulation on a single core and on an increasing amount of cores, typically doubling the core count each time. Then put the speedup factor in a chart...
When you are unsure if you are spending those 26000$ wisely, this might help you to make the right decision.

And I have to ask: are you sure that your old workstation has its memory populated correctly?

[havref]

Ah, thanks! Now I understand a bit more. I've been scratching my head regarding the memory channels/memory bandwitdth question, but could not find an answer stating it clearly enough until now. Maybe it is too obvious?

I'll do a check of both speedup factor and number of DIMMs in the old system. Thanks again.

[SLC]

Originally Posted by flotus1

For this setup, you MUST use 24 DIMMs, 6 DIMMs per CPU. If memory channels are not populated, the CPU can not use their additional memory bandwidth. With only 8 or 12 DIMMs you will waste more than half of the performance this system can have for CFD.
With that fixed, the setup should be more than twice as fast compared to your old setup. But of course only if your simulations are not i/o-bound and scale well on such a large amount of cores.
If it does not scale so well, you can settle for a lower core count processor model and save a few bucks. Or if money is not that much of an issue use less cores per CPU but spread evenly among the sockets.

Edit: if you haven't done it yet, you can see how your jobs scale on your old system by running the same simulation on a single core and on an increasing amount of cores, typically doubling the core count each time. Then put the speedup factor in a chart...
When you are unsure if you are spending those 26000$ wisely, this might help you to make the right decision.

And I have to ask: are you sure that your old workstation has its memory populated correctly?

If you were to go with a dual-socket setup (say 2 x Intel Xeon Gold 6154 3.0GHz), would the optimum setup be with 12 DIMMS then? 6 DIMMS per CPU still?

[flotus1]

Exactly. Each memory channel needs its own DIMM (dual-rank preferably). Using two or three DIMMs per channel should usually be avoided unless it is the only way to get the memory capacity you need.