[Freewill1]

Hi all,

I need a machine for OpenFOAM modeling with budget of $15000~$20000.

My typical number of cells is ~125M (500x500x500).

I inclined to choose AMD's EPYC 7002 CPU because of their

• 8 channels of ECC RAM support
• higher price/performance ratio over Intel's Xeon CPU

Can anyone help with questions as follows:

• Is a single-socket or two dual-socket workstation competent to the task?
• If I choose 64-core CPU/CPUs, for the same cores (up to 64 available for each 2nd gen EPYC CPU), which should I choose, one single-socket or two dual-socket, e.g., 1x7742 (64 cores/128 threads) vs. 2x7542 (32 cores/64 threads each)?
• How many GBs of ECC RAM should I prepare for the job, 256 GB, 512 GB, or 1.0 TB?

AMD EPYC.jpg
Here are the reference prices for key components from NEWEGG:
CPU:
1xEPYC 7742: $8,513x1 = $8,513
2xEPYC 7542: $4,122x2 = $8,244

ECC RAM (512GB = 32GBx16 optimized for the 8-channel RAM support of CPU): ~$170x16 = $2,710

Thanks in advance.

[flotus1]

Originally Posted by Freewill1

I want to buy a machine for OpenFOAM modeling with budget of $1500~$2000.

Missing a few zeros? The options you are discussing are way more expensive.

[Freewill1]

Originally Posted by flotus1

Missing a few zeros? The options you are discussing are way more expensive.

Wrong numbers fixed.

[flotus1]

Is a single-node or two dual-node workstation competent to the task?

Anything with enough memory can run it. Rule of thumb: more CPUs equals more faster
So if you can squeeze it into your budget, two dual-socket nodes would be ideal. If you want everything inside one workstation-style box, then dual-socket is the way to go. Of course, that option will be slower than two nodes, even if they have lower core count CPUs. Avoid a single CPU with extremely high core count, that's just a waste of money for CFD.

f I choose 64-core CPU/CPUs, for the same cores of up to 64 cores (for available 2nd gen EPYC CPUs), which should I choose, one single-node or two dual-nodes, e.g., 1x7742 (64 cores/128 threads) vs. 2x7542 (32 cores/64 threads each)?

Shared resources are key here, especially memory channels. The more CPUs you have, the more shared resources are available, which helps avoiding bottlenecks.
Sticking to your example: 2x Epyc 7542 will be about twice as fast for running simulations in OpenFOAM, compared to a single Epyc 7742. The latter will be severely limited by a lack of memory bandwidth, and stop scaling somewhere between 24-32 cores.

How many GBs of ECC RAM should I prepare for the job, 256 GB, 512 GB, or 1.0 TB?

While 256GB should be enough for standard aerodynamic simulation with 125M cells, I would recommend 512GB total for your budget. Once you have the compute power to run larger models, your cell counts will increase automatically

With all that out of the way: don't buy the Epyc 7542 either. It costs almost twice as much as an Epyc 7452 (same core count and chip layout, but lower TDP), and the performance difference running OpenFOAM will be negligible.
If you feel up to the task of connecting two machines and running the case distributed, two machines with 2x16 or 2x24 cores and 256GB RAM (16x16GB) each would be even better.

For memory, make sure to get DDR4-3200 reg ECC. Epyc can't handle unbuffered memory.

[Freewill1]

Originally Posted by flotus1

Anything with enough memory can run it. Rule of thumb: more CPUs equals more faster
So if you can squeeze it into your budget, two dual-socket nodes would be ideal. If you want everything inside one workstation-style box, then dual-socket is the way to go. Of course, that option will be slower than two nodes, even if they have lower core count CPUs. Avoid a single CPU with extremely high core count, that's just a waste of money for CFD.


Shared resources are key here, especially memory channels. The more CPUs you have, the more shared resources are available, which helps avoiding bottlenecks.
Sticking to your example: 2x Epyc 7542 will be about twice as fast for running simulations in OpenFOAM, compared to a single Epyc 7742. The latter will be severely limited by a lack of memory bandwidth, and stop scaling somewhere between 24-32 cores.


While 256GB should be enough for standard aerodynamic simulation with 125M cells, I would recommend 512GB total for your budget. Once you have the compute power to run larger models, your cell counts will increase automatically

With all that out of the way: don't buy the Epyc 7542 either. It costs almost twice as much as an Epyc 7452 (same core count and chip layout, but lower TDP), and the performance difference running OpenFOAM will be negligible.
If you feel up to the task of connecting two machines and running the case distributed, two machines with 2x16 or 2x24 cores and 256GB RAM (16x16GB) each would be even better.

For memory, make sure to get DDR4-3200 reg ECC. Epyc can't handle unbuffered memory.

Hi Alex, thank you for your pertinent suggestions to the points.

I feel surprised by what you mentioned about the bottleneck problem of the memory bandwidth for CFD (maybe other types of numerical computations too), as well as the low efficiency of budget if choosing EPYC 7742/7542, which I was not acquainted with.

[Freewill1]

Hi Alex,
After careful thinking, new questions arise for me.
If I manage to set up two nodes working as a cluster, as mentioned above as the best option among others, additional interconnection between nodes will be needed, e.g., Infiniband now at node-to-node speed of ~100Gbps at most.
Will this low-bandwidth network counteract the benifits of the much more rapid eight-channel Epyc-RAM exchange and bring in new bottleneck?
Also, Infiniband network system seems quite expensive and requires extra budget?
I have on idea if I understand these issues correctly.

[flotus1]

I feel surprised by what you mentioned about the bottleneck problem of the memory bandwidth for CFD (maybe other types of numerical computations too), as well as the low efficiency of budget if choosing EPYC 7742/7542, which I was not acquainted with.

It's not a new concept, or something I came up with.
Most FV CFD codes have a rather high code balance. Meaning a high amount of data traffic per floating point operation.
Combined with machine balance getting smaller and smaller thanks to ever increasing core counts and IPC, with memory bandwidth barely keeping up, memory bandwidth bottlenecks have long been a thing in CFD, and will become more frequent in other applications in the future.
Here is what that looks like in OpenFOAM:
scaling.png
Results were taken from the pinned thread in this sub-forum. The 7551 result is mine.
No, this is not just the test case being too small for high core counts. This case can scale to even higher core counts, albeit on several machines.

Will this low-bandwidth network counteract the benifits of the much more rapid eight-channel Epyc-RAM exchange and bring in new bottleneck?

Infiniband as a node interconnect is not a bottleneck for OpenFOAM, especially not for only two nodes. If that was the case, clusters would not be a thing.
Parallelization in OF is implemented via MPI+domain decomposition. The only data that has to be transferred between cores are values on the domain boundaries. This pales in comparison to the amount of data that has to be accessed for the algorithm itself.
As a matter of fact, you could probably get away with 10Gig Ethernet as an interconnect for only two nodes. If you want Infiniband, all you need are two cards and a cable for two nodes. No expensive switch required. The main benefit of Infiniband over Ethernet as node interconnect is not bandwidth, but latency. So you don't have to go all in with 100G or even faster Infiniband cards. 40G or 56G would be plenty.

[Freewill1]

Many thanks, I have learned quite a lot from your expertise.