[lasersim]

Dear CFD users,

I work in a company and we are trying to scale up our CFD capabilities from a server to a small cluster. Our major CFD tool is ANSYS Fluent. We have 3 HPC pack licenses and that is good for running Fluent on 128 cores. I have got 2 quotes from different cluster suppliers based on the same budget. I am not familiar with cluster stuff and so I am here to seek suggestions/comments. I want to know which quote is better and how to optimize them. Here I list the details of these 2 clusters:

Cluster 1:
8 compute nodes @ 128 cores
Processor: Intel XEON e5-2667 v4 CPU (3.2GHz/3.6GHz HT) (16 X 8c)
Chassis: 4 x 1U x 2 Node chassis (1000 Watt power supply)
Memory: 8 x 128 GB DDR4-2400MHz LRDIMM (8 X 16 GB)
OS Data: 8 x 2 x 1TB SATA 7200 RPM 6Gbp/s
Interconnect: 8 x 56 Gbps Infiniband
Interconnect Switch: 12 port Mellanox FDR switch
Operating System: CENTOS 7

Cluster 2:
8 compute nodes @ 192 cores
Processor: Intel Scalable Xeon SKL-SP GOLD 5118 12C/24T 2.3G 16.5M 10.4GT UPI
Head Node: 8 cores (2 x SILVER 4112), 96GB Memory (6 x 16GB) DDR4 ECC RDIMM, 32TB Storage (8 x 4TB) RAW-RAID 10/5/6 all available 9SAS3 Discrete RAID)
Each Compute Node: 24 cores ( 2x GOLD 5118), 192 GB Memory (12x 16GB) DDR4 ECC RDIMM 2666MHz
Interconnect: 36 port EDR InfiniBand Switch from Mellanox
24 Port Management Switch
Operating System: Read Hat Enterprise Linux

Any suggestions or comments are welcome! Thank you in advance!

[evcelica]

CFD is usually all about memory bandwidth.

The Gold CPUs have 6 memory channels, where the v4 only have 4. So cluster 2 will have 150% of the memory bandwidth, and should be faster. Looks like that CPU only supports 2400 MHz Ram, not 2666 as you show.
The XEON Gold 6126 is higher CPU clock and supports the 2666MHz memory, with a modest price increase ($450 per CPU)
Here is a list of the new INTEL CPus: https://ark.intel.com/products/serie...ble-Processors

People have been looking at the AMD EPYC CPUs recently as well, and there have been some discussions on this forum.

[lasersim]

Thank you, Erik! I will discuss with the vendor about XEON Gold 6126.

[Kaskade]

Some notes since I am also looking into getting hardware for Fluent and 3 HPC-Packs:

- FDR will not be significantly slower than EDR, but much cheaper
- Omnipath offers the bandwidth of EDR at the price of FDR
- go with Skylake-SP because of the number of memory lanes
- I am going with 16x Xeon Gold 6130. They run at high frequencies when only 8 of the available 16 cores are in use. But if I rent a fourth HPC-Pack I can unlock 50 to 75 % extra performance by running on all available 256 cores.
- one 4-socket system should be slightly cheaper than two 2-socket system. Plus it is easier to handle.

[lasersim]

Thank you, Kaskade!