[Simbelmynë]

Originally Posted by blackcatxiii

Hi everyone

I have collected the benchmark data posted in this thread in an Excel spreadsheet and plotted out walltime as well as speed up for comparison.

I hope it is helpful for those who plan to build a new PC for OpenFOAM.

Nice, thank you. It would also be interesting to have a column for memory speed and rank, but I guess that is out of the question right now seeing that the work to do so is quite massive.



What makes me most interested when looking at this is the extremely impressive results of Ryzen 3900X. It manages the same results as the Threadripper 1950X (and similar 4 channel setups), while it uses only half of the memory channels. It is very clear that Ryzen 3rd generation benefits immensely from tight timings on the memory.

[pred]

HPE DL385 GEN10 Plus 2*EPYC 7542 32Core, 16*32GB 3200MHz Memory
Nothing optimized - just set Bios to HPC and installed Centos 8 with Openfoam 5.0

# cores Wall time (s):
------------------------
1 600.7
2 348.89
4 152.27
6 101.78
8 76.09
12 53.49
16 40.26
20 36.63
24 31.95
28 29.98
32 27
36 26.45
40 25.85
44 25.09
48 23.27
52 23.47
56 23.46
60 22.5
64 21.96

[giovanni.medici]

Hi there,
we ran the benchmark on a similar setup as @pred, and found pretty similar results:

• 2x AMD Epyc 7542
• PowerEdge R6525
• 256gb RAM 3200Mhz

Ubuntu server

# cores   Wall time (s):
------------------------
1 724.16
2 346.29
4 165.72
6 107.43
8 82.14
12 55.02
16 41.32
20 37.03
24 33.5
32 26.79
48 22.99
64 21.5

[fedez91]

2*Intel Xeon Gold 6140
96gb (12*8gb) 2666 MHz

#cores   Wall time (s):
------------------------
1            981.72
2            488.92
4            217.97
6            146.36
8            113.38
12           85.34
16           68.23
20           60.27
24           55.94
28           52.5
32           50.76
36           49.87

Improvement seems to become insignificant after 24 cores... any idea why this may happen?

[flotus1]

Improvement seems to become insignificant after 24 cores... any idea why this may happen?

Same old story: running out of memory bandwidth.

[fedez91]

Originally Posted by flotus1

Same old story: running out of memory bandwidth.

Any way of improving this or it just means that I got 12 extra cores that are basically useless? Ot this would change with bigger meshes? I apologise if my questions are naive, but I do not a really understand the details of how cpu works.

Thank you for your help!

[flotus1]

There is not much you can do about it, once you purchased the hardware. Higher cell counts won't improve this behavior. It is a function of code balance, which does not change drastically with cell count.
With Intel Xeon CPUs, you can try to enable "cluster on die" mode in the bios. Might also be called "sub-NUMA cluster" with this generation. That should improve latency and bandwidth a bit for NUMA-aware software like OpenFOAM. Don't expect huge improvements though. https://en.wikichip.org/wiki/intel/m...UMA_Clustering
And it might be a good idea to check whether the DIMMs are populated correctly, so each memory channel of each CPU has one DIMM. You should be able to look up the correct way in your server/workstation/motherboard manual, and then compare it to what you got.
At least the additional cores are not entirely wasted, there is still a small speedup. And contrary to commercial CFD solvers, you don't have to buy additional licenses to use them.

[sida]

I feel lucky after reading this thread and before making my decision.

With 2500$ budget,( previously going to spend on Threadripper 3970x and its cooling system) which alternative CPUs do you suggest?

I'm can't decide between one AMD EPYC 7452 or 2X EPYC 7302???

Thanks in advance

[fedez91]

Thanks flotus1. I will enable 'cluster on die' and re-run the test. I will post the results if I see improvements. Thanks again!

[wkernkamp]

Originally Posted by sida

I feel lucky after reading this thread and before making my decision.

With 2500$ budget,( previously going to spend on Threadripper 3970x and its cooling system) which alternative CPUs do you suggest?

I'm can't decide between one AMD EPYC 7452 or 2X EPYC 7302???

Thanks in advance

The 2x 7302 will be faster, because it has twice the memory channels of the single 7452.

[linnemann]

Tested on 2xEPYC Rome 7302, 256Gb ram 32x8Gb@2933
No core binding or trickery.

With AOCC 2.1/GCC 9.2.0 and Openfoam 19.12

# cores	Wall time (s)		
	AOCC 2.1.0        GCC 9.2.0 
        -march=znver2     -march=znver2    Diff %
1	693.3            692.5	            0%
2	470.3            470.88	            0%
4	167.2            164.52	            -2%
8	78.5             77.16	            -2%
12	59.5             60.26	            1%
16	42.3             41.79	            -1%
20	41.2             41.07	            0%
24	33.3             33.59	            1%
28	34.0             32.36	            -5%
32	28.2             27.95	            -1%

So no real benefit going for AOCC/Clang

[Fabian2602]

Results on a Ryzen 7 3700X with overclocked Memory (3800 - CL 16).

# cores Wall time (s)
1 857.26
2 380.49
4 253.1
6 219.94
8 212.16

[Simbelmynë]

Originally Posted by Fabian2602

Results on a Ryzen 7 3700X with overclocked Memory (3800 - CL 16).

# cores Wall time (s)
1 857.26
2 380.49
4 253.1
6 219.94
8 212.16

I think you can do much better. Have you tried Ryzen DRAM calculator? I only managed 3600 CL16 @ 1:1 infinity fabric, but with the secondary and tertiary timings set properly I got around 170 s in this test on my Ryzen 3700X system.

[Fabian2602]

Okay that's a huge difference. I have chosen the values from a thread in the computerbase forum. The if-fabrik is 1:1 for me as well. I will check today.

[Simbelmynë]

You can download the dram calculator here.


There are so many settings so it is better if you just try the calculator yourself rather than me posting them all here. Good luck!

[ships26]

For anyone considering an older build, I ran this benchmark with 2 x E5645. Mind you, it is openfoam4.1 on debian Jessie (Kernel 3.16):

# cores   Wall time (s):
------------------------
1 1546.8
2 859.93
4 414.82
8 309.71
10 297.45
12 295.6

I'm not being able to check the memory specs atm, but I'll edit this post if I manage to.
My guess is that the tri channel memory might holding it back from a better 10-12-thread scaling.

[Simbelmynë]

Originally Posted by ships26

For anyone considering an older build, I ran this benchmark on an E5645. Mind you, it is openfoam4.1 on debian Jessie (Kernel 3.16):

# cores   Wall time (s):
------------------------
1 1546.8
2 859.93
4 414.82
8 309.71
10 297.45
12 295.6

I'm not being able to check the memory specs atm, but I'll edit this post if I manage to.
My guess is that the tri channel memory might holding it back from a better 10-12-thread scaling.

If you have sudo rights then you might be able to find out using:

dmidecode -t 17

[ships26]

Unfortunately I don't
I tried it without sudo, and it only gives me the info that there's 50GB, nothing about frequencies. I can try to find a way around it, though.

[wkernkamp]

Originally Posted by ships26

For anyone considering an older build, I ran this benchmark on an E5645. Mind you, it is openfoam4.1 on debian Jessie (Kernel 3.16):

# cores   Wall time (s):
------------------------
1 1546.8
2 859.93
4 414.82
8 309.71
10 297.45
12 295.6

I'm not being able to check the memory specs atm, but I'll edit this post if I manage to.
My guess is that the tri channel memory might holding it back from a better 10-12-thread scaling.

My results with the faster X5670 processors:




2xX5675 3.07ghz 6 cores per cpu

Meshing Times:
1 1998.08
2 1313.22
4 719.71
6 558.17
8 466.22
12 449.43
Flow Calculation:
1 1322.84
2 787.4
4 375.77
6 305.44
8 286.3
12 278.02


Looks to me like you are doing about as well as you can with the setup.

[ships26]

Thank you for sharing your results, wkernkamp! It's always great to have a point of comparison.



Did you stick to a single cpu in hyperthreading when running the benchmark with 12 threads?