[ym92]

Not much difference for dual 64-core setup compared to 64 cores on Epyc Rome (as can be expected). Wanted to post these results already for a while but better late than never..

Hardware: 2x AMD Epyc 7742, 16x32GB DDR4-3200
Software: Ubuntu 20.04, OpenFOAM v1812

# cores Wall time (s):
------------------------
1 | 936.35
2 | 521.5
4 | 236.56
6 | 158.72
8 | 120.83
12 | 77.94
16 | 57.41
20 | 46.4
24 | 39.23
48 | 22.79
64 | 19.02
126 | 15.46

[hami11]

So I could expect a roughly 20% speedup from 2x 7452 vs 2x7742 config? That is about the same as the speedup from rome to milan.

[ym92]

Originally Posted by hami11

So I could expect a roughly 20% speedup from 2x 7452 vs 2x7742 config? That is about the same as the speedup from rome to milan.

Hmm, I don't know if there is a benchmark here for 2x7452 here, but I would expect a 2x7452 to perform better than 2x 7742 with 64 cores in use. Therefore, the gain would probably be less than 20%. Anyway, I would rather discourage you from buying 2x 7742 unless you get them for the same price as 2x 7452 or you have other use for it. The memory bandwidth is still the same for both CPUs. But I think there are other people in this forum that have more experience .

[oswald]

Hardware: 2x EPYC 7573X, 16x 32GB DDR4
Software: Ubuntu 20.04.3, OF7

cores   Wall time (s)
1    492.5
4    113.53
8    57.91
12    39.68
16    31.88
20    28.08
24    25.14
28    24.14
32    22.34
40    21.49
48    17.17
56    12.53
64    11.55

I did not use core bindings, which might explain the bad scaling behaviour when using 20 to 40 cores. Compared to my 2xEPYC7543 workstation, this machine is ~33% faster on 64 cores.

[flotus1]

That's awesome!
If you don't mind, could you run the 64-thread case once more? Since you are the first person here with Milan-X, I would really like to know where the limits are.

first clear caches by running this command with root privileges:
echo 3 > /proc/sys/vm/drop_caches

Then change to run directory for 64 threads and run the solver once more with:
mpirun -np 64 --report-bindings --bind-to core --rank-by core --map-by numa simpleFoam -parallel > log.simpleFoam 2>&1

[oswald]

Of course, no problem. With this I get an execution time of 11.3s.


Do you have a recommendation regarding NUMA-setup in the BIOS?

[flotus1]

Just the usual: NPS=4 -> results in a total of 8 NUMA nodes
Or if it's available, depending on your motherboard: ACPI SRAT L3 cache as NUMA domain -> results in 16 NUMA nodes.

The latter is really only good for software like OpenFOAM. And other solvers that can also be run on distributed memory systems. For most other software, it will do more harm than good.

[oswald]

Thank you for the hints regarding the NUMA stuff. Here is the results with "ACPI SRAT L3 Cache as NUMA" enabled:

#cores Wall time (s)
 1      476.71
 4      111.69
 8      53.96
12      38.12
16      28.58
20      26.73
24      23.51
28      20.43
32      18.05
40      17.71
48      15.34
56      12.26
64      11.22

As the workstation is in principle only used for OpenFOAM, I will let this option turned on for now.

[mllokloks]

I've been looking the results posted above. It seems that EPYC 7532 perform slightly better than EPYC 7542.

I'm planning to build a workstation for OpenFOAM, mostly using simpleFoam/pisoFoam/pimpleFoam with ~20M mesh. Should I go for 7532 instead of 7542?

Do the results posted here still applicable to ~20M mesh?

Thank you guys


7542:
Cache: 128MB
Base Clock: 2.9GHz

7532:
Cache: 256MB
Base Clock: 2.4GHz

[flotus1]

Do the results posted here still applicable to ~20M mesh?

Yes, they do. The mesh used here is large enough so the results are also applicable to much higher cell counts.

Base clock speeds for these CPUs don't really tell us much about actual performance. Even with semi-decent cooling, they will operate at higher clock speeds 24/7. And CPU clock speeds don't translate 1:1 into lower solver times, because 32 cores on 8 memory channels are not entirely compute-bound.
The Epyc 7532 is the faster CPU for OpenFOAM/CFD compared to the Epyc 7542, thanks to twice the amount of L3 cache.

[naffrancois]

Hello,

The creator of this thread has gone for a long time now. There is now a factor 3 between first gen EPYC 7501/7601 and this last CPU, which is quite impressive ! Would there be any chance to compile some of the results in an updatable chart, as was the original plan ? I understand this would require a lot of work but maybe by only keeping main CPU's from each generation ? This would help, I think, to get a clearer idea and redirect people asking for a new configuration ?

[flotus1]

If *someone* were to compile such a chart, I would be happy to include it in the first post.

[naffrancois]

ok, if you think it can be useful I can give it a try. I don't promise anything though

[danbence]

On OpenFOAM 10 drivaerFastback, the EPYC 9554 2P (192 cores) was 49% faster than the Epyc 7773X 2P (128 cores), so the performance has scaled with the core count. Impressive.

https://www.phoronix.com/review/amd-...-benchmarks/14

[flotus1]

9554 should be a 64-core part.
It's the 96-core 9654 that has a 49% advantage in this benchmark. It's only a 28% increase for the 64-core comparison.
Or if we want to leave 3D v-cache out of the picture for now and compare 64-core SKUs:
2P 7763: 277s
2P 9554: 130s
Which falls in line with the 2.2x increase in theoretical memory bandwidth.
Epyc Genoa with 3D v-cache will follow in the first half of 2023.

But the usual disclaimer still applies: openbenchmarking.org
I don't trust the OpenFOAM numbers there.

[naffrancois]

Originally Posted by flotus1

If *someone* were to compile such a chart, I would be happy to include it in the first post.

I compiled the results published so far, took the best or the one with the most complete description when duplicated configurations were available. I set them into two charts, max perf and single core perf. If you have some other filters that may be interesting let me know, it is all stored in an excel file. Thank you all for the contribution, it is great to gather so many results and get the big picture among all these cpus

[naffrancois]

here's the excel if anyone wants to have a look or modify (unfortunately I realized too late some members already started a similar data base)

[Crowdion]

Originally Posted by naffrancois

here's the excel if anyone wants to have a look or modify (unfortunately I realized too late some members already started a similar data base)

Thanks for taking the time to create the spreadsheet !

Small clarification: my systems Cisco C460 M4 (message number 544) has 4 x E7-8880 V3 (not 4 x E7-8880) CPUs.

Installed RAM: 32 x 16Gb DDR4-2133, CISCO UCS-MR-1X162RU-A. The RAMs runs at 1600MHz, however, the servers motherboard has installed "Jordan Creek 2" scallable memory buffer interface, which enable to double the memory channels bandwidth up to 3200MT/s (actual bandwidth measured using intel mlc 3.9a is 320 GB/s). It is important that at least 32 RAM slots be filled with 4 processors installed.
It seems that at the moment the Cisco C460 M4 is one of the most productive systems in this topic evaluating "it/s/$".

[naffrancois]

Originally Posted by Crowdion

Thanks for taking the time to create the spreadsheet !

Small clarification: my systems Cisco C460 M4 (message number 544) has 4 x E7-8880 V3 (not 4 x E7-8880) CPUs.

Installed RAM: 32 x 16Gb DDR4-2133, CISCO UCS-MR-1X162RU-A. The RAMs runs at 1600MHz, however, the servers motherboard has installed "Jordan Creek 2" scallable memory buffer interface, which enable to double the memory channels bandwidth up to 3200MT/s (actual bandwidth measured using intel mlc 3.9a is 320 GB/s). It is important that at least 32 RAM slots be filled with 4 processors installed.
It seems that at the moment the Cisco C460 M4 is one of the most productive systems in this topic evaluating "it/s/$".

Thanks for the explanation, I will add your remarks to the spreadsheet. I would have liked to add the cost as another indication. But unfortunately I have no idea how to manage this as prices evolve a lot in time and space.

[hami11]

Originally Posted by flotus1

9554 should be a 64-core part.
It's the 96-core 9654 that has a 49% advantage in this benchmark. It's only a 28% increase for the 64-core comparison.
Or if we want to leave 3D v-cache out of the picture for now and compare 64-core SKUs:
2P 7763: 277s
2P 9554: 130s
Which falls in line with the 2.2x increase in theoretical memory bandwidth.
Epyc Genoa with 3D v-cache will follow in the first half of 2023.

But the usual disclaimer still applies: openbenchmarking.org
I don't trust the OpenFOAM numbers there.

So it looks like genoa is 2x faster than rome on a core for core basis?