[aparangement]

Originally Posted by flotus1

I can't rerun the case since I no longer have access to this computer.

However...
One of the goals for proper benchmarking is reducing variance. Controlling core binding is one necessary step along that route. Without it, results can be all over the place. It's just due to statistics that with this many threads and NUMA nodes, variance isn't actually that high, and the average drops significantly. Pulling average performance towards the maximum is a nice side-effect

Milan is not that much faster than Rome. We already have results from a Rome CPU that are closer to mine: OpenFOAM benchmarks on various hardware

Simlar results here: OpenFOAM benchmarks on various hardware

Thanks for the information.

But talking about impovement from epyc Rome to Milan, I think 15% (16/14, actually I could only get it to around 18 seconds) is pretty decent already .. I was assuming that only single thead performance could achieve such improvement.

Originally Posted by aparangement

7532*2, numa NPS4, ddr4 3200 16g*16 2R, ubuntu18.04, ofv8 commit-30b264 (copiled normally without any special gcc tuning.):

# cores   Wall time (s):
------------------------
1 730.97
2 342.93
4 171.78
8 81.72
16 41.99
24 29.84
32 23
48 20.04
64 18.4

[cfdInCivil]

Thank all of you for the benchmarks!

I can build at the moment my own configuration and am now thinking of which processor/configuration would be the best. (15-20k€)
Now I am thinking of 2x AMD Milan 7643 (48c). Does anyone have experience with this processor?

Does the amount of cash have an big influence on the performance?

https://www.phoronix.com/review/epyc-7003-linux-perf/8
In this benchmark with openfoam 8 the EPYC 7713 (64c) and 75F3 (32c) are really close. Well, also many benchmarks showed not so big improvements above 32 cores. So is it worth it to invest in 2x48c ?

[flotus1]

Yes, both the amount of cash and the amount of cache have an impact on performance.
As you already noticed, there is not much to be gained beyond 32 cores per CPU. If you want further improvements, Milan-X CPUs are the way to go because of the larger L3 cache.
The 32-core 7573X is currently the best processor for CFD. We don't have any benchmarks here, but you can expect somewhere around 20-30% higher performance compared to the 7543. Whether that is worth the 6000€ price tag is up to you.

[cfdInCivil]

Thank you, that helps a lot!

In theory: if the larger cash helps in the scalability, could then be expected that the Milan 7773X with 64 cores and 768MB cash could have further increase in performance?
Or is the "performance limitation" to 32 cores more due to the communication bottle neck between the cores?

[flotus1]

It's less about scaling, and more "performance per core".
The added L3 cache give a performance uplift at all thread counts, and thus won't allow better scaling beyond 32 cores.
That and the 9000€ per CPU won't be easy to fit into a 20000€ budget.

[cfdInCivil]

Ah ok, thank you!
Yes, the 9000€ does of course not fit into the budget.

[Crowdion]

Interesting article on WSL2 vs Linux (HPL HPCG NAMD) by Dr. Donald Kinghorn
https://www.pugetsystems.com/labs/hp...PCG-NAMD-2354/

[xuegy]

14in MacBook Pro base model (M1 Pro, 6P+2E, 16GB RAM), macOS 12.5.1, Apple clang 13.1.6, OpenFOAM-v2206

# cores Wall time (s):
------------------------
1 463.91
2 249.77
4 141.38
6 108.45
8 143.51

The efficiency core doesn't help at all. It's slightly faster than my M1 Mac mini due to more memory bandwidth (200GB/s).
I really want to see the result on the 16-core Mac Studio with 800GB/s bandwidth.

[draagnuab]

I just ran the benchmark on my laptop from 2018.

OS: Windows 10 with WSL1 running Ubuntu 22.04.1 LTS
CPU: i5-8265u (4 cores)
RAM: 2 x 8GB DDR4-2400
OF: openfoam.com precompiled ubuntu v2206

Meshing Times:
#Cores    Wallclock time [s]    Speedup
1            1313.91              1.00
2            905.5                1.45
4            804.88               1.63

Flow Calculation:
#Cores    Wallclock time [s]    Speedup
1            1187.68              1.00
2            543.19               2.19
4            442.45               2.68

It puzzled me to see a flow calculation speedup of 2.19 by using two cores, so I ran the testcase again, but got a similar result. Does anyone have an idea of why?
My laptop chokes at 4 cores, which I presume is because of its low memory bandwidth.

[draagnuab]

Originally Posted by draagnuab

I just ran the benchmark on my laptop from 2018.

OS: Windows 10 with WSL1 running Ubuntu 22.04.1 LTS
CPU: i5-8265u (4 cores)
RAM: 2 x 8GB DDR4-2400
OF: openfoam.com precompiled ubuntu v2206

Meshing Times:
#Cores    Wallclock time [s]    Speedup
1            1313.91              1.00
2            905.5                1.45
4            804.88               1.63

Flow Calculation:
#Cores    Wallclock time [s]    Speedup
1            1187.68              1.00
2            543.19               2.19
4            442.45               2.68

It puzzled me to see a flow calculation speedup of 2.19 by using two cores, so I ran the testcase again, but got a similar result. Does anyone have an idea of why?
My laptop chokes at 4 cores, which I presume is because of its low memory bandwidth.

I updated from WSL1 to WSL2, which gave a significant faster 1-core performance (18% faster mesh, 62% faster flow calculation), while the 2- and 4-core timings were more or less the same. Hence, the flow calculation speedup is no longer superlinear.

[wkernkamp]

Upgraded my DL560 Gen 8 with E5-4657Lv2 processors:
OpenFOAM v2112, 4x E5-4657Lv2, 16x *Gb R2 DDR3-1866


Meshing Times:
36 169.59
40 166.07
44 167.83
48 182.17
Flow Calculation:
1 1206.04
2 599.21
4 240.81
6 167.69
8 124.5
10 107.11
12 89.31
16 70.49
18 66.72
22 57.48
24 53.07
28 48.24
32 45.56
36 43.43
40 41.97
44 41.07
48 40.41



E5-4627v2 fastest time 48.62 on 32 cores.

[danbence]

This website compares the 7950X to the 5950X and Pro 5995WX on motorbike

https://www.pugetsystems.com/labs/hp...2368/#OpenFOAM

[wkernkamp]

I like to see that the Tr Pro 5995wz (cpu = $7000) completes the benchmark in the same time as my cheap DL560 G8 (total $700).

[c_reyner]

Hello,
I've read almost all about the guidelines and comparisons here, which really lightened me up, thank you for all the efforts!

My first preference is to use a dual Xeon E5 2680/2683 V4,

However, I am budget tight, and going to build for my research project, and found several cheaper options (half processor price):
E5 2650v4 and E5 2640v4
How is the performance difference? is it far or just around 5-10% ?

Or, should I invest a bit more in E5 2667v4 ?
(based on my understanding, this might be faster than E5 2680v4)

Thank you very much!

[wkernkamp]

Originally Posted by c_reyner

Hello,
I've read almost all about the guidelines and comparisons here, which really lightened me up, thank you for all the efforts!

My first preference is to use a dual Xeon E5 2680/2683 V4,

However, I am budget tight, and going to build for my research project, and found several cheaper options (half processor price):
E5 2650v4 and E5 2640v4
How is the performance difference? is it far or just around 5-10% ?

Or, should I invest a bit more in E5 2667v4 ?
(based on my understanding, this might be faster than E5 2680v4)

Thank you very much!

Normally, this type of question is asked in a separate thread within "hardware". Once you have run the benchmark on your new system, you publish your results here.

The E5-2640 v4 is less desirable for CFD, because it is limited to DDR4-2133 and a QPI link of 8 GT/s, while the others go to DDR4-2400 and 9.6 GT/s. This will cause a performance difference of about 12.5%. Probably a bit more due to the lower core count and smaller cache.

On ebay, I saw the E5-2680 v4 for $62 and the E5-2650 v4 for $14.94. The E5-2650 v4 performance will probably be in your range of 5-10% less than the E5-2680 v4.

[sourav90]

Originally Posted by xuegy

14in MacBook Pro base model (M1 Pro, 6P+2E, 16GB RAM), macOS 12.5.1, Apple clang 13.1.6, OpenFOAM-v2206

# cores Wall time (s):
------------------------
1 463.91
2 249.77
4 141.38
6 108.45
8 143.51

The efficiency core doesn't help at all. It's slightly faster than my M1 Mac mini due to more memory bandwidth (200GB/s).
I really want to see the result on the 16-core Mac Studio with 800GB/s bandwidth.

• 2021 MacBook Pro 16" with M1Pro & 32GB memory
• MacOS 13 (Ventura Beta!)
• pre-compiled OpenFOAM.app (v2206, https://github.com/gerlero/openfoam-app)
  

PHP Code:

# cores   Wall time (s):
------------------------
1 463.06
2 246.83
4 143.24
8 92.3 


Will be doing it with OpenFOAM complied on this set-up (using https://github.com/BrushXue/OpenFOAM-AppleM1), once I manage to get scotch working

[xuegy]

Will be doing it with OpenFOAM complied on this set-up (using https://github.com/BrushXue/OpenFOAM-AppleM1), once I manage to get scotch working

The result would be the same. Both are native compiled.

[hami11]

Anyone have any comparisons between naples and rome?

[flotus1]

Epyc Rome: OpenFOAM benchmarks on various hardware

Epyc Naples: OpenFOAM benchmarks on various hardware

-Solver run time in seconds-
nthreads   Naples    Rome
      01    907.7   643.8
      64     27.7    16.0

So a pretty huge generational leap. Big enough to dismiss Naples entirely when shopping for used components.Some of that advantage comes from the 7532 having 256MB of L3 cache, which you won't find the cheapest deals for.

[hami11]

What would be the performance advantages from going to a 64-core dual epyc system to a 128-core dual epyc system?