[flotus1]

Do you mean the compiled data in the first post of this thread?

eric collected the data at an early stage, when the amount of contributions was still manageable.
Now we are at over 700 posts in this thread, with a pretty wide variety of different setups. There must be some results with Xeon Platinum CPUs hidden somewhere in here. I'm sure they can be found via the search function. Some people even tried compiling the additional data points into new tables.

Side-note: there probably aren't too many Platinum CPUs, because for this workload, there is usually a Xeon Gold CPU that delivers similar performance, at a much lower price.

[Crowdion]

Lenovo ThinkStation P520c, Xeon W-2145 (8x4.3Ghz, HT On), 4 x 32GB DDR4 2666Mhz , OpenFoamV11 (precompiled), Ubuntu 22.04.3 LTS, bench_template_v02.zip (default settings)

Meshing (real) 
# cores | Meshing Wall time| Solver Wall time(s):
------------------------
1  | 30m33s  |  1006
2  | 19m16s  |  513
4  | 10m29s  |  248
8  | 6m14s   |  172

UPD: nor compiling of OFv11, nor disabling of HT almost does not change the wall time.

UPD2:the same workstation and benchmark running via WIN10(22H2)+ WSL2 + Ubuntu 22.04.2 LTS + OFv11 (precompiled) (HT on)

Meshing (real) 
# cores | Meshing Wall time| Solver Wall time(s):
------------------------
8  | 8m54s  | 217

WSL slows down calculations by ~26%.

UPD3: the same workstation, the benchmark Motorbike_bench_template.tar.gz(default settings) WIN10(22H2)+ WSL2 + Ubuntu 22.04.2 LTS + OpenFOAM-v2306 (precompiled) (HT on),

Meshing (real) 
# cores | Meshing Wall time| Solver Wall time(s):
------------------------
8  | 4m11s  | 196

UPD4: the same workstation, pure Ubuntu install, the benchmark Motorbike_bench_template.tar.gz(default settings) Ubuntu 22.04.3 LTS + OpenFOAM-v2306 (precompiled) (HT on)

Meshing (real) 
# cores | Meshing Wall time| Solver Wall time(s):
------------------------
1  | 11m11s  |  903
2  | 7m43s  |  445
4  | 4m34s  |  224
8  | 2m54s   |  154

It is interesting, why OpenFOAM-v2306 runs considerably faster that OFv11 ?

[moving-boundaries]

2 x Xeon Gold 6132, 12 x 8GB DDR4-2666, OpenFOAM 2306
(got as a refurbished workstation for $550)

# cores   Wall time (s):
------------------------
1 1019.59
2 478.62
4 210.5
8 113.5
12 85.26
16 72.25
20 65.63
22 63.43
24 62.35
26 61.08
28 60.18

Seems to be consistent with previously reported 2x6136, 2x6140, and 2x6148. All are 2 x 6 channels @ 2666MHz ~ 255 GB/s

BTW, is this a coincidence that perfect scaling stops around 12 threads?

[moving-boundaries]

A few more random CPUs

i7-8750H (laptop), 2 x 8GB 2400MHz

# cores   Wall time (s):
------------------------
1 759.96
2 519.61
4 471.3
6 475.32

Ryzen 5900HX (laptop), 2 x 16GB 3200MHz

# cores   Wall time (s):
------------------------
1 640.32
2 341.75
4 257.75
6 239.89
8 233.84

i7-12700K, 2 x 16GB 3200MHz

# cores   Wall time (s):
------------------------
1 427.68
2 257.55
4 186.73
6 169.95
8 164.91
10 173.77
12 186.5

[Kolan]

2023 Apple 16" MacBook Pro
M3 Max 16 core (4 efficiency, 12 performance)
48 GB of RAM

Running 2306 via Gabriel Gerlero's natively compiled app.

# cores   Wall time (s):
------------------------
16 85.26
12 55.37
10 58.78
8 64.31
6 76.24
4 107.32
2 191.6
1 347.03

Updating this post.

I build v2206 using Alexey Matveichev‘a patch.
https://github.com/mrklein/openfoam-os-x

I ran this on battery while on the bus.

# cores Wall time (s):
------------------------
16 89.72
12 54.46
10 58.52
8 65.13
6 75.8
4 106.51
2 198.54
1 350.02

I’ll run it again when I’m home plugged in and I’ll enable high-power mode.


Here is the result while plugged in for what it's worth.

# cores Wall time (s):
------------------------
16 90.09
12 54.9
10 58.05
8 65.53
6 75.68
4 107.58
2 197.41
1 350.19

[jd210]

Hi Everyone,

Promised some time ago results for the 9454 and I've finally got around to it!

2x AMD Epyc 9454, 24x 16GB 4800MT/s RAM, lightly modified Rocky Linux 8.3, OpenFOAM v2306.

Cores	Wall Time
1	463.17
2	253.62
4	101.15
8	46.88
12	38.1
16	26.09
24	19.35
32	16.48
48	14.36
64	10.74
96	8.94

SMT switched off, bind to core, rank by core & map to numa mpi flags. Numa nodes are set to memory controllers which is probably harming the 8 - 16 core results.

Slightly surprised at the gap to the 9474F but I guess a ~30% cost increase translating to a ~15% performance increase isn't so surprising.

[boredjohn]

Hi,
There are some results for a workstation that my company got in the summer
2XIntel Gold 6430 32 cores, 16*32 DDR5 4400 MHz , Ubuntu V22.04 , OpenFOAM v2212, hyperthreading off .

Cores	Wall Time
16	42.04
32	25.12
48	20.06
56	18.51
60	18.02
64	18.62

We want now to get a new machine and we are leaning towards AMD EPYC 9454, but we are afraid of CPU scaling with mesh size, our typical workflow case may vary from 20 mil to 50 mil cells. Are there any benchmark cases with higher cell count available?

Thank You.

[alexvaleije]

Hi,

I have a 2xEPYC 9454 server availiable. If you find a benchmark case suitable for you that you want me to run, let me know.

Regards,

Originally Posted by boredjohn

Hi,
There are some results for a workstation that my company got in the summer
2XIntel Gold 6430 32 cores, 16*32 DDR5 4400 MHz , Ubuntu V22.04 , OpenFOAM v2212, hyperthreading off .

Cores	Wall Time
16	42.04
32	25.12
48	20.06
56	18.51
60	18.02
64	18.62

We want now to get a new machine and we are leaning towards AMD EPYC 9454, but we are afraid of CPU scaling with mesh size, our typical workflow case may vary from 20 mil to 50 mil cells. Are there any benchmark cases with higher cell count available?

Thank You.

[flotus1]

Everything else about the benchmark case being equal, scaling can't really get worse with higher cell count.
Parallelization overhead for MPI+domain decomposition decreases as cell count increases.
An exception would be if the small test case is small enough to mostly fit into CPU caches, while the larger case does not. But the benchmark here should already be large enough to avoid that.

[ym92]

2x Epyc 9554 (64 cores) with 24x16GB DDR5-4800MHz
Ubuntu 20.04, OF2206

# cores   Wall time (s):
------------------------
16      30.19
32      16.31
64      9.97
96      8.06
127    7.92

Basically no difference compared to the 9474F. Maybe interesting for those using Star-CCM+: I get ~10% higher performance for 9554 compared to 9474F (case ~40M cells, multiphase, DES).

[ERodriguez]

2x Epyc 9354 (32 cores each) with 24x32GB DDR5. no optimization or changes are done in the BIOS / processor.
Ubuntu 22.04, OF2212. I have tried downloading the precompiled release or compiling it by myself with very little impact on the performance.

------------------------
# cores   Wall time (s):
------------------------
1 534.71
2 294.82
4 117.78
6 73.13
8 54.67
12 37.64
16 28.18
24 20.47
32 15.89
40 13.56
48 12.01
56 11.23
64 10.6

If I mirror the benchmar twice to get ~8M mesh elements instead of ~2M mesh elements I obtain:

------------------------
# cores   Wall time (s):
------------------------
2 964.4
4 482.14
6 381.98
8 328.22
12 270.89
16 227.64
24 234.98
32 226.32
40 231.71
48 245.88
56 245.38
64 250.44

Regards

[AlexKaz]

Originally Posted by ERodriguez

2x Epyc 9354 (32 cores each) with 24x32GB DDR5. no optimization or changes are done in the BIOS / processor.
Ubuntu 22.04, OF2212. I have tried downloading the precompiled release or compiling it by myself with very little impact on the performance.

------------------------
# cores   Wall time (s):
------------------------
1 534.71
...
64 10.6

If I mirror the benchmar twice to get ~8M mesh elements instead of ~2M mesh elements I obtain:

------------------------
# cores   Wall time (s):
------------------------
2 964.4
...
64 250.44

Regards

Thanks a lot! Very-very interesting results. It's about 8 times scaling only.

Can you upload your version of benchmark archive or settings change?

[AlexKaz]

I've modified the original test, now it includes double mirrorMesh execution.

LGA4189 2x8352Y ES DDR4 16ch 3200 single rank OFv2112:

# cores Mesh time (s) Wall time (s):
------------------------
1 988.02 3922.69
2 653.37 1910.67
4 377.6 950.52
6 258.99 605.29
8 216.41 484.61
10 190.56 404.16
12 170.16 350.52
16 152.95 293
20 133.89 261.14
24 132.87 240.93
28 127.01 226.25
32 128.22 219.12
36 121.84 209.76
40 123.52 205.7
44 125.45 204.72
48 137.79 201.52

Originally Posted by ERodriguez

2x Epyc 9354 (32 cores each) with 24x32GB DDR5.
If I mirror the benchmar twice to get ~8M mesh elements instead of ~2M mesh elements I obtain

250 seconds Epyc4 DDR5 vs 202 seconds LGA4189 DDR4 on ~8M cells? Very strange result. It needs more tests on other systems. Also it needs to check my archive if it is correct or not.

[Daniel_C]

Here is my new puppy:

2XAMD EPYC 7F52 16 cores, 16*16 DDR4 3200 dual rank , Ubuntu V22.04 , OpenFOAM v2312, hyperthreading off etc.

# cores Wall time (s):
------------------------
1 661.93
2 346.51
4 211.7
6 129.1
8 92.43
10 67.93
12 55.51
18 35.02
22 27.37
24 25.09
32 18.66

[wkernkamp]

Processor: AMD 7800X3D
Motherboard: Gygabyte A620M S2H
Memory: Team T-Force Delta RGB 32GB (2 x 16MB) / 288-Pin DDR5 SDRAM / DDR5 6400 / 40
Power supply: Lenovo 54Y8886 24 Pin 240 W SFF for lenovo ThinkCentre M73

Out of the box:
(Defaults results in DDR5-4800 memory)
Meshing Times:
1 638.65
2 425.97
4 253.48
6 222.81
8 174.33
Flow Calculation:
1 483.78
2 235.57
4 154
6 134.11
8 128.31


Tuned:
1. Upgrade BIOS from F1 to F22b
2. Set XMP/EXPO (memory sets to XMP1 profile DDR5-6400)
3. Set auto bandwidth and latency optimization

Meshing Times:
1 605.93
2 401.26
4 234.99
6 202.85
8 156.21
Flow Calculation:
1 430.53
2 205.56
4 123.38
6 101.59
8 95.44

Todo:
Undervolt

[indy07cz]

2xCPU AMD EPYC Milan 7443, RAM 128 GB (16x8 GB 3200MHz), Ubuntu 22.04, OF 2312

# cores Wall time (s):
------------------------
1 695.35
2 285.25
4 134.44
8 68.76
16 44.21
32 31.45
48 24.12

[wkernkamp]

Originally Posted by AlexKaz

I've modified the original test, now it includes double mirrorMesh execution.

LGA4189 2x8352Y ES DDR4 16ch 3200 single rank OFv2112:

# cores Mesh time (s) Wall time (s):
------------------------
1 988.02 3922.69
2 653.37 1910.67
4 377.6 950.52
6 258.99 605.29
8 216.41 484.61
10 190.56 404.16
12 170.16 350.52
16 152.95 293
20 133.89 261.14
24 132.87 240.93
28 127.01 226.25
32 128.22 219.12
36 121.84 209.76
40 123.52 205.7
44 125.45 204.72
48 137.79 201.52

Ran this on 7800x3d and OpenFOAM v2312. Had to use the original setup, because of some naming changes in OpenFOAM. However, easy to insert the mirror operation in the old template.


Meshing Times:
8 155.05
Flow Calculation:
8 495.51

[Simbelmynë]

Originally Posted by wkernkamp

Processor: AMD 7800X3D
Motherboard: Gygabyte A620M S2H
Memory: Team T-Force Delta RGB 32GB (2 x 16MB) / 288-Pin DDR5 SDRAM / DDR5 6400 / 40
Power supply: Lenovo 54Y8886 24 Pin 240 W SFF for lenovo ThinkCentre M73

Out of the box:
(Defaults results in DDR5-4800 memory)
Meshing Times:
1 638.65
2 425.97
4 253.48
6 222.81
8 174.33
Flow Calculation:
1 483.78
2 235.57
4 154
6 134.11
8 128.31


Tuned:
1. Upgrade BIOS from F1 to F22b
2. Set XMP/EXPO (memory sets to XMP1 profile DDR5-6400)
3. Set auto bandwidth and latency optimization

Meshing Times:
1 605.93
2 401.26
4 234.99
6 202.85
8 156.21
Flow Calculation:
1 430.53
2 205.56
4 123.38
6 101.59
8 95.44

Todo:
Undervolt

I have not kept up with the AM5 platform. Is there a sweet-spot for memory frequency as with the AM4 platform? I.e., the point where latency optimization gives a much higher impact compared to increasing bandwidth? Do you still have to manage the IF?

[wkernkamp]

Originally Posted by Simbelmynë

I have not kept up with the AM5 platform. Is there a sweet-spot for memory frequency as with the AM4 platform? I.e., the point where latency optimization gives a much higher impact compared to increasing bandwidth? Do you still have to manage the IF?

Yes, I have gone down to 6200 MT/s and FCLK at 2067. On the 7800x3d the infinity fabric is the bottleneck. So I have FCLK:UCLK:MCLK 2:3:3. My performance improved to:

Flow Calculation:
1 283.76
2 164.22
4 106.8
6 93.08
8 88.49

The memory controller is very good. People have run memory at 8200. I think my memory could go there. FCLK 2000, UCLK 4000 and MCLK 8000 gives very good 128 GB/s bandwidth. However, the single CCD is limited to ~65 MB/s proportional to FCLK. I am wondering if a 7900X3D solving 8 processes with six on the die with the 2D cache and 2 on the other low cache die will solve in 66 seconds. This would be a well balanced run on a processor that is not well balanced. Each process would have 16MB of L3 cache. There are measured read bandwidths from AIDA64 above 80 GB/s on the 7950X. In theory, it should be able to reach 4x2000x8x2 = 128 GB/s on the infinity fabric when used symmetrically versus the 7800X3D half that number at 2000 FCLK.

For my 7800X3D my only way forward is to try to raise FCLK. I have run 2133 with the 6400 XMP profile. That works sometimes, but mostly the run will not finish due to segmentation faults. I had one run with 8 processors finish in 83 seconds. This confirms what the numbers already say: the FCLK is the limiting factor.


My motherboard was very cheap: $64. It is mATX with just one slot per channel, which is perfect for fast memory. The memory was just over $100 for 32MB total and it does the rated 6400 MT/s out of the box. I haven't tried yet, but I plan to run that memory at 8000, just to see if it will. If that works, I may spend the ~$400 for a 7900X3D here in the US to see if It can solve in 66 seconds. Obviously I don't need two cpus for one computer, so feel free to try and post results here.


I ran openfoam drivaerFastback small and medium with phoronix. My solution times were much better than those for the 7900X3D and 7950X3D (just a single result for each). I did Medium in 1835 seconds and they timed in above 2000. This ought not happen if you do it right, but the benchmark probably runs all cores on both CCDs. That will cause the non-X3D CCD to slow down the other one.

[MangoNrFive]

We finally have our new Workstation, we got quite a good deal for the 9684-X

Processor: 2x AMD EPYC 9684-X
RAM: 24x16GB DDR5 4800MHz
CPU-cooling: 2x XE360-SP5 AiO
OS: Ubuntu 22.04.3
OF: v2112
SMT: off
NPS: 4
Core-bindings etc.: none (just the default benchmark)

I used the benchmark provided by wkernkamp for the 2M-case and mirrored it twice as suggested by ERodriguez for the 8M-case.

Cores	Time [s]	        Speedup	
	Meshing	Simulation      Meshing Simulation
	        2M	8M	        2M	8M
1	795,8	556,5	3019,1	1,0	1,0	1,0
2	506,2	267,7	1417,3	1,6	2,1	2,1
3	346,5	135,9	806,7	2,3	4,1	3,7
4	293,2	108,8	652,0	2,7	5,1	4,6
6	198,2	67,0	350,3	4,0	8,3	8,6
8	161,9	50,1	254,1	4,9	11,1	11,9
12	128,0	35,9	164,8	6,2	15,5	18,3
16	114,2	27,1	125,1	7,0	20,5	24,1
24	93,4	18,6	76,2	8,5	29,9	39,6
32	85,2	14,8	61,1	9,3	37,6	49,4
48	82,2	10,7	44,2	9,7	52,0	68,4
64	71,0	8,6	35,0	11,2	64,6	86,3
72	71,7	7,9	31,8	11,1	70,9	94,9
96	71,8	6,8	27,5	11,1	81,4	109,7
120	73,0	6,3	25,4	10,9	88,1	118,8
128	74,2	6,2	25,3	10,7	90,0	119,3
144	77,4	5,8	24,1	10,3	95,5	125,5
160	80,8	6,1	24,6	9,9	91,4	123,0
168	84,9	5,6	23,5	9,4	98,8	128,4
172	85,5	6,0	24,5	9,3	92,4	123,3
192	89,4	6,3	24,9	8,9	88,9	121,3
192*	89,4	5,6	23,4	8,9	88,9	121,3
* with SMT turned on

Some findings:

• Meshing seems to scale worse. I suppose, this is due to re-balancing and in general more cross-processor communication during the meshing?
• A pattern emerges, where the Core-counts of multiples of 24 (bold) perform very good. I´m pretty sure, this is due to the 24 CCDs (12 per CPU) being balanced
• The scaling is very good up to 24 Cores (the full CCDs and L3-cache is only used after 24 Cores), it is a bit worse up to 72 Cores (I don´t know what is special about 3 Cores per CCD, maybe to do with how the V3-Cache works, or a sweet-spot of some sorts) the scaling drops significantly after that
• The best setup is 168 Cores (7 of 8 Cores of each CCD being used), but with Hyperthreading turned on the full 192 Cores take the lead
• The CPU-temperatures were around 50ºC up to 60ºC in this benchmark, but I´ll upgrade the radiator-fans (the stock-ones are a bit too noisy to my liking) and add some more case-fans