[julieng]

Hello

2 x Intel E5-2678 v3 @ 2.50GHz with SZMZ X99 Z8 motherboard from Aliexpress default settings

128 GB (8x16) DDR4 2133 MHz
openFoam 8
Ubuntu 20.04

Same test with Hyperthreading disabled

# cores Clock time (s):
------------------------
1 622
2 339
4 161
6 118
10 82
12 70
14 64
16 58

18 56
20 53
22 51

24 53

[victoryv]

What is the best place to get these components ? What do you suggest - build it yourself or get from a system integrator (any suggestions ?) ?

Haven’t built a PC for 15 years now. Thinking of building one with $1500-$2000 based on this thread
Amd epyc 7302/7532 with 64/96 Gb RAM

[Tobermory]

I went through a system integrator to build my dual Epyc 7302 rig (and am REALLY happy with it). I am UK-based, though, so there's no point me recommending them (!).

I don't believe that the build is particularly special/difficult, but I wasn't sure that I could do it to a high standard, and was a little lazy. As a note of caution - the mobo (Supermicro H11DSi-NT) took a long time to get hold of, for some reason - maybe supplies have eased a little now. Good luck!

[fanta]

HI @ all, 4 weeks ago i received a new PC in my company, built for OpenFOAM.
Dual Epyc 7402, 128 GB RAM, Supermicro D11DSI-NT, AMD Graphicscard etc.
I would like to participate at the Benchmark test, but i am using OpenFOAM.com v2012 and the Mastercase for the Benchmark doesn't run with it properly. Are there plans to do an Update for compatibility with OF.com in the future? Otherwise i would have to setup .org (v8). .....

[wkernkamp]

fanta



What was the problem running the benchmark?

[fanta]

Hey,
right at the moment i am very busy in my job and i cannot run another benchmark on that machine, cause it's full with a queue of OpenFOAM studies.
I think it will take 1-2 weeks until the queue is up and i can run the benchmark again. I remember that the benchmark test didn't complete cause of errors. But unfortunately i didn't make a log, but i will when i run it again. I will post my errors here but i will take 1-2 weeks. After 5 weeks i am very happy with that machine, it's fast, it's free of errors and our solver now is capable of running 10-12 studies in 24 hours - our old Beowolf cluster of two dual Intel Xeon made 2-4 of that studies in 24 hours. It's great.

[danbence]

Here are some openfoam benchmarks for Epyc Milan.


https://www.phoronix.com/scan.php?pa...nux-perf&num=8


Looks like a 27% increase compared to the fastest Rome tested.

[Simbelmynë]

Originally Posted by danbence

Here are some openfoam benchmarks for Epyc Milan.


https://www.phoronix.com/scan.php?pa...nux-perf&num=8


Looks like a 27% increase compared to the fastest Rome tested.

Nice!



Regardless if I have missed OF in the Phoronix test suite before or if this is a new addition - this is really good considering the rather extensive and recurring testing Michael Larabel performs under Linux.

[RolandS]

Hi Benchmarkers.

New to this forum. And rather a newby to OpenFOAM.

Wanted to improve simulation performance with a faster computer. I selected this conguration for my limited purse:
Refurbished, HP Z840 2x Xeon 12C E5-2678 V3, 2.50Ghz, 128GB (16x8GB) DDR4, Quadro M2000 4GB.

Here is my benchmarking result using your script on appr. 2 million cells:

# cores Wall time (s):
------------------------
6 228.279
12 181.426
24 167.931
48 220.492

Remarkably, the 24 operation is substantially faster than the 48 core exercise.
In fact, the same trend is observed for another model of 6 million cells:

# cores Wall time (s):
------------------------
24 11
48 15

Do you have a clue of what might be a reason of this (for me) unexpected behaviour? (Note 48 cores is slower than 12 cores)

Greetz

Roland

[flotus1]

One reason might be that you don't really have 48 physical CPU cores available. You have 2 CPUs with 12 cores each. Using SMT is usually advised against for this type of workload, i.e. MPI with domain decomposition. Most of the time, there is no performance increase, and in some cases performance gets even worse.
Overall, your numbers look rather slow compared to similar hardware already posted here. Either you did something different compared to the "standard" benchmark, or it might be worth digging deeper why your system is under-performing.

[RolandS]

Thanks Flotus1 for yr suggestion.


Following your suggestion, I made a number of tests with Hyperthreading on and off, to find the difference. These are the results:


Hyperthreading On

# cores Wall time (s):

6 233
12 157
18 175
24 165
30 162
36 165
42 209
48 226

Hyperthreading Off

# cores Wall time (s):
6 228
10 170
14 168
18 182
22 206
24 217

Hyperthreading does not appear to make much difference. The number of activated cores does, though.

At Core # = 12, the Hyper = On is fastest.

If you think this can go faster, would could you give suggestions of which options to try?

Best regards.

[RolandS]

Thanks Flotus1,



I followed yr suggestions, and found the following:


Multithreading = On
# cores Wall time (s):
------------------------
6 232.656
12 156.989
18 175.16
24 165.161
30 161.755
36 164.739
42 209.001
48 225.97

Multithreading = Off
# cores Wall time (s):
------------------------
6 228.016
10 169.817
14 168.035
18 181.847
22 205.908
24 216.563



Further to the above:


We might compare the reported performance with that of Julieng (OpenFOAM benchmarks on various hardware), with the same chipset.

My system performs 3 x more lazy.

This describes my current system:

HP Z840 2x Xeon 12C E5-2678 V3, 2.50Ghz, 128GB (16x8GB) DDR4, 256GB SSD + 3TB HDD/DVDRW, Quadro M2000 4GB, Win 10 Pro

The OpenFOAM-v2006 windows version has been cross-compiled in OpenSUSE environment using mingw cross-compiler. (as prepared for the FreeCAD windows software).

Questions:
1/ Any specific BIOS settings to be adapted?
2/ Or, work directly under Linux, rather than under Win10/mingw?

Best regards

Roland

[gpouliasis]

Originally Posted by RolandS

Thanks Flotus1,



I followed yr suggestions, and found the following:


Multithreading = On
# cores Wall time (s):
------------------------
6 232.656
12 156.989
18 175.16
24 165.161
30 161.755
36 164.739
42 209.001
48 225.97

Multithreading = Off
# cores Wall time (s):
------------------------
6 228.016
10 169.817
14 168.035
18 181.847
22 205.908
24 216.563



Further to the above:


We might compare the reported performance with that of Julieng (OpenFOAM benchmarks on various hardware), with the same chipset.

My system performs 3 x more lazy.

This describes my current system:

HP Z840 2x Xeon 12C E5-2678 V3, 2.50Ghz, 128GB (16x8GB) DDR4, 256GB SSD + 3TB HDD/DVDRW, Quadro M2000 4GB, Win 10 Pro

The OpenFOAM-v2006 windows version has been cross-compiled in OpenSUSE environment using mingw cross-compiler. (as prepared for the FreeCAD windows software).

Questions:
1/ Any specific BIOS settings to be adapted?
2/ Or, work directly under Linux, rather than under Win10/mingw?

Best regards

Roland

Hi Roland,

while I am not an expert, I have been told from people that know much more than me about parallel computing that using a Windows Subsystem is very suboptimal. I am not sure if it can cause such a difference. I would definetely check it out. But first I would make sure that all the memory channels are working properly. You can check if all memory sticks are present in the bios. Run a few memory benchmarks as well.

[wkernkamp]

RolandS,


Your best result should be below 100 seconds, probably for 20 or 22 cores with HT off.


One easy way to run is using docker. I do it that way using linux and see very good performance. Not sure if windows docker does as well, but worth a try. I think you might have some degradation but still below 100 seconds. I agree with the above remark that you should check memory operations. For good results, you should have 4 channel memory on both processors. They should be numa aware so that cores get local memory assigned. Are the dimms in the correct slots?



Will

[wkernkamp]

Originally Posted by RolandS

Thanks Flotus1,



I followed yr suggestions, and found the following:


Multithreading = On
# cores Wall time (s):
------------------------
6 232.656
12 156.989
18 175.16
24 165.161
30 161.755
36 164.739
42 209.001
48 225.97

Multithreading = Off
# cores Wall time (s):
------------------------
6 228.016
10 169.817
14 168.035
18 181.847
22 205.908
24 216.563



Further to the above:


We might compare the reported performance with that of Julieng (OpenFOAM benchmarks on various hardware), with the same chipset.

My system performs 3 x more lazy.

This describes my current system:

HP Z840 2x Xeon 12C E5-2678 V3, 2.50Ghz, 128GB (16x8GB) DDR4, 256GB SSD + 3TB HDD/DVDRW, Quadro M2000 4GB, Win 10 Pro

The OpenFOAM-v2006 windows version has been cross-compiled in OpenSUSE environment using mingw cross-compiler. (as prepared for the FreeCAD windows software).

Questions:
1/ Any specific BIOS settings to be adapted?
2/ Or, work directly under Linux, rather than under Win10/mingw?

Best regards

Roland

Julieng is running a problem that is ~ 8x smaller, but for longer: 500 instead of 100 iterations (the "motorBike" tutorial.) My estimate is that he would be 25% slower if he ran the benchmark.

[wkernkamp]

The e5-2678 v3 turbos at 3.3 GHz and so does the e5-2695 v4. Assuming the same 1084 seconds for the 1 core result, we get the following speed-up factor comparison. The result at 6 cores is just a little low. Looks like you are operating on just four memory channels and not eight.



cores E5-2678 v3 E5-2695 v3
1 1 1
6 4.75 5.73
10 6.37 9.03
14 6.44 11.94
18 5.95 14.02
22 5.25 15.36
24 4.99 15.26


If you fix this, I think your best run time will be about 80 seconds.

[RolandS]

Thanks Will, I will be checking out yr suggestions. Shall report back asap, but it may take some time.


Best regards

[RolandS]

Thanks George, checking yr suggestions ...
Best regards

[RolandS]

Originally Posted by wkernkamp

........TLooks like you are operating on just four memory channels and not eight.
...............
If you fix this, I think your best run time will be about 80 seconds.

Hi Will, (and George).

I tried to find out about your suggestion that perhaps only 1x4 in stead of 2x4 channels are open between the CPUs and the Memories. For this to analyse, I ran memory check in the UEFI. After 30 minutes testing it gave "Passed". Not very informative.


Would you know a way of testing whether the 2 CPUs communicate each via 4 channels?


I tested another setting selectable in the BIOS, i.e. Snoop Configuration: Home Snoop, in stead of Early Swap with Hyper-Threading = On!! That gave the fastest results so far for 30 cores (for the benchmarking tool of this forum thread):
# cores Wall time (s):
------------------------
24 151.412
30 118.304
36 161.392
42 173.762
48 231.174
That is 0.85 iter/s.


One possible next test seems to install linux as dual boot option, to test whether that works faster.


Greetz


Roland

[wkernkamp]

Originally Posted by RolandS

Hi Will, (and George).

I tried to find out about your suggestion that perhaps only 1x4 in stead of 2x4 channels are open between the CPUs and the Memories. For this to analyse, I ran memory check in the UEFI. After 30 minutes testing it gave "Passed". Not very informative.


Would you know a way of testing whether the 2 CPUs communicate each via 4 channels?

The windows program CPU-Z (free) will tell you the memory configuration under the memory tab. I think there is something big wrong in your memory set-up. It is not normal that your best result happens for 30 cores. Did that run complete normally?


Will