[wkernkamp]

Originally Posted by ships26

Thank you for sharing your results, wkernkamp! It's always great to have a point of comparison.

Did you stick to a single cpu in hyperthreading when running the benchmark with 12 threads?

I don't remember if I turned hyperthreading off or not. I do know that it will schedule 1 thread per cpu even with hyper threading on if you request 12 or less. The performance difference from hyperthreading off is not large, so I typically leave it on. Having two threads per cpu does not speed the calculation up.

[ships26]

Well, it's just that I was expecting the gap between your setup and mine to be larger, due to the advantage of having two processor vs one, which leads to twice the amount of memory channels (2*3) in this case.


However I was wrong. Due to my newbness at this, I got a little confused, as the E5645 allows hyperthreading and I thought it was enabled in this machine. I must've misread the output of lscpu, because I just re-checked it and there are two E5645 processors (and hyperthreading is off), so it all makes sense now! Thanks for helping me clear that up!

[Simbelmynë]

What type of memory do you use? 800 up to 1333 1R vs 2R. This makes a huge difference

[hokhay]

Run the benchmark on AWS EC2 C6g.12xLarge (Graviton 2)
vCPU=48 (all physical cores)
Arm64

OpenFOAM compiled with Gcc

PHP Code:

# cores      Wall time (s):
------------------------
1            902.8
2            468.9
12           91.96
16           74.57
20           66.08
24           60.35
48           48.23 


[steve_b]

Hi all, thank you for prepping this benchmark. I am very new to openfoam user so apologies for stupid questions

I did manage to run the smaller motorbike tutorial which comes with the openfoam installation at

$FOAM_TUTORIALS/incompressible/simpleFoam/motorBike/

However when I try to run this one using openfoam.org v7 I get:

[12] 
[12] 
[12] --> FOAM FATAL ERROR: 
[12] Attempt to return primitive entry ITstream : IOstream.functions.streamLines.seedSampleSet, line 0, IOstream: Version 2.0, format ASCII, line 0, OPENED, GOOD
    primitiveEntry 'seedSampleSet' comprises 
        on line 0 the word 'uniform'
 as a sub-dictionary
[12] 
[12]     From function virtual const Foam::dictionary& Foam::primitiveEntry::dict() const
[12]     in file db/dictionary/primitiveEntry/primitiveEntry.C at line 189.
[12] 
FOAM parallel run aborting

I am trying to run it on a cluster via slurm so I can't use run.sh directly, but I created an sbatch script which performs the same steps (although only for 32 processes, which is one node, rather than a range of nodes).

Any suggestions as to what's wrong appreciated.

[hokhay]

Originally Posted by steve_b

Hi all, thank you for prepping this benchmark. I am very new to openfoam user so apologies for stupid questions

I did manage to run the smaller motorbike tutorial which comes with the openfoam installation at

$FOAM_TUTORIALS/incompressible/simpleFoam/motorBike/

However when I try to run this one using openfoam.org v7 I get:

[12] 
[12] 
[12] --> FOAM FATAL ERROR: 
[12] Attempt to return primitive entry ITstream : IOstream.functions.streamLines.seedSampleSet, line 0, IOstream: Version 2.0, format ASCII, line 0, OPENED, GOOD
    primitiveEntry 'seedSampleSet' comprises 
        on line 0 the word 'uniform'
 as a sub-dictionary
[12] 
[12]     From function virtual const Foam::dictionary& Foam::primitiveEntry::dict() const
[12]     in file db/dictionary/primitiveEntry/primitiveEntry.C at line 189.
[12] 
FOAM parallel run aborting

I am trying to run it on a cluster via slurm so I can't use run.sh directly, but I created an sbatch script which performs the same steps (although only for 32 processes, which is one node, rather than a range of nodes).

Any suggestions as to what's wrong appreciated.

Remove streamlines from controlDict.

#include streamlines

#include wallBoundedStreamlines

[Geon-Hong]

Thanks for the important information for making an HPC system.

My testing environment is as follows.

- CPU: AMD Ryzen Threadripper 3990x
- RAM: 128GB (32GB x 4 / DDR4 / 2,666MHz)
- M/B : TRX40 (Gigabyte TRX40 AORUS Pro Wifi)
- SSD: SAMSUNG 1TB M.2
- OF : OpenFOAM-v2006
(function objects for generating stream lines were deactivated)

And the results are here:

# cores   Wall time (s):
------------------------
1      620.9
2      355.72
4      177.92
8      110.08
16     66
24     66.08
32     62.7
40     63.79
48     63.18
56     63.97
64     63.11

As you can see, the parallel performance was saturated around 16 cores.

Many thanks.

[steve_b]

Originally Posted by hokhay

Remove streamlines from controlDict.

Thank you!

I also had to make a change for etc directory locations:

sed -i -- 's|caseDicts|caseDicts/mesh/generation|' system/meshQualityDict

Here's some preliminary results on a cluster, using all physical cores of 1-16 nodes. Each node has 2x Intel Xeon E5-2683 v4 CPUs. 100Gb IB between nodes. Openfoam v7 compiled with gcc7 + openmpi4.

# cores   Wall time (s):
------------------------
32        78.40
64        43.26
128       21.85
256       13.20
512       9.09

Not sure process placement etc is correct yet so this may not be optimal.

[flotus1]

Neat. Seems like you finally found out where strong scaling for this test case tapers off.

[steve_b]

I see this benchmark pulls "ExecutionTime" out of the output rather than ClockTime. However I'm a bit confused about what this actually means. For other MPI codes, if you were running on e.g. 4x processes you would expect to see a "cpu time" which was probably a bit less than 4x the (wall)clock time. I.e. a "total cpu time". However on these openfoam runs the ExecutionTime and ClockTime are pretty similar, despite using lots of processes. From some googling this seems like this is expected if openfoam is running efficiently, but can anyone tell me how ExecutionTime is actually calculated?

[Fabian2602]

Please take care when using OpenFOAM 8. surfacesFeatureExtract is not existing anymore. Instead you can use surfaceFeatures.

[otaolafr]

Hello, looking for some help/recommendations,
I have 2 main questions:
we have two small clusters to available and we would like to compare the performance for OF. so from searching i finish in this thread, I have never done any benchmark for OF (or any CFD software...). I would not say that I am experience OF user either but at least an advance beginner/ intermediate user. in any case, i know how to run cases and save the log files, but to compare the performance of the two of them i have seen post in this thread where people show speeds of each CPU a total time. the only time I know how to take is the total time for the snappy in the log file or how much time each iteration of a solver takes.
1. how can i save/ see this times? normally I use decomposePar with decomposeParDict and runParallel command for example "runParallel snappyHexMesh -overwrite" (and the should i run my simulation differently?or is there any other thing that i should do?)

2.
from what i saw in the files, there is only "Allmesh"file , the benchmark is only the meshing? or there is more?

thanks!
also an extra question, that could help the benchmark, what should I save as information about the cluster and results data to be functional for other people to compare?
best regards

[flotus1]

The bench template in the initial post extracts timing information of the solver run from the log files. You can either use that, of just check where the timing information is coming from in the script. This thread is getting a bit dated, so there will be some adjustments necessary to use this benchmarks with newer versions of OF. What exactly has been discussed a few times already.
So no, the benchmark is not only for meshing.

[steve_b]

The reason it looks like it's only doing meshing is because the benchmark script pulls files from the installations tutorial directory.

Confused me too to start with :-)

[otaolafr]

hello everbody,
so to resume my experience to run the benchmarc now:

• if your version is fondation in the \basecase\system\controlDict the line 50 should be erased/comment (#include "wallBoundedStreamLines") wallBoundedStreamLines it is not a function object (at least with that name in this vr, in the ESI version of OF yes, so it might work (not tested)
• streamLines does not work from the start but the file can be copied from incompresible/simpleFOAM/motorBike/system/streamLines
• the meshQualityDict should be modified and change line 18 from (#includeEtc "caseDicts/meshQualityDict") to (#includeEtc "caseDicts/mesh/generation/meshQualityDict") with out the ()

also another thing to notice, is that the script shows only the run time of the simpleFoam solver and not the meshing or at least the real/user/system times were not the same that the snappyHexMesh log file said. if anyone wants to see this execution time they are in each "run_*/log.snappyHexMesh" file
and to run the script we use ./run.sh from the terminal
hope it helps someone like me that it was a little bit lost ^^

[Shayor]

Can anyone please run a test for the Ryzen 9 3900X CPU for me and tell me how it scales if all 12 cores are used.Please.Please help me out.I am going to buy it 3 weeks from now but am going to rethink if it's a bad choice.Going to use ANSYS and Code Saturne.No problem with licenses for core count.Going to use 32GB 3200Mhz memory for now(going to upgrade to 64 GB if ALLAH wishes).

Note:I have to go with mainstream CPUs.Server CPUs are not available here and the ones available are unusually high maybe due to country's policy for importing these stuffs.Sad but true.

[flotus1]

We already have those results, plus a few other Zen2 based Ryzen CPUs. Use the search function

[Shayor]

Originally Posted by blackcatxiii

Hi everyone

I have collected the benchmark data posted in this thread in an Excel spreadsheet and plotted out walltime as well as speed up for comparison.

I hope it is helpful for those who plan to build a new PC for OpenFOAM.

Please mention the configuration if you can.It would help a lot.Thank you.

[ships26]

If anyone is on a tight budget (~350ish euros) and looking at what a Ryzen APU can do besides being an everyday desktop, here's a Ryzen 3 3200G @stock with 2x8GB 3200MHz 16-18-18-38 memory (F4-3200C16D-16GVGB). This is out of the box performance, only XMP was enabled.

It was ran on Ubuntu 20.04 and OpenFoam 7:

# cores   Wall time (s):
------------------------
1          809.1
2          482.12
4          333.96

And the meshing:

# cores   Wall time (s):
 ------------------------
1          20m05s                                   
2          13m50s
4          9m21s

Maybe some latency improvements could give this little guy some extra juice?

[steve_b]

Hi,

I've run this benchmark before (results upthread) but I'm trying to repeat it on a new system. So I didn't expect any problems, but I'm getting the following in the error log:

[170] #0  Foam::error::printStack(Foam::Ostream&) addr2line failed

The end of the output log looks like this, so it's started running ok I think:

Time = 9

smoothSolver:  Solving for Ux, Initial residual = 0.604931, Final residual = 0.0517055, No Iterations 93
smoothSolver:  Solving for Uy, Initial residual = 0.667747, Final residual = 0.0647787, No Iterations 85
smoothSolver:  Solving for Uz, Initial residual = 0.775275, Final residual = 0.0712619, No Iterations 87
GAMG:  Solving for p, Initial residual = 1, Final residual = 2.51329e+10, No Iterations 1000
time step continuity errors : sum local = 2.62033e+45, global = 4.22415e+41, cumulative = 4.22415e+41
smoothSolver:  Solving for omega, Initial residual = 6.48317e-07, Final residual = 3.65153e-08, No Iterations 2
smoothSolver:  Solving for k, Initial residual = 0.661027, Final residual = 2.48442e-05, No Iterations 2
ExecutionTime = 46.78 s  ClockTime = 47 s

forceCoeffs forceCoeffs1 write:
    Cm    = 1.51746e+60
    Cd    = 6.88127e+59
    Cl    = 8.6879e+60
    Cl(f) = 5.86141e+60
    Cl(r) = 2.8265e+60

Time = 10

This is with openfoam-org-7 using gcc-9.1.0 on openmpi 3.1.6.

Any suggestions very gratefully received, happy to post sbatch file, log files etc if that helps!