[linuxguy123]

I have a question.

How do the various hardware setups rate if we change from this benchmark simulation to one with more cells or if we change the simulation from simplefoam to LES ?

I know larger models will take more processing time and especially LES, but are these simulations as memory bandwidth sensitive or do they become more processor speed sensitive ?

Phrased another way: are some simulations more processor power sensitive than others ? Or are they all memory bandwidth limited ?

Thanks

[wkernkamp]

I think the memory bandwidth will still dominate the results. After all, the vectors are much larger than the caches. So to perform an iteration, it is always necessary to bring in the data for each bunch of cells chunk by chunk, irrespective of the type of solution.

[ErikAdr]

I'm new to OpenFOAM but not to cfd. I don't want to use OpenFOAM, except for runing this testcase. Is there a tutorial in this very large thread about what to install, where to fine the testcase, and how to run it? I ask because I would like to test a new system using DDR5 memory, but I can't defend spending a lot of time for this.



So far I have installed 'openfoam2112-default' on fedora 35. The foamInstallationTest fails with '2 fatal errors', and then it adds 'Critical systems ok'.Can I run the testcase with the pre-compiled version, and how?

[Simbelmynë]

Originally Posted by ErikAdr

I'm new to OpenFOAM but not to cfd. I don't want to use OpenFOAM, except for runing this testcase. Is there a tutorial in this very large thread about what to install, where to fine the testcase, and how to run it? I ask because I would like to test a new system using DDR5 memory, but I can't defend spending a lot of time for this.



So far I have installed 'openfoam2112-default' on fedora 35. The foamInstallationTest fails with '2 fatal errors', and then it adds 'Critical systems ok'.Can I run the testcase with the pre-compiled version, and how?

Not sure what type of tutorial you need. The instructions how to install OpenFOAM on openfoam.org or openfoam.com are very informative and covers all your needs unless you are also completely new to Linux.


If you are new to Linux, then the easiest option is to install Ubuntu 20.04 and use the repository from openfoam.org.


This benchmark was constructed for OpenFOAM V6 (V5?), we are soon on OpenFOAM V10 so there are a few things that you need to change in order for it to run (even on V6 you need to comment out a few lines to avoid errors).

[flotus1]

If someone wants to give me an up-to-date list of things that need to be changed, in order to run the benchmark with more recent OF versions, I can edit that into the first post.
I think that would be a good idea by now.

[linuxguy123]

Originally Posted by flotus1

If someone wants to give me an up-to-date list of things that need to be changed, in order to run the benchmark with more recent OF versions, I can edit that into the first post.
I think that would be a good idea by now.

I started the attached document to better understand various computer systems in this thread. Feel free to use it as you may. I might work on it more in the near future.

[Simbelmynë]

Originally Posted by flotus1

If someone wants to give me an up-to-date list of things that need to be changed, in order to run the benchmark with more recent OF versions, I can edit that into the first post.
I think that would be a good idea by now.

Here is a modified version that should be good to go with OF V9.

[linuxguy123]

Originally Posted by ErikAdr

I'm new to OpenFOAM but not to cfd. I don't want to use OpenFOAM, except for runing this testcase. Is there a tutorial in this very large thread about what to install, where to fine the testcase, and how to run it? I ask because I would like to test a new system using DDR5 memory, but I can't defend spending a lot of time for this.

I'm hoping you'll share your results with us in this thread.

So far I have installed 'openfoam2112-default' on fedora 35. The foamInstallationTest fails with '2 fatal errors', and then it adds 'Critical systems ok'.Can I run the testcase with the pre-compiled version, and how?

Here are the steps I used:

1) install openfoam-default from copr. This installed openFOAM 2112 on my computers. You need to install openfoam-default because that installs the tutorials. The benchmark copies the motorbike tutorial files to use in the benchmark.

2) add the following line to bashrc for the user(s) that are going to run openfoam:
source /usr/lib/openfoam/openfoam2112/etc/bashrc.

Restart your console.

2b) Test that everything works by running openFOAM. It should start the openfoam command interpreter.

$ openfoam 
openfoam = /usr/lib/openfoam/openfoam2112 

 * Using:     OpenFOAM-2112 (2112) - visit www.openfoam.com 
 * Build:     _14aeaf8d-20211220 
 * Arch:      label=32;scalar=64 
 * Platform:  linux64GccDPInt32Opt (mpi=sys-openmpi) 

OpenFOAM shell session - use 'exit' to quit 

openfoam2112:~/openfoam/OF-Benchmark-template-master/ 
me$ exit 
exit

3) Make a new directory to run the case in.

4) Unpack the bench_template.zip provided in the post above.

5) Edit run.sh to include the number of cores you want to use, if your machine has more than 24 cores.

 
for i in 1 2 4 6 8 12 16 20 24;do

You need to do this in 3 places: #prepare cases, # Run Cases and #Extract Times.

6) Run the benchmark: ./run.sh

To get the best representation of the machine speed, I log out of any graphical UI sessions I am in and run this from a text session command line. I use a ssh session to start it if I'm running it on a server. I run openFOAM directly on the hardware, not in a container.

7) if you want to run it again, remove the generated run folders with $rm -rf run_* and run it again.

[linuxguy123]

My experience and findings on openFOAM hardware selection.

I've spend a considerable amount of time and effort researching and testing hardware upon which to run openFOAM. Some of these points might be old news to people on this site, but they weren't to me, thus I'll share them.
I'm not an expert on openFOAM or openFOAM hardware. I'm just sharing what I learned in order to lessen the learning curve for others who might be just starting down this path.

1) The openFOAM benchmark used in this thread is very memory bandwidth intensive. The speed at which it executes is highly dependent upon 2 factors: 1) the number of memory channels (RAM) a processor has and 2) the speed at which the RAM operates.
You can see this effect take hold when the number of processing cores exceeds 3 per memory channel because past 3 cores per memory channel the processing time doesn't decrease a lot. The 3 cores per channel is almost a universal rule across hardware families as the speed of a processor (MFLOP/sec) is almost always paired with memory of a similar performance level. (Faster processors are paired with faster memory.)

2) Other openFOAM benchmarks are slightly less dependent on memory bandwidth and need more processing power relative to memory bandwidth. The benchmark used at https://openbenchmarking.org/test/pts/openfoam is an example of this. While it also uses the motorbike tutorial, it takes more processing power to reach bandwidth saturation. However, the limiting factor in overall performance is still memory bandwidth.
Whereas this benchmark tends to saturate memory bandwidth at 3 cores per memory channel, other openFOAM simulations might saturate at 4 cores per memory channel.

3) Once the processing power exceeds the memory bandwidth of the platform, increases in processing power (more cores, faster CPU clock speed, etc.) make almost no difference in overall run times. The path to decreasing run time in this situation is to increase the memory bandwidth by using faster memory and more channels of it.

4) Most openFOAM simulations need a relatively small amount of RAM. For example, on my Linux machine, the running of this benchmark takes 6GB of RAM.
Simulation time will increase dramatically as soon as the entire simulation variable set cannot be held in RAM and hard drive swapping starts to occur.

5) When selecting hardware, there are 2 situations to consider.
The first is when you are running large/complicated simulations with really long run times. Large/complicated simulations with long run times cannot be split between multiple machines easily. The best solution to this situation is a single machine with sufficient CPU processing power (GFLOPs) and the highest memory bandwidth you can get.
The second is when you are running lots of smaller simulations with reasonable run times, but need to run lots of them. In this situation the work load can easily be split among several smaller machines, which can run simulations simultaneously, because they are independent.
The reason this distinction is important is because cost increases non linearly with computer capability. Doubling the capability of a single computer may 3x or 4x the cost. Doubling the capability of a single computer will cost 2x to buy a second computer. Or maybe less if quantity discounts occur.
The upside of this is that if you are providing computing resources to a group of people running independent simulations, it is probably more cost effective to provide more machines to run on rather than to provide a single fast machine in order to decrease run times and thus waiting for use of the machine.

5b) It is almost always more cost efficient to run dual processors on a single board than it is to upgrade to a faster single processor setup. Why ? Dual processors have twice the number of memory channels. And the cost of 2 of a processor is less than the cost of a processor that is twice as fast There are exceptions to this rule, so apply it carefully.

6) Users need to be taught that investigations/simulations can be done incrementally. The first step might be a 2D RANS. Next step might be 3D RANS. Final step might be 3D LES. And so on.
The first step, and every step thereafter, doesn't need to be a full model 3D LES that takes 2 days to run and returns data that reflects 14 different design elements into one result. I call those "big bang" simulations. Taking an agile approach to simulation can pay huge dividends in learning and in making the most of simulation processing power.

7) Fine meshes can take a long time to generate ! Most simulations will involve first generating the mesh and then running the solver. These are two independent tasks. As explained above, rather than trying to make a simulation run fast on one fast computer, it is often more cost efficient to use 2 computers running tasks simultaneously. So User2 can use computer 2 to run a mesh while User1 runs his simulation on computer 1, rather than User1 waiting for User2 to be done both meshing and solving his simulation on a faster single computer.

8) Don't allow users to use the meshing/solving computer to view the results. Any workstation with a decent graphics card can review results. Users don't need to use the meshing and solver computers for viewing results. To enforce this, I recommend making the meshing and solving computers headless. Do not make the meshing and solving computer(s) someone's personal workstation.

7) We are entering a golden age for cheap, fast simulation hardware.
AMD's release of Naples (7001) EPYC processors, with 8 DDR4 memory channels, dramatically changed the openFOAM simulation landscape. Those processors are now turning up in the used market for reasonable prices. Newer and better processors will continue to show up at better price/performance ratios as data centers retire their Naples (and newer) processors in favor of newer, faster processors.
And recently it seems as though Intel is becoming serious about competing with AMD with its new line of server processors. This might result in a flood of cheap used Intel processors as well as Intel based data centers have been putting off upgrades for the last few years.
Buyer beware: the used market is currently flooded with older, slower used Xeon server equipment, at very attractive pricing. However, the performance of many of these systems is very poor due to low memory bandwidth - only 6 memory channels and very low speed RAM. Like 1666MHz DDR3, for example. Sometimes worse.
I'm not saying that Xeon based systems can't be cost effective for simulation workloads. What I am saying is you really have to watch what you are getting, both of in terms of memory bandwidth and processing power. In my recent experience, it is very, very hard to beat a used 24 or 32 core Naples processor for price/performance.

8) My quick and dirty method of evaluating the performance potential of a computer for CFD work is to calculate memory bandwidth by multiply the number of memory channels by the speed of the memory. For example:
Ryzen: 2 memory channels x 3600MT/s = 7200 system MT/s
Threadripper: 4 memory channels x 3600MT/s = 14,400 system MT/s
Threadripper Pro: 8 channels x 3600MT/s = 28,800 system MT/s
EPYC Naples: 8 x 2,666 = 21,328
Dual EPYC Naples: 16 channels x 2,666 = 42, 656
EPYC Rome: 8 x 3200 = 25,600
Dual EPYC Rome: 16 x 3200 = 51,200
Milan is even faster, but out of my price range.
Older Xeons: 6 channels x 1,333 MT/s = 7,998
Dual older Xeons: 12 channels x 1,333 = 15,966
Newer Xeons 6 channels x 2,133 = 12,798
Dual newer Xeons 12 channels x 2,133 = 25,596 system MT/s.
This is only an approximation. Memory latency and rank play a role, but only about 5% or so. Generally it is not worth buying lower latency or higher rank memory to speed up a system.

The other aspect is having enough CPU power to use all the memory bandwidth. For this I rely on Geekbench5. Although some Geekbench benchmarks do use a lot of memory bandwidth, most of them don't. And thus the Geekbench5 score is a good reflection of raw CPU processing power. www.geekbench.com
As far as I can tell, a multicore Geekbench5 score of about 1,500 pnts per memory channel is sufficient to saturate a 2,666 memory channel. For example, this predicts a Naples 8 channel system will be bandwidth limited with a processor with a multi core Geekbench score of about 8x1500 = 12,000 pnts.
In reality, an EPYC 7601 has a GB5 multi core score of 18,000 or so for 32 cores. A 7601 will saturate the memory bandwidth with about 24 cores or about 13,500 points. It isn't a perfect rule of thumb, but it is close.

I could say more, but I'll stop there. I hope this helps.

[wkernkamp]

Originally Posted by linuxguy123

My experience and findings on openFOAM hardware selection.



5) When selecting hardware, there are 2 situations to consider.
The first is when you are running large/complicated simulations with really long run times. Large/complicated simulations with long run times cannot be split between multiple machines easily.

Some good news: Openfoam scales very well on clusters. If you have just a few machines, it works well with 1Gb ethernet even. With infiniband you can create a large cluster with excellent scaling. So in the calculation you add all the memory channels in the cluster and multiply by the GT/s to get your approximate result. There are some cheap machines with cheap slow memory that may be attractive for someone's application.



Symberle showed his result in this thread. I had the same experience.

[linuxguy123]

Originally Posted by wkernkamp

Some good news: Openfoam scales very well on clusters. If you have just a few machines, it works well with 1Gb ethernet even. With infiniband you can create a large cluster with excellent scaling. So in the calculation you add all the memory channels in the cluster and multiply by the GT/s to get your approximate result. There are some cheap machines with cheap slow memory that may be attractive for someone's application.

Symberle showed his result in this thread. I had the same experience.

I'm really glad to hear this. Which post in which thread ?

Could you point me to documentation on setting up such a system ?

[ErikAdr]

[QUOTE=linuxguy123;822372]I'm hoping you'll share your results with us in this thread.

Here are the steps I used:

1) install openfoam-default from copr. This installed openFOAM 2112 on my computers. You need to install openfoam-default because that installs the tutorials. The benchmark copies the motorbike tutorial files to use in the benchmark.

2) add the following line to bashrc for the user(s) that are going to run openfoam:
source /usr/lib/openfoam/openfoam2112/etc/bashrc.
.......


Thanks to Simbelmynë and linuxguy123 for helping, and yes I will provide the results if I obtain any.

I followed the steps and it runs, but the results are not realistic. For one core the wall time is less than 10 s, and I would expect at least 400 s. To run 'openFoam' I had to write 'openFoam2112', but the results where the same as for linuxguy123.

In i.e. run_1/log.snappyHexMesh there is a FATAL io error:
Cannot open etc file "caseDicts/mesh/generation/meshQualityDict" while reading dictionary "system/snappeHexMeshDict.meshQualitycontrols"

Any ideas what to do?

[saeedg]

Hey there,

By any chance, has anyone tried to import fluent results to paraview? I've been trying to do that, but some variables are lost like species properties.

Appreciate it if anyone can enlighten me)

Saeed

[wkernkamp]

Originally Posted by linuxguy123

I'm really glad to hear this. Which post in which thread ?

Could you point me to documentation on setting up such a system ?

In addition to what has already been said, I offer the following on fundamentals:


1. Get openfoam running on all your nodes for the same user(s).


2. Get ssh login with key pair configured between all your nodes. So if you type ssh 192.168.144.2 from node 192.168.144.1 it works and no passwd needed. mpi needs this to start commands on other nodes.



3a. As a cluster, the nodes will need to work from a common file system. On a big cluster this is critical, but for just a few nodes, you can export run directories from your graphics capable machine using nfs or sshfs. Anything that causes the same directories to show up in the same place in the file system will work. (It is possible to specify different directories per node in mpirun. Some confused souls may do it that way, but I suggest you don't for simplicity's sake.)


3b. You can also share your openfoam source and compile directories in the same way so you can maintain software in just one location. Set this up before step 1.



4. Check the options for your mpirun command. One way to declare your cluster is with a "hostfile". It contains ip addresses (or dns names) of the nodes and the number of processes to run on each. Add the declaration of the hostfile in the mpirun command that is already used for a parallel run. Also increase -np to the total for the cluster. It should look like this:

% cat hostfile
myHost1 slots=8
192.168.144.2 slots=12


mpirun -np 20 -hostfile hostfile -wdir path/to/wdir simpleFoam

mpirun has a lot of options. However, the defaults are usually good. This hostfile will work as long as ssh myHost1 and ssh 192.168.144.2 work.


The number of slots does not have to be equal. This allows you to use dissimilar machines. In addition, you could offload a slower machine. You can see how your nodes compare by running the benchmark on each node at step 1. Lets say you have a machine with 16 cores. The benchmark shows 1000 sec on 1 cores and 100 sec on 16 cores. The latter is faster, but each core does only 1/16th of the work in the single core case. Correcting for that, the equivalent per core completion time for slots=16 is 100x16 = 1600 sec. Now select the number of slots on each node in such a way that for each node the equivalent per core completion time is the same (or close). The slowest machine (highest equivalent per core value) is going to determine the cluster speed! (because it causes the others to have to wait).

[linuxguy123]

Originally Posted by ErikAdr

I followed the steps and it runs, but the results are not realistic. For one core the wall time is less than 10 s, and I would expect at least 400 s. To run 'openFoam' I had to write 'openFoam2112', but the results where the same as for linuxguy123.

Did you append the source.... command to your .bashrc file ? And log out and log back in ? At that point you should be able to run $openfoam from the command line, anywhere in your file system.

In i.e. run_1/log.snappyHexMesh there is a FATAL io error:

Cannot open etc file "caseDicts/mesh/generation/meshQualityDict" while reading dictionary "system/snappeHexMeshDict.meshQualitycontrols"

Any ideas what to do?

See here: SnappyHexMesh "Cannot open etc file"

[ErikAdr]

I have edited one file named meshQualityDict, but there is about 50 more. The one I have changed is /usr/lib/openfoam/openfoam2112/tutorials/incompressible/simpleFoam/motorBike/system/meshQualityDict. Is this the correct file to change?

No change in results though.



In log file run_1/log.surfaceFeatures I get:
/usr/lib/openfoam/openfoam2112/bin/tools/RunFunctions: line 271: surfaceFeatures: command not found


Is this error related to the other? Ideas?

[masofshad]

Originally Posted by linuxguy123

Ryzen: 2 memory channels x 3600MT/s = 7200 system MT/s
Threadripper: 4 memory channels x 3600MT/s = 14,400 system MT/s
Threadripper Pro: 8 channels x 3600MT/s = 28,800 system MT/s
EPYC Naples: 8 x 2,666 = 21,328
Dual EPYC Naples: 16 channels x 2,666 = 42, 656
EPYC Rome: 8 x 3200 = 25,600
Dual EPYC Rome: 16 x 3200 = 51,200
Milan is even faster, but out of my price range.
Older Xeons: 6 channels x 1,333 MT/s = 7,998
Dual older Xeons: 12 channels x 1,333 = 15,966
Newer Xeons 6 channels x 2,133 = 12,798
Dual newer Xeons 12 channels x 2,133 = 25,596 system MT/s.

Thanks Linuxguy123 for your great summary!

Just want to point out that DDR5 is out now and there are a couple of specs. One spec released this year is for AMD's new Zen3+ architecture supporting LPDDR5-6400 dual channel. That puts it at 12,800 right at "Newer Xeons 6 channels x 2,133 = 12,798"

Zen3+ is a new laptop APU, doesn't sounds like it'll make it to the Desktop but maybe Zen4 will be similar? There are a few trims available with this new Zen3+ architecture the highest of which have 8cores.

The media embargo lifted today and apparently there are sample laptops out with reviewers now. Don't know if any have the LPDDR5. Will be interesting to see benchmarks of Zen on this LPDDR RAM.

[ErikAdr]

The version openfoam2112 did not provide an executable of 'surfaceFeatures' and that was why I could'nt get any results for the motorbike benchmark. I gave up on openfoam2112. I went for a go with version 9 from openfoam.org. First I installed ubuntu 20.04.3 (standard installation) and then I followed the guide from openfoam.org to install on ubuntu. Next I downloaded the 'bench_template.zip' in post 447 by Simbelmynë and then I followed the guide from step 4 to 7 in post 448 by linuxguy. Everything worked in first attempt. That was a strange feeling after the many previous attempts.



I have build a quite low end computer, but with quality components. It is using a i5-12600 cpu with two channel DDR5 Ram. I'm pleased with the performance.



12th Gen Intel(R) Core(TM) i5-12600 DDR5 @ 4800

# cores Wall time (s):
------------------------
1 427.45
2 234.12
4 149.91
6 125.75



12th Gen Intel(R) Core(TM) i5-12600 DDR5 @ 5600

# cores Wall time (s):
------------------------
1 410.79
2 219.95
4 137.39
6 112.58

12th Gen Intel(R) Core(TM) i5-12600 DDR5 @ 6000

# cores Wall time (s):
------------------------
1 399.94
2 213.75
4 131.87
6 107.09

[wkernkamp]

Originally Posted by ErikAdr

It is using a i5-12860 cpu with two channel DDR5 Ram. I'm pleased with the performance.


12th Gen Intel(R) Core(TM) i5-12600 DDR5 @ 6000

# cores Wall time (s):
------------------------
1 399.94
2 213.75
4 131.87
6 107.09

Congratulations! Those results are indeed excellent. How much did the system cost you? You may qualify for price performance leader.

[ErikAdr]

Originally Posted by wkernkamp

Congratulations! Those results are indeed excellent. How much did the system cost you? You may qualify for price performance leader.

I think something found on ebay could have a better quality for price ratio, but for new hardware you could be right. The Ram prices are high at the moment, so the ratio could improve with lower Ram prices. The Ram was the most expensive component.

In total Dkr 7.533 which is € 1005 or $ 1147 (prices are excl V.A.T)
The components are:

Intel i5-12600 (Dkr 1599,20)
Kingston Fury Beast 2x16 GB 6000 MHz (Dkr 2636,80)
Asus TUF Gaming B660M-plus wifi (Dkr 1247,20)
Samsung 970 EVO plus 1TB (Dkr 709,60)
Noctua cpu (NH-U9B SE2) and cabinet coolers (Dkr 525,60)
Seasonic Focus GX550 (Dkr 479,20)
Fractal Design Core 1100 (Dkr 335,19)

I had hoped to be able to just use the cooler included with the cpu, but it was to noisy to share office with (or I had installed it poorly).

I'm considering to buy three more of these as an alternative to a dual EPYC 7313 system, since I have given up finding EPYC 7313 for sale. I think/hope the Ram prices will reduce in near future. The chipset/board is born with 2.5 Gb ethernet, and I think that could be sufficient for our need. A 5 port switch supporting 2.5 Gb ethernet is about Dkr 750 ~ 100 €.