[nmc1988]

Originally Posted by Simbelmynë

Here's the modified script file. There is no fail-safe to check if the meshes exist so you have to run the cases at least once before using this. It is reasonably easy to add that check though.

#!/bin/bash


for (( t=0; t<3; t++ )); do

    # Clear old data
    for i in 2 4; do
        cd run_${i}
        if [ -d ./100 ]; then
        # Control will enter here if $DIRECTORY exists and delete it if so.
        rm -r 100
        fi
        x=${i}
        for (( c=0; c<x; c++ ));do
        if [ -d ./processor${c} ]; then    
           cd processor${c}
           if [ -d ./100 ]; then
               # Control will enter here if $DIRECTORY exists and delete it if so.
               rm -r 100
           fi
           cd ..
        fi
        done
        if [ -f ./log.simpleFoam ]; then
        # Control will enter here if $FILE exists and delete it if so.
        rm log.simpleFoam
        fi    

        cd ..
    done

    # Run cases
    for i in 2 4; do
        echo "Run for ${i}..."
        cd run_$i
        if [ $i -eq 1 ] 
        then
        simpleFoam > log.simpleFoam 2>&1
        else
        mpirun -np ${i} simpleFoam -parallel > log.simpleFoam 2>&1
        fi
        cd ..
    done

    # Extract times
    echo "# cores   Wall time (s):"
    echo "------------------------"
    for i in 2 4; do
        echo $i `grep Execution run_${i}/log.simpleFoam | tail -n 1 | cut -d " " -f 3`
    done

done

It works well. Thank you

[codygo]

First run on 1x Epyc 7351, 8x 16GB rank 2 memory @ 2666hz, H11DSI-NT motherboard. The benchmark ran on OpenFOAM v1806, DP, without the streamline functions.

# cores   Exec. time (s)  Clocktime (s)
--------------------------------------------------
1            1023.46         1024
2            545.19          545
4            222.54          223
6            157.33          158
8            115.97          116
12           98.95           99
16           76.71           77 
20           85.53           108
24           64.33           99

I decided to rename Exec and Clocktime due to the results over 16 cores showing "better" performance. I was watching the runs complete in top and saw that the runs with over 16 processes were indeed slower.

Same hardware on Windows 10 Pro, OpenFOAM v1806 running natively in bash on Windows. Half the speed!

# cores   Wall time (s):
------------------------
1           1605.89
2           598.03
4           320.08
6           228.38
8           188.62
12          162.96
16          154.91

[gkarlsen]

Some really good deals on first generation Threadrippers out there now. Here are my results for the 1920x (stock). 3200 MT/s RAM, Ubuntu 18.04LTS, OpenFoam 6. SMT off

# cores Wall time (s):
------------------------
1 779.33
2 391.56
4 218.26
6 180.71
8 155.25
10 149.34
12 142.2

[nmc1988]

Originally Posted by gkarlsen

Some really good deals on first generation Threadrippers out there now. Here are my results for the 1920x (stock). 3200 MT/s RAM, Ubuntu 18.04LTS, OpenFoam 6. SMT off

# cores Wall time (s):
------------------------
1 779.33
2 391.56
4 218.26
6 180.71
8 155.25
10 149.34
12 142.2

It looks better than 1950x (result on the first page)

[Simbelmynë]

Originally Posted by gkarlsen

Some really good deals on first generation Threadrippers out there now. Here are my results for the 1920x (stock). 3200 MT/s RAM, Ubuntu 18.04LTS, OpenFoam 6. SMT off

# cores Wall time (s):
------------------------
1 779.33
2 391.56
4 218.26
6 180.71
8 155.25
10 149.34
12 142.2

Yeah, the results are better than the 1950X we have. Exactly what memory did you use? Is is single or dual rank?

[gkarlsen]

Originally Posted by Simbelmynë

Yeah, the results are better than the 1950X we have. Exactly what memory did you use? Is is single or dual rank?

I use 4x8 GB of GSkill FlareX 3200 MT/s, CL 14-14-14-34. Model no: F4-3200C14Q-32GFX.

SMT off improved my results by a tiny bit. So did flashing the BIOS for the most recent AGESA. I also had significantly better results with OpenFoam 6 packaged for Ubuntu than with v-1806 through Docker (which makes sense I guess). Which OpenFoam version did you use?

Some day, I will attempt pushing the RAM and/or CPU to higher clocks to see if there is any gain to be had. I have decent thermal headroom with the Enermax TR4 AIO

[edomalley1]

Here's some results from the slowest end of the spectrum. I ran this on a brand new HP Spectre x360 13 inch touchscreen convertible laptop. i7-8550U 4 core 1.8 Ghz (up to 4 GHz) processor with 8MB cache. 16 GB LPDDR3-2133 memory.

OpenFOAM 6 run on Windows 10 with Ubuntu app.

# cores   Wall time (s):
------------------------
1        1434.42
2        575.16
4        469.43

[naffrancois]

Hello, OpenFoam v1806 linux binaries:


2x Intel Xeon Gold 6148 +12x8Go 2666MHz single rank
# cores Wall time (s):
------------------------
10 105.82
20 64.77
40 49.28

[DÆdalus]

Hi. Are there any benchmarks on the new ryzen threadripper 2990wx with 32 cores and 64 threads? I would like to see how it scales on some time dependent 3D fluid dynamic problem, on some mpi implementation.

[flotus1]

You can take a look at the Threadripper results that are already there. It won't get any better with a 2990WX.

[DÆdalus]

Originally Posted by flotus1

You can take a look at the Threadripper results that are already there. It won't get any better with a 2990WX.

Ok, thank you. Should I expect it to scale linearly over the 64 threads as if it were 64 independent processors? BTW, I would like to know what kind of algorithms are being tested, how is the algorithm parallelized. Are there any benchmarks with time dependent domain decomposition methods?

[Simbelmynë]

Originally Posted by DÆdalus

Ok, thank you. Should I expect it to scale linearly over the 64 threads as if it were 64 independent processors? BTW, I would like to know what kind of algorithms are being tested, how is the algorithm parallelized. Are there any benchmarks with time dependent domain decomposition methods?

No. You can expect it to scale according to memory bandwidth. In the case of Threadripper and 3200 MHz memory we see that up to 8 cores give nice scaling and at 12 cores we hit the wall.



Regarding time-dependent benchmarks I suggest the damBreak 3D. We could open a separate thread for that.

[flotus1]

tl;dr: don't buy Threadrippr CPUs with more than 16 cores for CFD. They are designed for the exact opposite of the computing spectrum: compute-bound algorithms.

Should I expect it to scale linearly over the 64 threads as if it were 64 independent processors?

Absolutely not. Best case scenario would be that you get the same performance and scaling as with a TR 1950X. And that will only happen if you manage to keep threads away from the "rendering cores" without direct memory access.

BTW, I would like to know what kind of algorithms are being tested, how is the algorithm parallelized

Parallelization is done via MPI with a static domain decomposition. IIRC, METIS is used by default.
Don't know the exact algorithm used in this benchmark, we both would have to look it up.

[DÆdalus]

Great, thank you both. What would be the technical reason why it wouldn't scale? there is latency related to the message passing? I would like to understand what you have said about the memory bandwidth.

[flotus1]

Building a "home workstation" for ANSYS Fluent
Building a "home workstation" for ANSYS Fluent
Epyc 7301 ws

Parallel CFD codes are usually bandwidth limited. This means that the code balance (memory traffic per floating point operation) is rather high. In other words: the amount of actual calculations is low.
Most current CPUs have a much lower machine balance (peak memory bandwidth divided by peak floating point operations per second). So while doing CFD calculations, most of the time is spent waiting for data from memory. This is what usually causes poor scaling beyond 2-3 cores per memory channel.
This imbalance gets worse the more cores a CPU has, rendering CPUs with very high core counts basically useless for CFD. You could have bough a cheaper CPU with less cores and get more or less the same performance.
Threadripper CPUs with 24 and 32 cores have another issue on top of that: half of their cores have no direct connection to memory. See my second link.

[Simbelmynë]

Finally after many weeks of waiting (the motherboard was DOA) I got our new system up and running.

2 x EPYC 7301. 16*8 GB DDR4 2666 MHz, Ubuntu 18.04.1, OpenFOAM6

# cores   Wall time (s):
------------------------
1   704.64
2   358.31
4   169.83
6   118.95
8   91.48
16  51.13
32  32.76

*** Update *** When comparing against other results here I became a bit skeptical to the results. They seemed too good. The reason may be that I used the motorbike basecase from the OpenFOAM6 directories since the original basecase did not work. Everything went fine but it seems that 500 rather than 100 iterations were taken (and obviously something else as well since this seems very unreasonable).



It turns out that the snappyHexMesh run needs some etc files which are not located in /caseDicts, but rather in /caseDicts/mesh/. I am re-running the cases now with the standard basecase.


Here are the correct (and more reasonable) results

# cores   Wall time (s):
------------------------
1   1023.87
2   507.03
4   237.98
6   151.23
8   123.04
16  57.86
32  36.81

[nmc1988]

Originally Posted by Simbelmynë

Finally after many weeks of waiting (the motherboard was DOA) I got our new system up and running.

2 x EPYC 7301. 16*8 GB DDR4 2666 MHz, Ubuntu 18.04.1, OpenFOAM6

# cores   Wall time (s):
------------------------
1   704.64
2   358.31
4   169.83
6   118.95
8   91.48
16  51.13
32  32.76

The result looks great. How much is your new system ?

[Simbelmynë]

Originally Posted by nmc1988

The result looks great. How much is your new system ?

Do you mean price? Around 5000 Euro, excluding tax. The memory was approximately 50% of the total cost (only 128 GB).

[nmc1988]

Originally Posted by Simbelmynë

Do you mean price? Around 5000 Euro, excluding tax. The memory was approximately 50% of the total cost (only 128 GB).

Yes, I mean the price! The DDR4 memory is still expensive. I am considering to buy a secondhand 2x2690v2 system with price around 1500$. What do you think ?

[Simbelmynë]

The 2690v2 is a great price/performance choice if you can accept buying a refurbished system!