[wkernkamp]

The most important factor for CFD is the memory. On my machine I have two RDIMMs per channel and forced to 2400 MT/s. The RDIMMS are rank 2. There are then four ranks per channel. Another machine that has a single RDIMM per channel is not as fast (64 seconds instead of 60). However, that is not the exact same machine, so I need to investigate a bit.


In addition there are the BIOS settings but that can wait. I would suggest you run the benchmark with the machine as is to see where you are at. Also list your memory configuration with "sudo dmidecode -t 17. In addition run -t 19 and -t 20. With that info, I can compare to my machine.


Good luck!

[Tibo99]

Thank you very much for your help with this. You'll see in attachment a text file for each command. If you want me to post the results in the text instead, let me know.

I don't know what was the performance of your machine before you tweak it, but regarding your actual results compare to the one I got, I'm desapointed with the performance to be honest....

I noticed that its rank 4. Its seems that too many ranks in the channel can cause excessive loading and decrease the speed of the channel. Do you think one of the reason for the lack of performance could come from this?

Also, I took a screenshot of the 'system monitor' app during the run and I noticed that even if the machine has 192GB of RAM, it shows that it use about 8.7GB. And it was like that during the whole process. Is that normal? Am I missing something here?



Lenovo ThinkStation P710, 2x E5-2699C v4 (44 cores, 12 * GB DDR4[384GB max.]) with 192GB, Ubuntu 22.04 LTS

Software: OpenFOAM 2212v

# cores   Wall time (s):
------------------------
1 2 4 8 12 16 20 24 28 32
Meshing Times:
1 1715.24
2 1107.93
4 632.17
8 367.64
12 278.97
16 256.85
20 221.81
24 209.41
28 219.07
32 203.43
Flow Calculation:
1 1286
2 627.89
4 277.85
8 161.43
12 139.14
16 135.37
20 131.43
24 131.68
28 130.73
32 131.21

[bigphil]

Hardware

• System: Mac Studio 2022
• CPU: Apple M1 Ultra, 20 cores (16 performance + 4 efficiency)
• Memory: 64 GB
• OS: Ventura 13.2.1 (22D68)
• OpenFOAM-v2212 built from source using Apple clang version 14.0.0 and open-mpi 4.1.4_2 from homebrew

Times up to 16 cores:

1 432.76
2 250.05
4 137.34
6 88.55
8 69.04
12 53.38
16 46.44

[wkernkamp]

Originally Posted by Tibo99

Thank you very much for your help with this. You'll see in attachment a text file for each command. If you want me to post the results in the text instead, let me know.

I don't know what was the performance of your machine before you tweak it, but regarding your actual results compare to the one I got, I'm desapointed with the performance to be honest....


Look below: Machine is fine, but memory config is not!


I noticed that its rank 4. Its seems that too many ranks in the channel can cause excessive loading and decrease the speed of the channel. Do you think one of the reason for the lack of performance could come from this?


Not a factor, because you need to go to a 1 DIMM per channel config.


Also, I took a screenshot of the 'system monitor' app during the run and I noticed that even if the machine has 192GB of RAM, it shows that it use about 8.7GB. And it was like that during the whole process. Is that normal? Am I missing something here?

That is normal, your machine can tackle much larger problems than the benchmark.

Lenovo ThinkStation P710, 2x E5-2699C v4 (44 cores, 12 * GB DDR4[384GB max.]) with 192GB, Ubuntu 22.04 LTS

Software: OpenFOAM 2212v

When I looked at 17.txt, I found that your DIMMs are not in the right slots
In sequence, it shows the speeds as:

Configured Memory Speed: 2133 MT/s
Configured Memory Speed: 2133 MT/s
Configured Memory Speed: 2133 MT/s
Configured Memory Speed: Unknown
Configured Memory Speed: Unknown
Configured Memory Speed: Unknown
Configured Memory Speed: 2133 MT/s
Configured Memory Speed: 2133 MT/s
Configured Memory Speed: 2133 MT/s
Configured Memory Speed: Unknown
Configured Memory Speed: Unknown
Configured Memory Speed: Unknown



The corresponding bank locations are:

Bank Locator: CPU1_A0
Bank Locator: CPU1_B0
Bank Locator: CPU1_B1
Bank Locator: CPU1_C0
Bank Locator: CPU1_D0
Bank Locator: CPU1_D1
Bank Locator: CPU2_A0
Bank Locator: CPU2_B0
Bank Locator: CPU2_B1
Bank Locator: CPU2_C0
Bank Locator: CPU2_D0
Bank Locator: CPU2_D1


You have DIMMs in A0, B0 and B1 for each cpu. This will give you just dual channel memory instead of quad channel. In addition, the memory is not balanced, because channel B has more memory than channel A. What you need to do is put a DIMM of the same type and size in A0, B0, C0 and D0 for each CPU. You will need two extra DIMMs of the same rank , type and size. A difference can slow your memory down and will never make it faster! Leave B1 and D1 empty. Typically, the B0 slot would be farther from the CPU than the adjacent B1 slot. The B1 and D1 slots have a different color than the "0" slots. This upgrade will get you most of the way to decent performance somewhere between 60 and 70 seconds.



Another less significant issue is that the speed is 2133 MT/s, which is below the maximum of the CPU. If the DIMMs are rated as 2400 MT/s then removing the DIMMs from the B1 slots will probably fix it. If your DIMMs are rated 2133 MT/s, you can go in the BIOS and enforce the higher speed. Overclocking your DIMMs like that is not guaranteed to succeed! You will find a setting for the memory speed that reads "auto", but can be set to "2400". Going from 2133 to 2400 will reduce time at high core counts by 11%. If the machine won't boot go back to auto in BIOS.

[wkernkamp]

Originally Posted by bigphil

Hardware

• System: Mac Studio 2022
• CPU: Apple M1 Ultra, 20 cores (16 performance + 4 efficiency)
• Memory: 64 GB
• OS: Ventura 13.2.1 (22D68)
• OpenFOAM-v2212 built from source using Apple clang version 14.0.0 and open-mpi 4.1.4_2 from homebrew

Very nice result! The MAC will also be very power efficient. Some people measure Watts at the socket and publish iterations per Wh. I am sure you would be the winner of that competition!

[Tibo99]

Again, I can't say thank you enough for your help!

I will follow you procedure and re-run the benchmark to see where it stand after this.

It will maybe take some time before I post the new results since I'll need to wait for 2 extra DIMMs.

Lastly, I would like to hear your insight about the comparaison of the actual processor I got,E5-2699C v4, and the E5-2698 v4? The reason why is, I ordered the E5-2698 v4 processor but I receive the other one. Then, after further review, I noticed that even the E5-2698 v4 processor has 4 cores and memory cache less then the E5-2699C v4 processor, the E5-2698 v4 processor can reach 3.6 GHz in turbo mode compare to 2.4 GHz for the one I actualy have. So, if the E5-2698 v4 processor is better, I'll certainly ask for a swap.

Thank you again and best regards,

[wkernkamp]

Originally Posted by Tibo99

Again, I can't say thank you enough for your help!

I will follow you procedure and re-run the benchmark to see where it stand after this.

It will maybe take some time before I post the new results since I'll need to wait for 2 extra DIMMs.

Lastly, I would like to hear your insight about the comparaison of the actual processor I got,E5-2699C v4, and the E5-2698 v4? The reason why is, I ordered the E5-2698 v4 processor but I receive the other one. Then, after further review, I noticed that even the E5-2698 v4 processor has 4 cores and memory cache less then the E5-2699C v4 processor, the E5-2698 v4 processor can reach 3.6 GHz in turbo mode compare to 2.4 GHz for the one I actualy have. So, if the E5-2698 v4 processor is better, I'll certainly ask for a swap.

Thank you again and best regards,

A good place to look at for comparing processors is this one: https://en.wikipedia.org/wiki/List_o...eon_E5-2698_v4


As you can see by comparing the E5-2698 v4 and the 2699C v4 below it, you can see that the 2699C has 2 more cores a lower all core turbo of 2.4 versus 2.7, a lower two and one core turbo of 2.4 versus 3.6, but a larger cache of 55 MB versus 50 MB and a higher Thermal Design Power (TDP) of 145 W versus 135 W. The cache is quite important for the benchmark, so the all core solution will probably compensate for the lower all core clock. The TDP is complicated, because the processor is allowed to exceed it for a limited time (setting in BIOS). When a core is waiting for memory, it uses less power, so probably no effect on the benchmark from TDP. The TDP is the power use when all cores are running at the base frequency. That is 2.2 GHz for both and 145/135 =~ 22/20. You can check the core frequencies for n+1 cores with:

sudo cpupower -c 0-n frequency-info

For cpupower install linux-cpupower



I would prefer the E5-2698 v4 because single core programs will be a lot faster. That is important for a work station. In addition, the higher clocks will benefit any gaming you night want to do.


On ebay (from China):

Processor   Price  Cores Turbo
E5-2699C v4 $216.00  22   2.4 
E5-2698 v4  $199.99  20   3.6
E5-2696 v4  $157.00  22   3.7
E5-2697 v4   $94.00  18   3.6
E5-2697A v4  $99.00  16   3.6
E5-2684 v4   $39.95  16   3.0

So they didn't do the substitution to save money.

Looking at these prices, you might be better off selling these E5-2699C processors yourself after you bought the cheaper processors on Ebay. That would leave some money to take your wife to dinner to make up for the time spent on your workstation.

[Tibo99]

Thank you very much for your suggestions and the IT support!

I especially liked this one - "That would leave some money to take your wife to dinner to make up for the time spent on your workstation."

I'll try to get some RAM asap, configure the machine the way you suggested and re-run the benchmark. So, we'll see from there where the machine will stand.

New benchmark's results soon to come.........

Best Regards,

[jd210]

Hi Everyone,

Lucky enough that new job equals new HPC benchmarking.

2x 9374F, OpenFOAM v2212, CentOS 7.9. 24x 16GB 4800MT/s RAM. SMT is on.

1 437.74
2 246.09
4 119.22
8 74.38
16 34.49
24 22.01
32 16.03
48 12.68
64 11.07

Not much of an improvement over the Milan-X and it's massive L3 cache.

[gpouliasis]

Originally Posted by jd210

Hi Everyone,

Lucky enough that new job equals new HPC benchmarking.

2x 9374F, OpenFOAM v2212, CentOS 7.9. 24x 16GB 4800MT/s RAM. SMT is on.

1 437.74
2 246.09
4 119.22
8 74.38
16 34.49
32 16.03
64 11.07

Not much of an improvement over the Milan-X and it's massive L3 cache.

Comparing to older benchmarks, it seems to me that you can leverage much more performance by using faster ram.

[flotus1]

Originally Posted by gpouliasis

Comparing to older benchmarks, it seems to me that you can leverage much more performance by using faster ram.

There is no faster RAM available for these CPUs. And once there is, it will still have to be run at 4800MT/s. It's a server platform, memory overclocking won't work. The boards don't offer the necessary settings, and the CPUs are locked.

[jd210]

Interestingly AMD said to me the other day that the 9454 is proving to be a good option for CFD. So may get some results for that and other SKUs at some point.

[gpouliasis]

Originally Posted by flotus1

There is no faster RAM available for these CPUs. And once there is, it will still have to be run at 4800MT/s. It's a server platform, memory overclocking won't work. The boards don't offer the necessary settings, and the CPUs are locked.

I clearly need to do some further reading of the spec sheets then

[jd210]

Originally Posted by gpouliasis

I clearly need to do some further reading of the spec sheets then

You're right that faster RAM is important but it's a little more complicated than that. You're after the fastest possble memory transfer speeds. So EPYC Genoa has faster RAM 4800MT/s vs 3200MT/s for EPYC Milan but also has 12 channels vs 8. This results in memory bandwidth going from 206GB/s to 460GB/s which is huge!

The caveat is the CPU cache, which is way faster than RAM, but quite small. Milan-X chips have 768MB while the Genoa's have up to 256MB. If your case is small enough (like this one might be) most if not all of it fits in cache making it really fast.

As flotus says you can't overclock, but you wouldn't want to anyway as it's meant to run 24/365. And in such situations you're less concerned about raw speed and more about speed per watt consumed or speed per $ spent to buy.

[flotus1]

I have my doubts that these numbers reflect the actual capabilities of your CPUs.
I contributed some benchmarks for regular (non-X) 32-core Milan CPUs here, that are only slightly slower.
Your single-core result seems fine, but scaling could probably be improved.
Here is what I would usually recommend for low variance, and maximum performance:

• disable SMT (bios setting, theoretically unnecessary when core binding is done properly)
• enable "ACPI SRAT L3 Cache as NUMA domain" (bios setting)
• clear caches before a run using "echo 3 > /proc/sys/vm/drop_caches" (might require root privileges)
• !!!core binding!!! E.g. mpirun -np 64 --bind-to core --rank-by core --map-by numa

[jd210]

Originally Posted by flotus1

I have my doubts that these numbers reflect the actual capabilities of your CPUs.
I contributed some benchmarks for regular (non-X) 32-core Milan CPUs here, that are only slightly slower.
Your single-core result seems fine, but scaling could probably be improved.
Here is what I would usually recommend for low variance, and maximum performance:

• disable SMT (bios setting, theoretically unnecessary when core binding is done properly)
• enable "ACPI SRAT L3 Cache as NUMA domain" (bios setting)
• clear caches before a run using "echo 3 > /proc/sys/vm/drop_caches" (might require root privileges)
• !!!core binding!!! E.g. mpirun -np 64 --bind-to core --rank-by core --map-by numa

Thanks for the suggestions, disabling SMT is on my list, but it's a demo system so not easy for me to do! And I have other tests to do

Tried the core binding, it knocked a few tenths off the 64 core run but others were generally similar.

My thought is that this case is now too small for the latest EPYC chips and they run it so fast that disk read etc is having quite an effect. That said a 25% bump over a 7543 is still not insignificant!

Might try running for more iterations or messing about with what it writes and see what I can come up with.

[iamchermac]

Hi all,

I am making an attempt to use the benchmark scripts provided in the below post [#624], however I get the above error related to' numberOfSubdomains'.

I note a previous post where this issue cropped up and it was suggested it could be an issue with sed and paths (as the decomposeParDict in that particular scenario didn't seem to get updated). I can confirm each numberOfSubdomains of my created run cases have actually been edited accordingly. For e.g.:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 2;

method scotch;
// method scotch;

simpleCoeffs
{
    n               (4 1 1);
    delta           0.001;
}

hierarchicalCoeffs
{
    n               (3 2 1);
    delta           0.001;
    order           xyz;
}

manualCoeffs
{
    dataFile        "cellDecomposition";
}


// ************************************************************************* //

Would anyone be able to provide an assist here? For reference, I am using the Windows binary from openfoam.com. Any help would be appreciated and thank you in advance.

Originally Posted by wkernkamp

System:
Gigabyte MD80-F34, 2x E5-2683 (16 core, 45 MB Cache), 16x *GB DDR4-2400 2R two RDIMMs per channel, Debian Linux 5.15


Software:
OpenFOAM v2212 from openfoam.com

Meshing Times:
1 1426.06
2 931.58
4 524.9
8 326.17
12 247.9
16 221.57
20 187.4
24 171.64
28 176.2
32 159.53
Flow Calculation:
1 1041.68
2 523.79
4 238.93
8 126.03
12 91.68
16 76.93
20 67.85
24 64.21
28 61.2
32 60.19

[wkernkamp]

I have not used windows myself. Sometimes windows programs require a carriage return and then a newline, instead of just newline as in unix.


The resulting file you produced shows "numberOfSubdomains 2;" while In the basecase/system directory it is 6. So it appears sed is working as intended.


If you change directory to the run_2 directory and then execute the commands by hand, do you still get the error? Note that before running openfoam in linux you need to do:

source OpenFOAM/OpenFOAM-v2212/etc/bashrc

There must be a windows equivalent. This tells the system where to find executeables such decomposePar. (You probably new that )


Do the log files for blockMesh, etc show successful completion?

[iamchermac]

Hi Will, thanks for the quick reply.

Originally Posted by wkernkamp

If you change directory to the run_2 directory and then execute the commands by hand, do you still get the error? Note that before running openfoam in linux you need to do:

source OpenFOAM/OpenFOAM-v2212/etc/bashrc

There must be a windows equivalent. This tells the system where to find executeables such decomposePar. (You probably new that )

The Windows binary uses an implementation of MSYS2 and environment variables are set (at least appears so) on launch of the shell via the launch script openfoam.com provides. Their implementation does not provide sudo, and I have a suspicion it may be an issue with file permissions - but unsure. In the meantime I have set up the machine for a Linux dual-boot.

I have got OpenFOAM installed and successfully ran the benchmarking in Ubuntu 22.04. I am using a Ryzen 5950x with 3600MHz Dual Rank and the tl;dr is to stick with the 5600x or 5800X3D if opting for this architecture for single simulations (as discussed in previous posts). TODO: Check out the efficacy with parallel parametric and multi-simulations.

I am still getting an odd failure when using 5-cores (for tinkering and interest purposes I benchmarked 1-16 in increments of 1). I will investigate this oddity a bit more later.

Originally Posted by wkernkamp

I have not used windows myself. Sometimes windows programs require a carriage return and then a newline, instead of just newline as in unix.

Originally Posted by wkernkamp

Do the log files for blockMesh, etc show successful completion?

I initially also thought the same about line endings, but the MSYS2 implementation is *nix based so this changed my mind on that one. I will investigate further.

I will do some more tinkering when I finally boot back into Windows. I'll review log files as suggested, and take the benchmark through a manual workflow to see if that makes a difference. I will probably (should have done really) run some of the tutorial cases I usually test when installing a new version of FOAM.

It will be useful for my purposes if I can get similar performance with this Windows binary as compared to bare metal under Linux as opposed to resorting to WSL. So I will keep at it and I'll update my findings here if I am successful.

[iamchermac]

I have resolved my issue with the Windows binary with the help of forum post [OpenFOAM.com] error while loading shared libraries: mingw.

It seems that just double-clicking on the installer causes it to be installed under the user's Roaming AppData folder and does not install MS-MPI. Correct installation requires Right-click > Install as Admin so that MS-MPI can be installed, with the OpenFOAM installation folder placed by default under Program Files instead of Roaming.