[spaceprop]

It's pretty amazing that 4x EPYC 7351's are roughly equivalent to (maybe slightly faster than) my whole cluster, which has 10x 10 core processors and almost perfect node scaling thanks to infiniband.

Memory bandwidth FTW

[naffrancois]

Originally Posted by spaceprop

That's faster than the dual 7601's in the main chart (data from havref posted earlier in this thread). Can you give some more info about your setup?

Yes it is a bit faster which surprised me as well, it does not make more sense to me than to you. Maybe more background tasks were running on the 2*7601 platform. I cannot say much more, I am running ubuntu 16.04 LTS and downloaded and installed the OF binaries following the website procedure using docker etc.

I just switched off smt nothing more on the optimization side. I only ran this serie of tests it is not an average

[flotus1]

A few things come to mind since the dual Epyc 7601 running 32 cores was also slower than my dual Epyc 7301 setup.
I suspect that internal memory organization plays an important role here. Mine used dual-rank modules which I highly recommend. I can only speculate that the 7601 setup used single-rank.
And of course different linux kernels were used. From my experience, older versions can severely hurt performance for Epyc CPUs and in general for CPUs released after the kernel.
Then when benchmarking (or running heavy jobs) I make sure the system is as idle as possible, caches are cleared and turbo modes are used to the full extent.

[spaceprop]

Originally Posted by flotus1

dual-rank modules which I highly recommend.

Agreed. I saw significant performance gains from dual rank vs single rank DDR3.

Originally Posted by flotus1

From my experience, older versions can severely hurt performance for Epyc CPUs and in general for CPUs released after the kernel.
Then when benchmarking (or running heavy jobs) I make sure the system is as idle as possible, caches are cleared and turbo modes are used to the full extent.

I never considered kernel versions...that's a good point. I make sure the system is idle and I check frequencies with turbostat to make sure they turbo, but what are the caches you mention?

[spaceprop]

Originally Posted by naffrancois

Yes it is a bit faster which surprised me as well, it does not make more sense to me than to you.

I saw you have 16x8GB 2666MHz RAM, but what rank is it? If it's 2R (dual rank), and the 2x 7601 was 1R, that might explain it.

Originally Posted by havref

2x Epyc 7601, 16x 8GB DDR4 2666MHz, 1TB SSD, running OpenFOAM 5.0 on Ubuntu 16.04.

havref: Is your RAM single rank?

I'm curious now, haha

[naffrancois]

It should be as I specifically asked for it. I did not check though. Is there a terminal command to check that, I d rather not open the case now

[flotus1]

Originally Posted by spaceprop

I never considered kernel versions...that's a good point. I make sure the system is idle and I check frequencies with turbostat to make sure they turbo, but what are the caches you mention?

When the system is new I check if it handles sustained heavy load with the advertised turbo frequencies.
Before running a time-critical simulation I do

# free && sync && echo 3 > /proc/sys/vm/drop_caches && free

to clear caches and

# echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

to make sure the highest possible CPU frequency is always used on all cores.
Many people will try to tell you that clearing caches beforehand is not necessary because the system will do a sufficient job of organizing memory. I found that this is not always the case.

[naffrancois]

dmidecode -t memory tells dual rank

[spaceprop]

Originally Posted by naffrancois

It should be as I specifically asked for it. I did not check though. Is there a terminal command to check that, I d rather not open the case now

sudo dmidecode -t memory

I think that will give you the part number, which you can then look up.

[spaceprop]

Originally Posted by naffrancois

dmidecode -t memory tells dual rank

Cool, thanks.

[spaceprop]

Originally Posted by flotus1

Before running a time-critical simulation I do

# free && sync && echo 3 > /proc/sys/vm/drop_caches && free

to clear caches

Huh...never knew this.

[tpearson-raptor]

2x IBM POWER9 Sforza 22 core CPUs [1], 8x16GB 2Rx4 DDR4-2400 registered ECC, OpenFOAM 5.x (GitHub version), Ubuntu 18.04, kernel 4.18-rc1

# Cores           Wall time [s]
------------------------------------------------------------              
1         677.38
2         366.04
4         180.1
6         124.17
8         96.64
12         70.16
16         56.39
20         47.47
24         41.76
44         36.71

[1] https://raptorcs.com/content/CP9M08/intro.html

[spaceprop]

Originally Posted by tpearson-raptor

2x IBM POWER9 Sforza 22 core CPUs

so much want, so not enough money

[tpearson-raptor]

Originally Posted by spaceprop

so much want, so not enough money

Fully understood, that's a high end professional workstation we benchmarked

In general, POWER9 pricing isn't that bad compared to Intel / EPYC; while the 22 core CPUs are the top-end, rather expensive parts, take a look at the 18 core devices (basically one step down from the premium 22-core CPUs) for best value. Performance will be pretty close to the full 22 core results in practice since the 18 core can boost to higher clocks before it hits thermal limits.

[flotus1]

That is pretty impressive for only 8 memory channels. Does it have some kind of L4 cache?

[tpearson-raptor]

Originally Posted by flotus1

That is pretty impressive for only 8 memory channels. Does it have some kind of L4 cache?

No, but each module has over 100MB of L3 cache; this system had 220MB L3 in total (5MB/core, 44 cores). POWER is also traditionally very strong on I/O of all sorts including to and from DRAM.

[kyle]

Doesn't POWER9 have 8 memory channels per socket? If so you're only using half of the memory channels.

[tpearson-raptor]

Originally Posted by kyle

Doesn't POWER9 have 8 memory channels per socket? If so you're only using half of the memory channels.

While POWER9 supports up to 8 memory channels in general per module, it also comes in three different packages: Sforza, Monza, and LaGrange. Sforza is the smallest package and only exposes four channels per module. LaGrange is the largest module, but consumes significantly more power and is much more expensive. Its main feature is BlueLink for GPU compute, versus the more balanced features of the Sforza package.

For an example of just how large LaGrange/Monza packages are, check out the picture here: https://www.tomshardware.com/news/ib...ers,36054.html That's what's needed for 8 memory channels to be exposed alongside all the PCIe lanes, etc. Sforza's roughly 1/2 the size on each side.

[tpearson-raptor]

After a bit of tuning....

2x IBM POWER9 Sforza 22 core CPUs, 8x16GB 2Rx4 DDR4-2400 registered ECC, OpenFOAM 5.x (GitHub version), Ubuntu 18.04, kernel 4.18-rc1, OpenFOAM modified to build with mcpu=power9 instead of mcpu=power8:

# Cores           Wall time [s]
------------------------------------------------------------              
1         659.81
2         355.5
4         176.6
6         121.2
8         94.65
12         68.4
16         55.63
20         46.81
24         41.51
44         36.3

EDIT: SMT2 results removed, looks like the benchmark script can let an error in the meshing stage propagate without halting, thus unexpectedly feeding bad data into the analysis stage.

[JBeilke]

Originally Posted by tpearson-raptor

Fully understood, that's a high end professional workstation we benchmarked

Just some questions:

• What about the noise of the machine? Are there plans for a quiet design?
• Did you try to run programs like StarCCM+ in a VirtualMachine? How is the performance?

Many Thanks
Jörn