[flotus1]

I somehow find it hard to believe that you get 100% scaling from 2xSMT. Especially for this kind of application. We already see diminishing returns up to 44 threads.

[tpearson-raptor]

Originally Posted by JBeilke

Just some questions:

• What about the noise of the machine? Are there plans for a quiet design?
• Did you try to run programs like StarCCM+ in a VirtualMachine? How is the performance?

Many Thanks
Jörn

The noise is already quite low; we offer a quiet workstation chassis option as well. Even the 4U server only measures around 47dB with the meter sitting right on top of the chassis. The 2U server machines are significantly louder, of course.

Looking in to StarCCM+, it looks like it is only available for x86 machines? If Siemens is willing to consider a recompile for ppc64el we could then run benchmarks against it. We are seeing more and more vendors start offering ppc64el builds of their software, so it probably wouldn't hurt to ask them.

Originally Posted by flotus1

I somehow find it hard to believe that you get 100% scaling from 2xSMT. Especially for this kind of application. We already see diminishing returns up to 44 threads.

EDIT: Yep, something went wrong in the meshing stage, but the benchmark did not halt or otherwise indicate there was a problem. Had to dig around in the log files due to the suspiciously good numbers. Looks like ~30s is the best time we have thus far.

[Simbelmynë]

Originally Posted by tpearson-raptor

EDIT: Yep, something went wrong in the meshing stage, but the benchmark did not halt or otherwise indicate there was a problem. Had to dig around in the log files due to the suspiciously good numbers. Looks like ~30s is the best time we have thus far.

I have also experienced ridiculous numbers (like 2 seconds on a 7600k CPU) when playing around with the script file (number of cores being used mostly). Did you find any information in the log files that reveal any strange behavior?

[JBeilke]

Originally Posted by tpearson-raptor

Looking in to StarCCM+, it looks like it is only available for x86 machines? If Siemens is willing to consider a recompile for ppc64el we could then run benchmarks against it. We are seeing more and more vendors start offering ppc64el builds of their software, so it probably wouldn't hurt to ask them.

StarCCM+ was just an example as a code which is not available as native port.
POWER is usually very good in running VirtualMachines. So my question is, how good do the the Intel/Amd ports run in a virtual environment. You might tell us how this benchmark behaves in a VM.

[kyle]

Did Siemens can the Power and Itanium ports of Star-CCM+? CD-adapco had them before the acquisition.

[Simbelmynë]

Here are some results from my old home computer.

I played around with over-clocking the CPU and the memory and it turns out (not surprisingly perhaps) that over-clocking the CPU has little to no influence on the CFD performance. It would be interesting to see if going to 4000+ MHz memory modules scales as well as 2133 to 3200 MHz does.

2400 MHz memory was much faster compared to 2133 MHz (not really linear increase but almost). I think my results can be extrapolated fairly well to the 8700k with 3200 MHz memory.

Note that I also under-clocked the CPU to 3 GHz (using 2400 MHz memory), which further points to the extreme benefit of faster memory rather than a faster CPU.

-------------------------------------------------------
core i7-7600K, DDR4 2133 MHz CL 15
# cores Wall time (s):
------------------------
2 423.92
4 357.42

core i7-7600K, DDR4 2400 MHz CL 15
# cores Wall time (s):
------------------------
2 389.9
4 322.65

core i7-7600K, DDR4 2400 MHz CL 13
# cores Wall time (s):
------------------------
2 388.86
4 320.97

core i7-7600K @ 4500MHz, DDR4 2400 MHz CL 15
# cores Wall time (s):
------------------------
2 385.99
4 319.42

core i7-7600K @ 4700MHz, DDR4 2400 MHz CL 15
# cores Wall time (s):
------------------------
2 379.26
4 321.16

core i7-7600K @ 3000MHz, DDR4 2400 MHz CL 15
# cores Wall time (s):
------------------------
2 441.56
4 334.5

[tarik]

Hello! Got a macbook pro 13" 2018 model to test some stuff and here's CFD Benchmark results for it. So, before that, it's running a Intel Core i5 8259U 28W Coffee Lake U CPU with base clock of 2.3 GHz and Turbo up to 3.8GHz. It has 8GB LPDDR3 2133 MHz memory. Benchmark was done post the supplemental update to correct thermal issues. I used Openfoam v5 under Docker for Mac.

SnappyHexMesh results:
Cores        Real time
  1            18m25.925s
  2            13m39.983s
  4            9m53.062s

Benchmark results:
# cores   Wall time (s):
------------------------
    1           729.07
    2           410.78
    4           370.82

Clocks were 3.6GHz on average and stable, power consumption slightly above TDP at about 30W, system had 0 fan rpm at 1 core, was silent between 1200-4000 rpm with 2 cores and audible at 6700 rpm when using all 4 cores. (before supplemental update clocks would cycle between 3.6 and 2.4 and consumption would drop below TDP, if anyone wants to know). Also, I had the same posted here before about particles outside of mesh and had to comment the 2 lines in controlDict. Also had to run 4 cores case twice as first time it gave a 6s result... One thing that interested me is how close 2 and 4 cores were. I wonder if thats bandwidth limitations or due to system having only 8GB of ram and maybe swap could be limiting (I couldn't find a way to see memory and swap usage) or could have to do with intel power gadget reporting only 50% utilization with 4 cores run. I did not disable Hyperthreading, btw.

[nmc1988]

Hello,
Here is the benchmark result with my PC : i7 6950x@3.5GHz (slightly OC) - 4 x8Gb DDR4 @ 3000Mhz. I am using Windows 10 and OpenFoam installed on bash

# cores   Wall time (s):
------------------------
4 249.42
6 184.3
8 167.84
10 163.67

I do not really know how to change the timing to OC Ram to get higher frequency. I have tried to changed the Ram frequency in Bios to 3200Mhz but after running CFD for few hours, the system is crashed and computer restarts.
I am using CPU cooler: Thermalright true spirit 120m (cost around 35$). Is it a problem? The CPU temperature at full load of 10 cores are around 75 degree Celsius

[Simbelmynë]

75 degrees is probably fine (noise levels might be a problem though). I would try to limit OC the CPU and try to max out the memory instead. If you add more voltage to the memory then a case fan directed at the memory can help a lot.


The best suggestion is to search for a dedicated tutorial on the 6950X and overclocking (there are several) and to approach the matter with caution since you can fry the components.

[wyldckat]

Originally Posted by nmc1988

I am using CPU cooler: Thermalright true spirit 120m (cost around 35$). Is it a problem? The CPU temperature at full load of 10 cores are around 75 degree Celsius

Originally Posted by Simbelmynë

75 degrees is probably fine (noise levels might be a problem though).

A few notes, because the 75C seems way too high based on my own experience:

1. 14nm lithography at 75 Celsius seems too much to me. Although I cannot find any clear instructions online, but keeping it at 70C or below is what I would prefer to avoid loosing the CPU after a year or two. 60C should run for a lifetime.
2. Thermal design power of the "Thermalright true spirit 120m", or at least of the variants: 160 watt
3. The i7 6950x has a TDP of 140W and 3.5 GHz should be the maximum stock speed if HyperThreading is disabled (4.0GHz is the max single core and 3.0 GHz when all threads are on, hence 3.5 GHz with just all cores-no-threads). Therefore, it doesn't feel like it should have been necessary to overclock.
4. The cooler should be blowing the air on a direct path to the outside of the tower, not to the inside of the tower. If it blows into the inside, it might be warming up everything inside, including the RAM, hence the crash.
5. If the RAM is not graded to run at 3200GHz, then it would be an overclock on its own and would overheat considerably, hence requiring its own fans.

[nmc1988]

Originally Posted by Simbelmynë

75 degrees is probably fine (noise levels might be a problem though). I would try to limit OC the CPU and try to max out the memory instead. If you add more voltage to the memory then a case fan directed at the memory can help a lot.


The best suggestion is to search for a dedicated tutorial on the 6950X and overclocking (there are several) and to approach the matter with caution since you can fry the components.

Originally Posted by wyldckat

A few notes, because the 75C seems way too high based on my own experience:

1. 14nm lithography at 75 Celsius seems too much to me. Although I cannot find any clear instructions online, but keeping it at 70C or below is what I would prefer to avoid loosing the CPU after a year or two. 60C should run for a lifetime.
2. Thermal design power of the "Thermalright true spirit 120m", or at least of the variants: 160 watt
3. The i7 6950x has a TDP of 140W and 3.5 GHz should be the maximum stock speed if HyperThreading is disabled (4.0GHz is the max single core and 3.0 GHz when all threads are on, hence 3.5 GHz with just all cores-no-threads). Therefore, it doesn't feel like it should have been necessary to overclock.
4. The cooler should be blowing the air on a direct path to the outside of the tower, not to the inside of the tower. If it blows into the inside, it might be warming up everything inside, including the RAM, hence the crash.
5. If the RAM is not graded to run at 3200GHz, then it would be an overclock on its own and would overheat considerably, hence requiring its own fans.

Thank you for all of your replies!
So Should I change the cooler ? Actually I have turned off HT. My RAM is designed to work at 3000MHZ and the stock speed of 6950x is 3.0GHz. By default the RAM works at 2132MHz and to get 3000 MHz, I have to activate XMP profile.
Also to get CPU works at 3.5 GHz, I also increase CPU cache, otherwise the CPU works at 3.0GHz only!
By the way, how do you think of my Benchmark result! I think it is quite similar to 7820x result and a little better than 7900x. But I see the quite much improvement of 7820x after RAM timing tweak, so I am curios that I can do the same for my system ?

[Simbelmynë]

I would continue to play around with the memory but don't go crazy on the voltage. Just test to increase the frequency and try to decrease the timings if it doesn't work. Baby steps



Regarding the temperatures, here is a reasonable read:


http://www.tomshardware.co.uk/forum/...ure-guide.html


The cooler you have might not be the best, but try to use stock operation first and compare to the benchmark results before you go out and buy another heatsink. If you still have 70+ degrees when running then you can change (corsair hydro series is a really nice option).



I think your results seem nice so be happy even if you can't manage higher memory speeds

[nmc1988]

Originally Posted by Simbelmynë

I would continue to play around with the memory but don't go crazy on the voltage. Just test to increase the frequency and try to decrease the timings if it doesn't work. Baby steps



Regarding the temperatures, here is a reasonable read:


http://www.tomshardware.co.uk/forum/...ure-guide.html


The cooler you have might not be the best, but try to use stock operation first and compare to the benchmark results before you go out and buy another heatsink. If you still have 70+ degrees when running then you can change (corsair hydro series is a really nice option).



I think your results seem nice so be happy even if you can't manage higher memory speeds

Thank you! I will try to play with the memory. If the system is not stable I will just stick with current configuration. I will update the result soon!

[nmc1988]

Originally Posted by nmc1988

Thank you! I will try to play with the memory. If the system is not stable I will just stick with current configuration. I will update the result soon!

Hello again,
I have OC the memory a little bit, here is the result
i7 6950x @3.5Ghz, RAM OC @ 3250Mhz 16-18-18-36 (XMP profile 3000 Mhz 15-17-17-35)

# cores   Wall time (s):
------------------------
8 156.36
10 152.68

The CPU temp around 70-75 degree at full load.
The result is good for me

But there is a strange thing that I do not understand. The first time I change RAM frequency and timing to 3250Mhz 16-18-18-36. I got unbelievable result like this

# cores   Wall time (s):
------------------------
8 139.26
10 105.67

I do not believe this result and I run again several time and I got the result similar to the one I posted

# cores   Wall time (s):
------------------------
8 156.36
10 152.68

Does anyone know what happened with my system at the first time ? Thank you

[Simbelmynë]

If it seems too good to be true, it generally is


You should check the log file and make sure that the solver has done a full 100 iterations. If it crashes it gives no indication and the results are posted from the time of crash.


So this just means that you have an unstable system and that you should try to tune it down a little bit.

[nmc1988]

Originally Posted by Simbelmynë

If it seems too good to be true, it generally is


You should check the log file and make sure that the solver has done a full 100 iterations. If it crashes it gives no indication and the results are posted from the time of crash.


So this just means that you have an unstable system and that you should try to tune it down a little bit.

Thank you. I think that could be the reason. But I cannot check the solver has done 100 iterations or not because I have deleted that folder to run another bench mark
Anyway, the current result is good for me now, because I just bought this second hand 6950x with price of 450$

[wyldckat]

Just a quick note about this:

Originally Posted by nmc1988

But there is a strange thing that I do not understand. The first time I change RAM frequency and timing to 3250Mhz 16-18-18-36. I got unbelievable result like this

# cores   Wall time (s):
------------------------
8 139.26
10 105.67

I do not believe this result and I run again several time and I got the result similar to the one I posted

# cores   Wall time (s):
------------------------
8 156.36
 10 152.68

Does anyone know what happened with my system at the first time ? Thank you

If the result was in fact real and not just due to a crash, then here are a couple of interesting calculations:

• CPU clock ratio: 3.5/3.0 = 1.1666 speed up if running at 3.5GHz versus 3.0 GHz stock.
• 156.36/1.16 = ~134.5 <- which is fairly close to 139.26 that you had gotten...

So if the run didn't in fact crash, then you could be seeing thermal throttling, i.e the CPU reducing speed to avoid killing itself.


Keep a close look at your Task Manager, in the Performance tab (if I remember correctly) and confirm at what clock speed the CPU is actually running, because Windows 10 does show that value with a fairly good accuracy.

[Simbelmynë]

I ran into similar strange results when overclocking too excessive on my 7600k. I believe this is due to the script is not displaying solver information in the terminal. Due to this we are not alerted when an error occurs and since the "time" terminal command is not connected to the solver it has no way of casting an error.


I have modified the original script to repeat the calculations three times (at least) and also to avoid re-meshing on the second and third simulation. You could do the same to really check that your OC setup is stable.


With stable configurations the fluctuations are of the order a couple %.


If you wish to check for thermal throttling then you can use the "top" command as well as

lscpu | grep MHz

checking the current memory frequency is done by

dmidecode -t 17

[nmc1988]

Originally Posted by wyldckat

Just a quick note about this:


If the result was in fact real and not just due to a crash, then here are a couple of interesting calculations:

• CPU clock ratio: 3.5/3.0 = 1.1666 speed up if running at 3.5GHz versus 3.0 GHz stock.
• 156.36/1.16 = ~134.5 <- which is fairly close to 139.26 that you had gotten...

So if the run didn't in fact crash, then you could be seeing thermal throttling, i.e the CPU reducing speed to avoid killing itself.


Keep a close look at your Task Manager, in the Performance tab (if I remember correctly) and confirm at what clock speed the CPU is actually running, because Windows 10 does show that value with a fairly good accuracy.

Thanks for your reply. But I think it is due to system crash, the value is not realistic for my PC configuration

Originally Posted by Simbelmynë

I ran into similar strange results when overclocking too excessive on my 7600k. I believe this is due to the script is not displaying solver information in the terminal. Due to this we are not alerted when an error occurs and since the "time" terminal command is not connected to the solver it has no way of casting an error.


I have modified the original script to repeat the calculations three times (at least) and also to avoid re-meshing on the second and third simulation. You could do the same to really check that your OC setup is stable.


With stable configurations the fluctuations are of the order a couple %.


If you wish to check for thermal throttling then you can use the "top" command as well as

lscpu | grep MHz

checking the current memory frequency is done by

dmidecode -t 17

Thank you! Could you share your script ? Because I also do not like to run the mesh generation all the time for the benchmark. I am not familiar with Open Foam

[Simbelmynë]

Here's the modified script file. There is no fail-safe to check if the meshes exist so you have to run the cases at least once before using this. It is reasonably easy to add that check though.

#!/bin/bash


for (( t=0; t<3; t++ )); do

    # Clear old data
    for i in 2 4; do
        cd run_${i}
        if [ -d ./100 ]; then
        # Control will enter here if $DIRECTORY exists and delete it if so.
        rm -r 100
        fi
        x=${i}
        for (( c=0; c<x; c++ ));do
        if [ -d ./processor${c} ]; then    
           cd processor${c}
           if [ -d ./100 ]; then
               # Control will enter here if $DIRECTORY exists and delete it if so.
               rm -r 100
           fi
           cd ..
        fi
        done
        if [ -f ./log.simpleFoam ]; then
        # Control will enter here if $FILE exists and delete it if so.
        rm log.simpleFoam
        fi    

        cd ..
    done

    # Run cases
    for i in 2 4; do
        echo "Run for ${i}..."
        cd run_$i
        if [ $i -eq 1 ] 
        then
        simpleFoam > log.simpleFoam 2>&1
        else
        mpirun -np ${i} simpleFoam -parallel > log.simpleFoam 2>&1
        fi
        cd ..
    done

    # Extract times
    echo "# cores   Wall time (s):"
    echo "------------------------"
    for i in 2 4; do
        echo $i `grep Execution run_${i}/log.simpleFoam | tail -n 1 | cut -d " " -f 3`
    done

done