[Phicau]

Dear all,

lately I have been working with a new computer and have been experiencing a very poor parallelization gain (subjective opinion) for what I was expecting. My computer is a dual Intel Xeon E52680 v3 (2.5GHz, 12 physical cores, 24 threads each), 64GB (4x16GB, 2133MHz DDR4) RAM.

To confirm my thought I decided to do a parallel performance test, as shown here:

https://www.pugetsystems.com/labs/hp...d-Opteron-587/

So basically I prepared a case based on the cavity tutorial, with a 1024x1024 mesh and reduced viscosity that iterates 100 times (no output to disk). Then I set a batch of cases with different mpirun flags, each to run independently while the computer was completely idle at night.

See graph attached. The X axis is the number of processes and the Y axis is the speedup.



Legend:
perfect -> the land of utopia
normal -> mpirun -np X simpeFoam -parallel
bc -> --bind-to core: overload-allowed
bcmc -> --bind-to core: overload-allowed --map-by core
bh -> --bind-to hwthread
bb -> --bind-to board

My findings confirm my fears, up to 12 processes, the scaling looks excellent, but after that it plateaus. Since the case is not huge, I could stand that scaling turned worse, but not as bad as shown by the graph. Furthermore, this guy has almost linear scaling all the way through to 40 cores (quad socket...), so I was expecting something similar up to 24.

What is interesting is that adding the flag --bind-to core: overload-allowed, increases the performance radically for large numbers of processes.

Does anyone have any clue on what could be happening or any thoughts to boost parallel performance further to 24 processes?

Thanks!

Pablo