My simulations don't scale when I use multiple nodes -help

KTG · February 11, 2020, 18:47

Hey all,

I am having some sort of issue where I am observing a speedup in my simulations as I add more cores within one node, but as soon as I go to two nodes, there is no increase. I can't find any info on this - I tried the -bind-to core mpi flag, but no luck. Has anyone observed this behavior of openFoam on a cluster, or have any ideas about how to fix it? I am using openMPI and SLURM, so I call the solver this way:

mpirun -np $SLURM_NTASKS pimpleFoam -parallel > log

I am using a cluster running centos with these specs:

144 Dual Socket-Six Core Nodes
2.8 GHz Intel Xeon (Westmere X5660) processors
RAM:
- 24 Gbytes memory per node (2 Gbytes per processor core) on most nodes
Mellanox QDR Infiniband interconnect
Gigabit Ethernet interconnect for management

I was thinking the issue had to do with a bottleneck? Is there a way to check that its actually using the Infiniband ? Any ideas welcome!

Thanks-

HPE · February 12, 2020, 00:09

how many cells - how many cores?

KTG · February 18, 2020, 22:00

Hi,

sorry for the delay! It seems that CFD forums doesn't send me e-mails anymore...

Last time I was playing with it, I used over 5 million cells, with 64 cores vs. 32, and got was appears to be the same result. Before I though it was a bottle neck issue, but for all I know the first node is doing all the work. I also notice that the averaging function object is taking an insanely long time, which may or may not be related.

Thanks for the reply!

KTG · March 4, 2020, 23:26

I was thinking maybe the MPI is throwing 64 jobs to the first 32 cores and multi-threading it? Does anyone know if there is a way to track the hardware that works on each processor directory / any way to get more information out of of the cluster about what is going on?

February 11, 2020, 18:47	My simulations don't scale when I use multiple nodes -help	#1
KTG Senior Member Abe Join Date: May 2016 Posts: 119 Rep Power: 10	Hey all, I am having some sort of issue where I am observing a speedup in my simulations as I add more cores within one node, but as soon as I go to two nodes, there is no increase. I can't find any info on this - I tried the -bind-to core mpi flag, but no luck. Has anyone observed this behavior of openFoam on a cluster, or have any ideas about how to fix it? I am using openMPI and SLURM, so I call the solver this way: mpirun -np $SLURM_NTASKS pimpleFoam -parallel > log I am using a cluster running centos with these specs: 144 Dual Socket-Six Core Nodes 2.8 GHz Intel Xeon (Westmere X5660) processors RAM: 24 Gbytes memory per node (2 Gbytes per processor core) on most nodes Mellanox QDR Infiniband interconnect Gigabit Ethernet interconnect for management I was thinking the issue had to do with a bottleneck? Is there a way to check that its actually using the Infiniband ? Any ideas welcome! Thanks-

February 12, 2020, 00:09		#2
HPE Senior Member Herpes Free Engineer Join Date: Sep 2019 Location: The Home Under The Ground with the Lost Boys Posts: 931 Rep Power: 13	how many cells - how many cores? __________________ The OpenFOAM community is the biggest contributor to OpenFOAM: User guide/Wiki-1/Wiki-2/Code guide/Code Wiki/Journal Nilsson/Guerrero/Holzinger/Holzmann/Nagy/Santos/Nozaki/Jasak/Primer Governance Bugs/Features: OpenFOAM (ESI-OpenCFD-Trademark) Bugs/Features: FOAM-Extend (Wikki-FSB) Bugs: OpenFOAM.org How to create a MWE New: Forkable OpenFOAM mirror

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Launching multiple CFD simulations in parallel on cluster	jthornton232	OpenFOAM Programming & Development	0	August 14, 2019 21:53
Fine/Marine parallel computation on multiple nodes	inarus	Fidelity CFD	2	February 24, 2015 08:38
MPI code on multiple nodes, scalability and best practice	t.teschner	Hardware	0	October 7, 2014 06:07
Can I scale up from 1 to 2 nodes with mixed CPU?	pongup	Hardware	2	April 14, 2014 16:50
Running in parallel on multiple nodes	kalyangoparaju	OpenFOAM	5	January 18, 2012 12:36

February 18, 2020, 22:00		#3
KTG Senior Member Abe Join Date: May 2016 Posts: 119 Rep Power: 10	Hi, sorry for the delay! It seems that CFD forums doesn't send me e-mails anymore... Last time I was playing with it, I used over 5 million cells, with 64 cores vs. 32, and got was appears to be the same result. Before I though it was a bottle neck issue, but for all I know the first node is doing all the work. I also notice that the averaging function object is taking an insanely long time, which may or may not be related. Thanks for the reply!

March 4, 2020, 23:26		#4
KTG Senior Member Abe Join Date: May 2016 Posts: 119 Rep Power: 10	I was thinking maybe the MPI is throwing 64 jobs to the first 32 cores and multi-threading it? Does anyone know if there is a way to track the hardware that works on each processor directory / any way to get more information out of of the cluster about what is going on?