[abstractbug]

Dear Foamers,

I have successfully installed OpenFOAM-in-box-dev-18.10v01 on CentOS 7.2.1511 (Core) computational cluster. On one of our other cluster with same hardware and software configuration, I am using OpenFOAM 4.x. I am observing linear scaling (reduction in computational time) with no. of nodes in OpenFOAM 4.x, however, I don't get any scaling in OpenFOAM-in-box , instead, there's a slight increase in time with no. of nodes. We have tested for no. of nodes from 1 to 10 with each node having 24 processors (not hyper-threaded). I suspect the issue is because of incorrect usage of open-mpi fabric (flags like --mca btl ^openib...) since OpenFOAM-4.x is using open-mpi 1.10.2 and OpenFOAM-in-box is using open-mpi 2.1.0.

We use pbs for submitting our jobs. Following is the script for OpenFOAM-in-box and OpenFOAM-4.x:

OpenFOAM-4.x:

mpirun --oversubscribe --mca btl ^openib -np 108 --host $HOSTS $foamExec $EXECUTABLE -parallel

OpenFOAM-in-box:

mpirun --oversubscribe --mca btl ^openib -np 108 --host $HOSTS  $foamExec $EXECUTABLE -parallel

Can you please suggest the appropriate openmpi fabric for getting the best speed-up .


Cheers !!!