[mrishi]

OpenMPI allows execution of programs in Multiple Programs Multiple Data (MPMD) setup. The required syntax is

mpirun -np x <program 1> : -np y <program 2>

If I supply program 1 and program 2 as two separate solvers, along with relevant paths to cases (which are located in subdirectories), I get the following error:

[0]
[0]
[0] --> FOAM FATAL ERROR:
[0] number of processor directories = 2 is not equal to the number of processors = 4
[0]
FOAM parallel run exiting
[0]

My job submission command is:

 mpirun \
        -np 2 interFoam -parallel -case $PBS_O_WORKDIR/2pIFaxis> $PBS_O_WORKDIR/2pIFaxis/log2pIF : \   

        -np 2 interFlow -parallel -case $PBS_O_WORKDIR/2pISOaxis > $PBS_O_WORKDIR/2pISOaxis/log2pIso

where $PBS_O_WORKDIR points to directory from where call is made to the cluster's job scheduler (hence no hostfile supplied).


Each case folder has 2 processor* directories (and hence a total of 4 processors needed). If I run multiple SERIAL cases in this manner, there is no issue. Individually, both the cases run properly. Why is this miscommunication happening with the solver with -parallel and how to resolve it?

If I understand correctly, OMPI is supposed to launch one solver and then the other according to its default processor allotment policy (--map-by core).

Otherwise, if I use two separate mpirun commands, it seems that the two programs compete for resources on the same node (wallTime is almost exactly 2*executionTime)


Details: OpenFOAM-6 running on HPC cluster with OpenMPI-1.10.4
I used the following for reference: https://www.open-mpi.org/faq/?category=running#mpmd-run

[mrishi]

One possible way I have managed to do this is using multiple mpirun commands:

mpirun --bind-to none -np 2 interFoam -parallel -case $PBS_O_WORKDIR/2pIFaxis> $PBS_O_WORKDIR/2pIFaxis/log2pIF &
mpirun --bind-to none -np 2 interFlow -parallel -case $PBS_O_WORKDIR/2pISOaxis > $PBS_O_WORKDIR/2pISOaxis/log2pIso

where using bind-to none seems critical to ensure that the processes are moved around. Without this argument, both the solvers appear to attach to the same cores, which shows up as wallTime = 2*executionTime in the log files (as noted above).


However, the method mentioned in my first post seems more elegant with possibilities to bind processes to sockets, while the lifeblood of this second method is the fact that things can float around.


I'll appreciate any help to make the first method work (or the best way to go about this).