[einatlev]

Hello!
I have been running interFoam in serial for a little while. Now that the models got bigger, I want to run them in parallel. Machine is a 16-core CentOS. OpenFoam version 2.2. Using ThirdParty mpi.
I created a decomposition following instructions on open foam website. When trying to run mpirun I get:

lava:damBreakNoObstacle>>mpirun --hostfile machines -np 8 interFoam -parallel
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------

Any ideas what I'm missing?
file "machines" holds just one line, saying:
lava cpu=8

Thanks!!

[zhengzh5]

Originally Posted by einatlev

Hello!
I have been running interFoam in serial for a little while. Now that the models got bigger, I want to run them in parallel. Machine is a 16-core CentOS. OpenFoam version 2.2. Using ThirdParty mpi.
I created a decomposition following instructions on open foam website. When trying to run mpirun I get:

lava:damBreakNoObstacle>>mpirun --hostfile machines -np 8 interFoam -parallel
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------

Any ideas what I'm missing?
file "machines" holds just one line, saying:
lava cpu=8

Thanks!!

hey, try:

mpirun -n 8 interFoam -parallel

I think the hostfile tag is only needed when you have the processors distributed over multiple machines.

[einatlev]

So strange --
Here is what I get:

lava:damBreakNoObstacle>>mpirun -np 8 interFoam -parallel[lava:11845] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file runtime/orte_init.c at line 121
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_base_select failed
  --> Returned value Not found (-13) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
[lava:11845] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file orterun.c at line 616

BUT -- when I login as another user (not SU, just another user not me), I run in parallel no problem. Strangest thing is, our environment variables related to MPI appear to be the same (I copied the other user's environment vars into mine and tried to run).

Any advice??

[wyldckat]

Greetings to all!

@einatlev: The following thread also addresses this issue: http://www.cfd-online.com/Forums/ope...entos-5-a.html - but conclusion on that thread was that the problem was related to the Open-MPI version being used.

But since you've tried with other user accounts and worked fine on them, here are a few questions:

1. Is the installation common to all users? For example, is OpenFOAM installed at "/opt" or similar?
2. In any of the accounts, is this command used, when starting up the shell environment?
   

   module load openmpi-x86_64

   You can check if it's already loaded, by running:
   

   module list

3. The other possibility are the groups to which each user belongs to. You can use:
   

   groups

   in your own account, or as root:
   

   groups the_user_name

   to see to which groups each user belongs to.

Best regards,
Bruno

[einatlev]

Thank you Bruno.
The users all belong to one group, their own (mane of group - name of user). The ThirdParty mpi files have permissions of rwxrwxrwx, and belong to user and group 503, which I suppose is some kind of default?

OpenFOAM installation is located under /usr/local and everyone use the same installation (only one on the server).

MPI modules have been loaded properly.

Is the above information helpful?

Thanks!
Einat

[Bernhard]

Can you show the output of

which interFoam
which mpirun

for both the users? There might be some unintended difference there?

[einatlev]

For myself:

lava:~>>which mpirun
/usr/lib64/openmpi/bin/mpirun
lava:~>>which interFoam
/usr/local/OpenFOAM/OpenFOAM-2.2.x/platforms/linux64GccDPOpt/bin/interFoam
lava:~>>

For second user:

[taylorredmond@lava ~]$ which mpirun
/usr/lib64/openmpi/bin/mpirun
[taylorredmond@lava ~]$ which interFoam
/usr/local/OpenFOAM/OpenFOAM-2.2.x/platforms/linux64Gcc45DPOpt/bin/interFoam

So it's the same as far as I can tell... Other ideas?
thanks!!!

[francescomarra]

Dear Einat,

I experienced the frustration of getting everithing working OK in a parallel environment, and I know it is not easy. I am not a ICT specialist, but looking for errors for several long times, at least I am now experienced to find differences: the two outputs are not equal !

Looks better (cut&paste from your mail):
/usr/local/OpenFOAM/OpenFOAM-2.2.x/platforms/linux64GccDPOpt/bin/interFoam
/usr/local/OpenFOAM/OpenFOAM-2.2.x/platforms/linux64Gcc45DPOpt/bin/interFoam
So you got two characters, 45, more in the taylorredmond version.

I can suspect the two binary have been built under a different environment setup, but I am not sure. Surely Bruno is much more skilled than me on this.

I would suggest to compare your OpenFOAM baschrc with that of your collegue, as well as the Open-MPI version and gcc compiler version loaded by the shells of the two users, as Bruno already suggested.

Hoping it helps.

Regards,
Franco

[alexeym]

Hi,

Originally Posted by einatlev

lava:~>>which mpirun
/usr/lib64/openmpi/bin/mpirun
lava:~>>which interFoam
/usr/local/OpenFOAM/OpenFOAM-2.2.x/platforms/linux64GccDPOpt/bin/interFoam
lava:~>>

In the first post you've said OpenFOAM use ThirdParty OpenMPI while location of mpirun suggests you're using system-wide installed OpenMPI. Which one is actually used to build OpenFOAM?

What is a contents of machines file? Do you run solver on a single node or on multiple nodes?

[einatlev]

Thanks for pointing out the "45" characters. Tis really lead me to finding the problem. Turns out I needed to define the following:

module load openmpi-x86_64 || export PATH=$PATH:/usr/lib64/openmpi/bin

source /usr/local/OpenFOAM/OpenFOAM-2.2.x/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=system WM_COMPILER=Gcc45 WM_MPLIB=SYSTEMOPENMPI

and now it works for all users, including myself.