[aylalisa]

Hi,

I've up-graded OpenSUSE to version 13.1.
and OpenFoam to version 2.2.2


I receive following error message once I try to start the calculation with mpirun:

It looks like opal_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

opal_shmem_base_select failed
--> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

opal_init failed
--> Returned value Error (-1) instead of ORTE_SUCCESS

With the old combination:
OpenSuse 12.2/OF 2.2.1 I didn't face that problem.

How can help?
Where else could I post that problem?
At the moment I am not able to run my calculations in parallel.


The version I use and that makes the trouble:

mpirun --version
mpirun (Open MPI) 1.7.2

OpenFoam /etc/bashrc/
export WM_MPLIB=SYSTEMOPENMPI


The version in the ThirdParty-2.2.2 folder
openmpi-1.6.3
I've changed WM_MPLIB=OPENMPI, sourced ~/.bashrc, checked $FOAM_MPI=1.6.3 and tried to start parallel run.
Same error message!


I've just found the existing thread to the same topic:
post428368http://www.cfd-online.com/Forums/ope...tml#post428368
Sorry' that I've opened a new one!
I've tried to solve the problem accordingly but unsuccessfully!




Please help!!!


Aylalisa

[nimasam]

i had similar issue, in Ubuntu when i install the mpich package, so if you have this package, you maybe want to uninstall it then every thing works smoothly

[aylalisa]

Dear Nimasam,

I've installed Open MPI 1.7.2 and 1.6.3
1.7.2 comes with OpenSuse 13.1
1.6.3 comes with ThirdParty 2.2.2

Originally Posted by nimasam

i had similar issue, in Ubuntu when i install the mpich package, so if you have this package, you maybe want to uninstall it then every thing works smoothly

I've tried to use openmpi-1.6.3 instead of 1.7.2:

In my PATH variable both mpi-versions were listed:
:/usr/lib64/mpi/gcc/openmpi/bin:
:/opt/ThirdParty-2.2.2/platforms/linux64Gcc/openmpi-1.6.3/bin:

I've export PATH= in my users .bashrc without the first version.

In OpenFOAMs bashrc I've set: export WM_MPLIB=OPENMPI
OpenFOAMs bashrc is sourced in my users bashrc.

After that change it seems that mpiversion 1.6.3 is used but gives another error message:

mpirun: error while loading shared libraries: libopen-rte.so.4: cannot open shared object file: No such file or directory

I've add another export in my home .bashrc including the new path where the missing file is located:
export LD_LIBRARY_PATH=/.../..../usr/local/lib64

Now I get another error:

Sorry! You were supposed to get help about:
opal_init:startup:internal-failure
But I couldn't open the help file:
/opt/ThirdParty-2.2.2/platforms/linux64Gcc/openmpi-1.6.3/share/openmpi/help-opal-runtime.txt: No such file or directory. Sorry!
--------------------------------------------------------------------------
[linux-0ze2:01786] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file runtime/orte_init.c at line 79
[linux-0ze2:01786] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file orterun.c at line 694


Why does Open MPI 1.7.2 (Systems version) not work with OP 12.3?
How can I remedy that problem?
Do I have to look for missing libraries manually? Does that problem occur because of OpenSuse update to 12.3?


Aylalisa

[alexeym]

Hi,

Can you describe your installation in more details? I've done fresh install of openSUSE 12.3, installed OpenFOAM 2.2.2 from RPMs (following the guide on the site) and...

1. System version of OpenMPI is 1.6
2. I can run solvers in parallel without any problems (well, there is a problem with default host file, but it is mentioned in guide and has nothing to do with shared memory)

[aylalisa]

Hi,

First step
I've installed OpenSuse 13.1. With that version came the current openmpi version 1.7.2-2-1.3 and finally OpenFoam 2.2.2.
I have followed the detailed installation documentation
http://www.openfoam.org/download/suse.php

I think openmpi 1.7.2 was installed with OpenSuse in /usr/lib64/mpi/gcc/openmpi/share/

My entries in OpenFoams etc/bashrc were

#- MPI implementation:
# WM_MPLIB = SYSTEMOPENMPI | OPENMPI | MPICH | MPICH-GM | HPMPI
# | GAMMA | MPI | QSMPI | SGIMPI
export WM_MPLIB=SYSTEMOPENMPI

This combination produced that error:

It looks like opal_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

opal_shmem_base_select failed
--> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer)

opal_init failed
--> Returned value Error (-1) instead of ORTE_SUCCESS:

Second step:
I've tried to install openmpi 1.6.3 (from ThirdParty-2.2.2 package). I had to download that package first. I've received a list of error messages because I set path variables wrong. I did not know which one.

Third Step:
I uninstalled openmpi 1.6.3 and installed a save version: openmpi 1.4.5
I've built that version in /usr/local and changed the file /opt/OpenFOAM-2.2.2/etc/config/settings.sh

OPENMPI)
#export FOAM_MPI=openmpi-1.6.3
export FOAM_MPI=openmpi-1.4.5
# optional configuration tweaks:
_foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config/openmpi.sh`

#export MPI_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$FOAM_MPI
#export MPI_ARCH_PATH=/usr/local/lib64/openmpi
export MPI_ARCH_PATH=/usr/local

# Tell OpenMPI where to find its install directory
export OPAL_PREFIX=$MPI_ARCH_PATH

_foamAddPath $MPI_ARCH_PATH/bin

# 64-bit on OpenSuSE 12.1 uses lib64 others use lib
_foamAddLib $MPI_ARCH_PATH/lib$WM_COMPILER_LIB_ARCH
_foamAddLib $MPI_ARCH_PATH/lib

_foamAddMan $MPI_ARCH_PATH/share/man
;;

That worked after I've followed the steps suggested by linnemann
post
only the second part:
cd $FOAM_SRC/Pstream/dummy wclean cd ../mpi wclean cd .. ./Allwmake
But it is not satisfying. Especially now since you have a openmpi 1.6.x version with openFoam 2.2.2 running.

Which version did you install? 1.6.3 or 1.6.5?

Third step:
I would like to make at least openmpi 1.6.x running!



Aylalisa

[alexeym]

Are you sure you've installed 12.3 and not 13.1? Can you post contents of /etc/SuSE-release?

Here is rpm -qi openmpi:

Name        : opempi
Version     : 1.6
Release     : 3.1.2
Architecture: x86_64

I wasn't able to find openmpi 1.7.x in 12.3 repositories (release and updates).

[aylalisa]

I am sorry! I mixed it up!

Your are right:
I've installed OpenSUSE 13.1 (SORRY)


Is there a way to make openmpi 1.6 or 1.7 running with OS 13.1?

I've corrected my posts!

[alexeym]

Well, as there is no binary distribution for 13.1, I guess you'd better build OpenFOAM from source (http://openfoam.org/download/source.php).

Cause I don't have openSUSE 13.1 right now I can not say if there will be any problems with installation from source (though under OS X and openmpi 1.7.2 everything was OK).

[aylalisa]

Do I have to uninstall the current version first?

[alexeym]

You can keep it if you want Or remove if it bothers you.

After building from sources you will be using bashrc from the new installation. Though you have to remove 'source <old bashrc location>' from your shell initialization files otherwise it will mess up environmental variables with broken version.

[aylalisa]

I've build from sources, as you suggested and followed the steps in the installation guide. Took some time.
Finally I've copied the tutorial files in the users OpenFoam directory and tried to test OpenFoam:

blockMesh
icoFoam
paraFoam


typing 'blockMesh' in the cavity test case gives that message:

If 'BlockMesh' is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf BlockMesh

I've built from source files, in /opt/OpenFOAM-2.2.2 but can not start it from within the users home account. Why?


If I do the RPM Pack installation in /opt/ everything works fine! After copying the tutorials in the user's OF directory I can start 'blockMesh' etc.

At the moment I can not test openmpi 1.6.3 because OF doesn't run anymore.


foamInstallationTest delivers:

Checking basic setup...
-------------------------------------------------------------------------------
Shell: bash
Host: linux-0ze2
OS: Linux version 3.11.6-3-desktop
-------------------------------------------------------------------------------


Checking main OpenFOAM env variables...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory Valid Crit
-------------------------------------------------------------------------------
$WM_PROJECT_INST_DIR /opt yes yes
$WM_PROJECT_USER_DIR /home/lisa/OpenFOAM/lisa-2.2.2 yes no
$WM_THIRD_PARTY_DIR /opt/ThirdParty-2.2.2 yes yes
-------------------------------------------------------------------------------


Checking the OpenFOAM env variables set on the PATH...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory Valid Path Crit
-------------------------------------------------------------------------------
$WM_PROJECT_DIR /opt/OpenFOAM-2.2.2 yes yes yes

$FOAM_APPBIN ...-2.2.2/platforms/linux64GccDPOpt/bin yes yes yes
$FOAM_SITE_APPBIN .../2.2.2/platforms/linux64GccDPOpt/bin no no
$FOAM_USER_APPBIN ...-2.2.2/platforms/linux64GccDPOpt/bin no no
$WM_DIR /opt/OpenFOAM-2.2.2/wmake yes yes yes
-------------------------------------------------------------------------------


Checking the OpenFOAM env variables set on the LD_LIBRARY_PATH...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory Valid Path Crit
-------------------------------------------------------------------------------
$FOAM_LIBBIN ...-2.2.2/platforms/linux64GccDPOpt/lib yes no yes
$FOAM_SITE_LIBBIN .../2.2.2/platforms/linux64GccDPOpt/lib no no
$FOAM_USER_LIBBIN ...-2.2.2/platforms/linux64GccDPOpt/lib no no
$MPI_ARCH_PATH ...2/platforms/linux64Gcc/openmpi-1.6.3 yes no yes
-------------------------------------------------------------------------------


Third party software
-------------------------------------------------------------------------------
Software Version Location
-------------------------------------------------------------------------------
flex 2.5.37 /usr/bin/flex
gcc 4.8.1 /usr/bin/gcc
gzip 1.6 /usr/bin/gzip
tar 1.26 /bin/tar
icoFoam
WARNING: Conflicting installations:
OpenFOAM settings : /opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/bin/icoFoam
current path :
CRITICAL ERROR

-------------------------------------------------------------------------------


Summary
-------------------------------------------------------------------------------
Base configuration ok.
The foam installation contains 1 critical error(s).

Review the output for warning messages and consult
the installation guide for troubleshooting.

Done

What can I do to be able to use OF from within:
~/OpenFOAM/lisa-2.2.2/run/tutorials/incompressible/icoFoam/cavity


Can you provide advice

Aylalisa

[alexeym]

I wish I were a telepath but it's not a case.

1. Did you uninstall your previous binary OpenFOAM installation?

2. Where did you install new version? (if the answer for the first question if no and you've done source install in /opt/OpenFOAM-2.2.2 you'll have conflicting installations; I think this is actually a case cause there is 'WARNING: Conflicting installations' in foamInstallationTest)

3. Can you show your bashrc? What MPI library do you use?

4. Do you build OpenFOAM for yourself or for shared environment? If it's just for you, why don't you install it in $HOME/OpenFOAM (which is the easiest way of building OpenFOAM)?

If 'BlockMesh' is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf BlockMesh

This means that it can not find blockMesh in PATH. And it means the you did not sourced OpenFOAM's bashrc.

[aylalisa]

Hi,

even if I run the risk of going down with my OpenFOAM boat I am somehow happy that you are not a telepath .

I've installed from source in /opt/
OpenFOAM-2.2.2.tgz
ThirdParty-2.2.2.tgz
Result: I could not start OpenFOAM from inside my home directory.
I really would like to know why it does not work. After my first OpenFOAM installation some time ago I came accross the same problem (but could not solve it). Do I have to set different variables manually? I would like to know the reason!


After that I've removed OpenFOAM and wanted to install again (RPM Packs) in /opt/.
Bevor I start I had assured myself that OpenFOAM was gone and /opt/ was empty.
BUT None of the three download, uncompress & install command lines from documentation guide were working! The message claimed that the programmes are already installed: scotch, ParaView, OpenFOAM.
sudo rpm -i http://www.openfoam.org/download/sus...0-1.x86_64.rpm --> programm already installed!????
How is that possible? /opt/ was empty (ls -l).
Finally I've looked for the programs in Yast and there I could find (and delete) them??!
The detail folder in YAST showed the location /opt/...?


At the moment - as u suggest - I build from source in my home directory. --> ~/OpenFOAM/
--> OpenFOAM-2.2.2.tgz
--> ThirdParty-2.2.2.tgz

content of my user bashrc (now):

test -s ~/.alias && . ~/.alias || true

source $HOME/OpenFOAM/OpenFOAM-2.2.2/etc/bashrc

With old installation in /opt/ (the one that didn't work from home directory):
I've definitely sourced OpenFOAMS /etc/bashrc in my users ~/.bashrc
the line was like that:

source /opt/OpenFOAM-2.2.2/etc/bashrc

Before I've discovered, that openmpi with new SUSE 13.1 and OF2.2.2 does not work anymore I've had OF installed in /opt/ as well and I could use it from my user's directory (bevor upgrade to SUSE 13.1 and OF.2.2.2 even with openmpi!).


Thanks for your time !

Aylalisa

[alexeym]

Hi,

I'd like to clarify things with removal of installed OpenFOAM. If you installed it using rpm (or zypper, or YaST), you have to remove it using the same utility. If you just delete directories it will create a mess (orphan entries in RPM database, so RPM will still think that packages are in place and will tell you that packages are already installed, though IIRC you can use --force flag to install the packages anyway).

When you were building OpenFOAM in /opt folder did you use root account? If so there might be certain mess with file permissions that created problems using OpenFOAM with user's account. AFAIR it was necessary to fix permissions after compilation when I was installing OF on cluster.

Still I didn't get if you're able to use OF now or not?

[aylalisa]

First of all I've installed OF in /opt/ --> rpm pack installation. As root. It worked without any problems, logged in as user, from my user directory. Only openmpi did not want to work.

Before reinstallation I've made that mistake and just removed the OF folder instead of using rpm utility.
This gruppy proceeding explains why I could not reinstall it again in the same folder. I could finally fix it.

When exactly do I have to set permissions during installation from source in /opt/ to make sure every user can start OF from its user account? It is possible that I have to install OF on a server to make it available for many users, then I have to install in /opt/ or /local/....and will face the same problem.

Situation now:
The source pack installation just finished. I've installed OP in ~/OpenFOAM/ as u suggested. OpenFOAM works again! Including openmpi 1.6.3 (which is included in ThirdParty-2.2.2.tgz)!

Remaining questions:
I would like to know why the original installation setup supplied the openmpi error I've posted in my first thread and I would like to know how I can change permissions after compilation to install OP in /opt/ (from source) without problems for different users.

Aylalisa

[alexeym]

Originally Posted by aylalisa

Remaining questions:
I would like to know why the original installation setup supplied the openmpi error I've posted in my first thread and I would like to know how I can change permissions after compilation to install OP in /opt/ (from source) without problems for different users.

1. The problem with OpenMPI can be lack of ABI compatibility between OpenMPI libraries in openSUSE 12.3 and 13.1. I.e. software built with OpenMPI 1.6 (version of the library in openSUSE 12.3) can't use OpenMPI 1.7 (version of library in openSUSE 13.1)

2. This depends on the level of control you'd like to have over users And I'll assume below that the problem was really in permissions and not in file mess between old binary installation and new installation from sources (when you were trying to install OF into /opt).

The easiest way is to install OF in /opt from sources using root account and then execute something like (assuming you've installed OF into /opt/OpenFOAM-2.2.2 and /opt/ThirdParty-2.2.2)

chmod -R a+r /opt/OpenFOAM-2.2.2 /opt/ThirdParty-2.2.2
find /opt/OpenFOAM-2.2.2 -type f -executable -exec chmod a+x {} +
find /opt/ThirdParty-2.2.2 -type f -executable -exec chmod a+x {} +

This will allow all users to read and execute files in these directories.

Or you can create group openfoam for the users who will be allowed to use OF. Then change gid of the files in OpenFOAM-2.2.2 to open foam with

chown -R :openfoam /opt/OpenFOAM-2.2.2 /opt/ThirdParty-2.2.2

Then allow files in these directories read only by owner (root) and group (openfoam) with

chmod -R ug=r,o=u-rwx /opt/OpenFOAM-2.2.2 /opt/ThirdParty-2.2.2
find /opt/OpenFOAM-2.2.2 -type f -executable -exec chmod ug+x {} +
find /opt/ThirdParty-2.2.2 -type f -executable -exec chmod ug+x {} +

After these commands only users in openfoam group will be able to use OpenFOAM.

You might want to separate users in just OF users and OF developers, create two difference groups, and set permissions accordingly. In this case you'll need to separate access to binaries and sources.

(There might be errors in provided commands as I don't have access to openSUSE installation right now to test them)

[alexeym]

Hi,

Today I have an access to my OpenSUSE 13.1 installation and I repeated installation process with root account. Here is my findings:

1. I had clean install of OpenSUSE 13.1
2. I installed OF to /opt/OpenFOAM. It was more or less straightforward:
- Unpack OpenFOAM-2.2.2.tgz and ThirdParty-2.2.2.tgz to /opt/OpenFOAM
- Modify bashrc

Installation folder

foamInstall = /opt/$WM_PROJECT

Usage of system installed OpenMPI (this is optional)

export WM_MPLIB = SYSTEMOPENMPI

- Went to /opt/OpenFOAM/OpenFOAM-2.2.2
- Run 'source etc/bashrc'
- Run ./Allwmake

That's it, OpenFOAM was built in /opt/OpenFOAM/OpenFOAM-2.2.2.

At this point user wasn't able to run cause while OpenFOAM-2.2.2 folder permissions was OK, ones for ThirdParty-2.2.2 folder was wrong, so it was necessary to run

chmod -R a+rX /opt/OpenFOAM/ThirdParty-2.2.2

After this user was able to run OF but not in parallel cause there was a problem with OpenMPI (with the same error message as you initially posted). This was resolved by installing openmpi-32bit and openmpi-devel (for some reason this package was omitted from instruction at openfoam.org, though I think instruction assumes that user will be using OpenMPI from ThirdParty) packages. I needed to repeat ./Allwmake step to compile MPI version of Pstream library.

And after all these user was able to run OF in parallel without any problems.

[alexeym]

Hi again

Yesterday I've got a suspicion that maybe the problem is not in OF binary distribution incompatibility with openSUSE 13.1 but with openmpi package of openSUSE 13.1.

So I've decided to check it with a fresh installation of openSUSE 13.1. And here's what I found.

As written in the installation guide at openfoam.org, I've installed opempi, run mpi-selector, logged out, logged in, and tried to do

mpirun -np 2 ls

and received

alexey@linux-09yn:~> mpirun -np 2 ls
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------

Error just like yours.

After I've installed openmpi-devel package, and voila, the same command returned me listing of ~ twice:

alexey@linux-09yn:~> mpirun -np 2 ls
bin     Documents  Music     Public       Templates
Desktop  Downloads  Pictures  public_html  Videos
bin     Documents  Music     Public       Templates
Desktop  Downloads  Pictures  public_html  Videos

After that I've proceeded with installation as described on the site. And that's it. User can use OpenFOAM in serial and parallel manner without any problems.

I'm sorry my unfamiliarity with openSUSE's idiosyncrasies lead to all this install/uninstall/build/rebuild cycle

[aylalisa]

Hi Alexey,

After that I've proceeded with installation as described on the site. And that's it. User can use OpenFOAM in serial and parallel manner without any problems.

Do you mean the description on OF- or MPI-site?

There are not enough 'likes'! I would give you three for your each of your answers.

Mega-Thanks!

Aylalisa

[alexeym]

I meant openfoam.org, i.e. for normal installation on openSUSE 13.1 you need to install openmpi (mentioned on openfoam.org) and openmpi-devel (not mentioned) to be able to run parallel jobs. Actually you must have for some reason openmpi-devel to run parallel job on openSUSE 13.1.