[cdm]

I've recently compiled foam-extend-3.0 from source and distributed to a cluster where we have different versions of OpenFOAM already installed from the Ubuntu repo. We've been able to run any version of OF with no issue by sourcing the appropriate version's bashrc on the host and the appropriate arguments are passed to load that version on the slave machines when mpirun is called.

However, this is not occuring in foam-extend. I have to manually source the /opt/foam-extend-3.0/etc/bashrc in the slave machines' $HOME/.bashrc on the first line; however, this prevents us from running other versions of OF since every mpirun call automatically sources foam-extend on the slaves (overwriting the mpirun-called version sourcing).

1. Is there a settings file I need to modify in foam-extend-3.0 to source the bashrc upon mpirun call? Or is there a way of passing the source command during the mpirun call?

2. On a semi related note, foamJob does not appear to work with foam-extend-3.0, with an error thrown:

/opt/foam-extend-3.0/bin/foamExec: 398: /opt/foam-extend-3.0/etc/bashrc: [[: not found
/opt/foam-extend-3.0/bin/foamExec: 21: export: -pthread: bad variable name

Thank you for any help.

[cdm]

Further information:

I can run both single-core with icoFoam and in parallel on a local machine with

mpirun -np 6 icoFoam -parallel

However, when I run with a machines file pointing to an external node

mpirun --hostfile machines -np 6 icoFoam -parallel

foam-extend-3.0 fails to find the executable:

--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not find an executable:

Executable: icoFoam
Node: cfd16

while attempting to start process rank 0.
--------------------------------------------------------------------------

[cdm]

I've tried installing the pre-compiled foam-extend-3.0 binaries as well, but there is no change in the behaviour. I've tried passing environment variables through the -x argument in mpirun to no avail. I can't piece together the behaviour difference between our openfoam installations and the foam-extend installation to determine why the former run flawlessly and the latter won't run at all in parallel.

[cdm]

Revisiting this issue as it remains unsolved:

Checking the environment variables for OF221, it appears that SYSTEMOPENMPI is set for $WM_MPLIB while for FE3 OPENMPI is set. $MPI_ARCH_PATH points to the ThirdParty package implementation of MPI for FE3. Now, I'm fine with using the newer version of OpenMPI but why will the environment variables not properly source under this version? The file structure, paths, users, hardware -- everything is identical across nodes. The /usr/lib/openmpi implementation in OF211/221/230/etc. works fine...

I cannot fathom me being the only person with this problem, though, and I would appreciate someone with a bit more knowledge helping me out.

[Kojote]

Originally Posted by cdm

Revisiting this issue as it remains unsolved:

Checking the environment variables for OF221, it appears that SYSTEMOPENMPI is set for $WM_MPLIB while for FE3 OPENMPI is set. $MPI_ARCH_PATH points to the ThirdParty package implementation of MPI for FE3. Now, I'm fine with using the newer version of OpenMPI but why will the environment variables not properly source under this version? The file structure, paths, users, hardware -- everything is identical across nodes. The /usr/lib/openmpi implementation in OF211/221/230/etc. works fine...

I cannot fathom me being the only person with this problem, though, and I would appreciate someone with a bit more knowledge helping me out.

Hi

I have the same problem like you !! OF2.3 is working fine with mpi,foamJob over Infinband. But 3.1 isn't.

Br

Christian

[cdm]

I still haven't been able to get it working, unfortunately. Would be nice to get some input from someone with more expertise on OpenMPI or this particular issue. I'll let you know if I ever figure it out, Kojote.

[wyldckat]

Greetings to all!

I should be able to help, but I need more information, namely:

1. What were the exact installation instructions that were followed by each one of you?
2. @cdm: Any chance you can try installing the latest foam-extend 3.1 instead of 3.0?

Regarding the first post, it seems that foamExec or foamJob in foam-extend 3.0 is broken. Which is why I'm suggesting that you try installing 3.1, which with any luck has this issue already fixed.

In addition to the above, try running with the following command:

mpirun --hostfile machines -np 6 /opt/foam-extend-3.0/bin/foamExec icoFoam -parallel

Best regards,
Bruno

[cdm]

Hi Bruno,

Thank you for taking an interest in this issue.

I have recently been trying to get foam-extend-3.1 to build using the instructions here; however, no more luck than with fe3. I still can't get parallel to work. I should note I've attempted to compile on multiple machines and multiple methods, including compiling in the local user directory and running from there, compiling in the local user directory and moving the files to /opt/ and changing permissions, and compiling as root in /opt/foam/ directly (all approaches using default ThirdParty OPENMPI settings). None of these methods differed in terms of correcting the mpi issues described above. If you have any ideas, please let me know.


Anyways, I have compiled this morning under root in the /opt/ directory with the following prefs.sh settings (to use SYSTEMOPENMPI instead):

FOAM_VERBOSE=1
compilerInstall=System
export WM_MPLIB=SYSTEMOPENMPI
export QT_THIRD_PARTY=1
export QT_BIN_DIR=/usr/bin

However, when I run:

foamJob -p -s icoFoam

I get the following:

Parallel processing using SYSTEMOPENMPI with 6 processors
Executing: mpirun -np 6 /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh

Followed by the verbose output of the call to settings.sh and aliases.sh.

Using mpirun as you suggest above yields similar results:

mpirun --hostfile machines -np 6 /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel

I get the error:

/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh

Again followed by verbose output of settings.sh and aliases.sh along with a couple more calls to the bashrc.

I tried running ./bin/foamInstallationTest and all was well except a warning regarding conflicting gcc versions, but I'm not sure this is relevant to the issue at hand since everything compiles and runs fine locally.

Third party software
-------------------------------------------------------------------------------
Software Version   Location 
-------------------------------------------------------------------------------
WARNING: gcc version does not match gcc supplied with this release of foam-extend
         Supplied version: 4.3.3
         User version    : 4.6.3
         Minimum required: 4.3.1

gcc      4.6.3    
WARNING:  Conflicting installations:
          foam-extend settings        : /bin/gcc
          current path             : /usr/bin/gcc
          CRITICAL ERROR

[wyldckat]

Hi cdm,

OK, I've finally managed to look into this in detail, report the issue and push a fix and respective merge request:

• Bug report: https://sourceforge.net/p/openfoam-e...ndrelease/252/
• Merge request: https://sourceforge.net/p/openfoam-e...e-requests/11/

Now... how am I going to explain how you can pull this fix?..
OK, the "simplest" way is to rely on git, so please follow these steps:

foam
git remote add wyldckat git://git.code.sf.net/u/wyldckat/foam-extend-3.1
git pull wyldckat BUGFIX/Issue252

The first command is an alias command and should jump to the main folder for the foam-extend source code.
This is the quickest and will solve most of the problems.

But there is one more fix that has to be manually fixed:

1. Edit this file:
   

   ThirdParty/packages/openmpi-1.6.5/platforms/linux64GccDPOpt/etc/openmpi-1.6.5.sh

2. Change these two lines:
   

   export PINC=$OPENMPI_COMPILE_FLAGS
   export PLIBS=$OPENMPI_LINK_FLAGS

3. To this:
   

   export PINC="${OPENMPI_COMPILE_FLAGS}"
   export PLIBS="${OPENMPI_LINK_FLAGS}"

4. Save the modified file.

And that's it! All patched up
Both foamExec and foamJob should now work as intended. You won't even have to start a new terminal

Best regards,
Bruno

[cdm]

Bruno,

I pulled the fix from your git and modified the file as outlined above, then distributed this copy to two machines. I'm still receiving the following error upon running foamJob or mpirun with a hostfile specified:

Parallel processing using OPENMPI with 6 processors
Executing: mpirun -np 6 -hostfile machines /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found

I can still run on either machine locally with:

mpirun -np 6 icoFoam -parallel | tee log

Appears to be similar to the issue mentioned in the ticket you created regarding the shell. Any ideas?

[wyldckat]

Hi cdm,

Originally Posted by cdm

Parallel processing using OPENMPI with 6 processors
Executing: mpirun -np 6 -hostfile machines /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found

[...]

Appears to be similar to the issue mentioned in the ticket you created regarding the shell. Any ideas?

It looks to me like that something went very wrong with the "git pull" command If you still are getting that error, then that means that the changes done through git were not applied correctly.

Perhaps you only applied the last change from my repository/branch? There are 3 commits needed from there, not just the last one.

If the commands had worked as intended, then running this command should give you the complete change list:

git diff origin/master

Attached is the full change list that the command above should give you.

If you prefer to apply the attached changes, you can use this command (after unpacking):

patch -p1 < changes.patch

If you prefer, you can download the files that need to be fixed from the following locations:

• "bin/foamExec": http://sourceforge.net/u/wyldckat/fo...e/bin/foamExec
• "etc/bashrc": http://sourceforge.net/u/wyldckat/fo...ree/etc/bashrc

Best regards,
Bruno