[romant]

Hej,

does anyone have any experience with installing openfoam on a debian etch system with the system provided libraries, namely

gcc = 4.1.2
openmpi = 1.1.4

and if not, does anyone know if those library versions are high enough to compile openfoam 1.6

[wyldckat]

Greetings Roman,

I have no experience with building OpenFOAM 1.6(.x) in Debian, but lots in Ubuntu, which derives from Debian

As for versions:

• you will have to use the gcc-4.3.3 that comes with the ThirdParty packages. You can either use the pre-compiled version that comes with the binary ThirdParty package or rebuild gcc-4.3.3 using the makeGcc script that is in the ThirdParty-1.6 folder. NOTE: the ThirdParty.General.gtgz package is missing 3 files for the MPFR library. You can find a patch for it in this thread: Compilation error for openfoam - 1.6
• as for OpenMPI, on this one I'm only guessing, but using the version that comes with the ThirdParty package should also be the best option. Although with OpenMPI there shouldn't be any problems with using different versions, since the base interface infrastructure shouldn't be different. So, if you want to use the system's OpenMPI version, you will have to edit the script settings.sh in $HOME/OpenFOAM/OpenFOAM-1.6/etc, and change the code pertinent to OpenMPI, so it won't use the version in the ThirdParty folder.

As for the rest, the instructions provided by OpenCFD in www.openfoam.com should be enough to build OpenFOAM.

Best regards,
Bruno

[romant]

Hej,

thank you. I got everything compiled and almost running. I am stuck with the parallel running though. I got myself the source of the MPI libs that the computer is running and compiled it with this changed in the settings.sh file.

Unfortunately, I get error messages when I try to run the simulation in parallel. Single thread does not pose a problem. But I don't really want to run anything on the master machine of the cluster.

CMD: mpirun -hostfile /home/roman/hostfile.uybOCy4QV -np 8 
compressibleInterFoam -parallel
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
     argv0-not-found
from the file:
     help-pls-base.txt
But I couldn't find any file matching that name.  Sorry!
--------------------------------------------------------------------------
mpirun noticed that job rank 0 with PID 0 on node "node0014" exited on 
signal 4.
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
     argv0-not-found
from the file:
     help-pls-base.txt
But I couldn't find any file matching that name.  Sorry!
--------------------------------------------------------------------------

Have you ever anyone ever seen such an error message and might be able to tell me what it is supposed to mean exactly?

[wyldckat]

Hi Roman,

I don't remember seeing an error like that, but by what it says, it seems to me it's looking for a help file, for it to report on the respective error message. It could be that the language files are missing in the MPI's installation.

As for testing the connections, try the parallel test utility. To build it,

Originally Posted by wyldckat

run:

cd  $WM_PROJECT_DIR
wmake applications/test/parallel

This will create an application named parallelTest in your user folder (more specifically, in $FOAM_USER_APPBIN). You can do this in both versions of OpenFOAM.

NOTE: when using OpenFOAM 1.6, there is a glitch that has been fixed in OpenFOAM 1.6.x; before running wmake, you will need to edit the file $WM_PROJECT_DIR/applications/test/parallel/parallelTest.C, search for the following lines (non lines contiguous):

                OPstream  toMaster(Pstream::masterNo());
...
            IPstream fromMaster(Pstream::masterNo());
...
                IPstream fromSlave(slave);
...
                OPstream toSlave(slave);

and change them respectively to:

                OPstream  toMaster(Pstream::scheduled,  Pstream::masterNo());
...
            IPstream fromMaster(Pstream::scheduled,   Pstream::masterNo());
...
                IPstream fromSlave(Pstream::scheduled, slave);
...
                OPstream toSlave(Pstream::scheduled, slave);

Then, go to your case has the decomposePar already done and run:

foamJob -p -s parallelTest

It should output something similar to this (source: my wiki page on openfoamwiki.net ):

Create  time

[1]
Starting transfers
[1]
[1] slave sending to master 0
[1] slave receiving from master 0
[0]
Starting transfers
[0]
[0] master receiving from slave 1
[0] (0 1 2)
[0] master sending to slave 1
End

[1] (0 1 2)
Finalising parallel run

If all goes well, you've gotten an output similar to this one above.

(source: post #19 from another recent thread)

The script foamJob should have created a file named log with the same output. It is only configured for using OpenMPI's mpirun, so adapt it to your needs if you're running another MPI application!
The host file that foamJob can use by default, is one of these files:

machines
hostfile
system/machines
system/hostfile

Either way, the parallelTest utility will help in isolating the error to a communication problem or firewall issue; because if it works, then it's because your case isn't properly divided into 8 cores or the environment for the spawned slave solvers doesn't have the OpenFOAM variables active.
As for the script foamJob, it has the advantage of requiring a lot less commands to type each time you need to run a parallel job
Now, like I said before, and assuming that you defined properly on how the case should be split in 8 sub-domains and you ran decomposePar. There are a few ways of setting the environment variables, whether:

• using a global .bashrc that sources OpenFOAM's bashrc (i.e., activates OpenFOAM's environment for all users, including root);
• or use an option in mpirun to tell it to send all local variables to the slave's environments... and this option I don't know by heart

Good luck and please don't forget to post if you get it working But if it still won't cooperate, feel free to ask

Best regards,
Bruno

[romant]

hej,

ok, i get the output plus something bad.

[node0014:05178] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0014:05177] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0014:05180] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0014:05179] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0014:05177] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0014:05178] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0014:05180] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0014:05179] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04624] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04626] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04623] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04625] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04626] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04624] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04625] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0015:04623] mca: base: component_find: unable to open: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)

then the second part is as you described

Build  : 1.6-f802ff2d6c5a
Exec   : parallelTest -parallel
Date   : May 05 2010
Time   : 18:32:28
Host   : node0014
PID    : 5177
Case   : /home/roman/OpenFOAM/roman-1.6/run/run250cmCondTankCluster
nProcs : 8
Slaves : 
7
(
node0014.5178
node0014.5179
node0014.5180
node0015.4623
node0015.4624
node0015.4625
node0015.4626
)

Pstream initialized with:
    floatTransfer     : 0
    nProcsSimpleSum   : 0
    commsType         : nonBlocking
SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

[3] 
Starting transfers
[3] 
[3] slave sending to master 0
[3] slave receiving from master 0
[1] 
Starting transfers
[1] 
[1] slave sending to master 0[2] 
Starting transfers
[2] 
[2] slave sending to master 0

[1] slave receiving from master 0
[2] slave receiving from master 0
[0] 
Starting transfers
[0] 
[0] master receiving from slave 1
[0] (0 1 2)
[0] master receiving from slave 2
[0] (0 1 2)
[0] master receiving from slave 3
[0] (0 1 2)
[0] master receiving from slave 4
[0] (0 1 2)
[0] master receiving from slave 5
[0] (0 1 2)
[0] master receiving from slave 6
[0] (0 1 2)
[0] master receiving from slave 7
[0] (0 1 2)
[0] master sending to slave 1
[0] [1] (0 1 2)
master sending to slave 2
[0] [2] (0 1 2)
master sending to slave 3
[0] [3] (0 1 2)
master sending to slave 4
[0] [4] 
Starting transfers
[4] 
[4] slave sending to master 0
[4] slave receiving from master 0
[4] (0 1 2)
master sending to slave [5] 
Starting transfers
[5] 
[5] slave sending to master 0
[5] slave receiving from master 0
[5] (0 1 2)
5
[6] 
Starting transfers
[6] 
[6] slave sending to master 0
[6] slave receiving from master 0
[0] [7] 
Starting transfers
[7] 
[7] slave sending to master 0
[7] slave receiving from master 0
master sending to slave [6] (06 1 2)

[7] (0 1 2)
[0] master sending to slave 7
End

Finalising parallel run

So, unfortunately, I don't have a clue what to do with the mca base component, or to which system this belongs.

I just found that it might be bug in the MPI version that i compiled, but I am not sure!

when this is done, i will def write a install article for debian etch

[wyldckat]

Hello again Roman,

Great! Communications work and I believe that the environment is properly set in all slaves. Your Debian box is missing the library libibverbs.so, so you could install it. Or you could try to use the precompiled OpenMPI that comes with OpenFOAM's ThirdParty binary package; or even try compiling it, using the script Allwmake at the ThirdParty-1.6 folder.

Now, if I'm not mistaken, MCA is part of the communications section of MPI. And libibverbs is an library for InfiniBand networking. You can check the forum for more info on mpirun or OpenMPI, or check your MPI's manual for command options. Just recently there has been a post about mca included in the command for calling mpirun: SimpleFoam run in Parallel post #4

Have you tried running your case with foamJob? That thing about libibverbs is just a warning, so it should work!

Best regards,
Bruno

[romant]

It seems that it is working with foamJob, but not with the wrapper supplied by the cluster. Which is good to know.

Thanks for the extensive help.

[romant]

here one can find the complete solution to the installation of openfoam 1.6 on debian etch in server mode, as it is used on clusters http://openfoamwiki.net/index.php/In...in_Server_Mode

[wyldckat]

Greetings Roman,

I just saw the post you made on the wiki. I'm astonished that a tweaked script of mine ended up on the wiki, without being me putting it there Ah the marvels of GPL Long live free source

But there is something I just want to make sure that it's working or not in your installation, specially since I'm the one responsible for the tweaked script! Did you need to change anything in settings.sh for gcc to be properly recognized? Are you sure you used the actual built gcc?
Please open a new terminal window, activate (if not sourced automatically) the OpenFOAM environment and confirm that you have the right gcc version up by running:

gcc -v

It should read in the last line of of the outputed text:

gcc version 4.3.3 (OpenFOAM)

Because that particular moded script version has a minor glitch, that may hinder the functionality of gcc, namely these three lines:

GMPROOT=${GMP_DIR}/platforms/$WM_ARCH$WM_COMPILER$WM_COMPILER_ARCH
MPFRROOT=${MPFR_DIR}/platforms/$WM_ARCH$WM_COMPILER$WM_COMPILER_ARCH
GCCROOT=${GCC_DIR}/platforms/$WM_ARCH$WM_COMPILER$WM_COMPILER_ARCH

Should actually be:

GMPROOT=${GMP_DIR}/platforms/$WM_ARCH$WM_COMPILER_ARCH
MPFRROOT=${MPFR_DIR}/platforms/$WM_ARCH$WM_COMPILER_ARCH
GCCROOT=${GCC_DIR}/platforms/$WM_ARCH$WM_COMPILER_ARCH

You can confirm that by comparing with the post #18 on that thread, although that particular post is for strict 64bit compilation of gcc (standard building allows multilib to be active, thus allowing to build both 32 and 64 bit applications with the same gcc version). At that time I completely forgot to fix the script that I posted, and now I can't fix it directly without making a new post
That particular issue on the first moded version is because I copy-pasted from the version I tweaked for cross-compiling OpenFOAM in Linux for running in Windows... and the two names collided (both paths were named linux), so I added $WM_COMPILER to the path for the build compiler (so it could be linuxGcc and linuxmingw32).


By the way, why didn't you build the OpenMPI 1.3.3 that comes in the ThirdParty package? It's more recent than 1.1.4! And the building process is already imbued in the Allwmake script in the ThirdParty-1.6 folder! Or is it incompatible with Debian Etch?

Best regards,
Bruno

[romant]

Hej,

yeah, your script worked great, thanks for that. I have gcc-4.3.3 (OpenFOAM) working here. So that is not a problem.

I didn't know at that point if the openMPI would work the way I intended it to work, so I basically chose the one that was already installed, but didn't seem to work correctly.

But right now, it works like a charm with the 1.1.4, I might try to update over the next couple of days, and see how it turns out. If it works, then I will just give it a try with a higher version of mpi and update the post.