[ciberbob1987]

Hi, I'm a student of Politecnico di Torino, and I'm working with OpenFOAM on the HPC cluster of my university.
The system is equipped with ROCKS Cluster Linux Distribution 5.4.3 OS and SGE scheduler. I'm working via SSH and my account is obviously not SU.

Actually I don't know if this is the right way, anyway I have done these steps:

[~]snft3@cclix7$ cd Marco
[Marco]snft3@cclix7$ wget http://downloads.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz?r=http%3A%2F%2Fopenfoam.org%2Fdownload%2Fsource.php&ts=1329690292&use_mirror=dfn
[1] 18979
[2] 18980
[Marco]snft3@cclix7$ --2012-02-20 00:05:08--  http://downloads.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz?r=http%3A%2F%2Fopenfoam.org%2Fdownload%2Fsource.php
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://garr.dl.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz [following]
--2012-02-20 00:05:08--  http://garr.dl.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz
Resolving garr.dl.sourceforge.net... 193.206.140.34, 2001:760:ffff:b0::34
Connecting to garr.dl.sourceforge.net|193.206.140.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32518358 (31M) [application/x-tar]
Saving to: `OpenFOAM-2.1.0.tgz'

100%[================================================================>] 32,518,358  7.81M/s   in 4.2s    

2012-02-20 00:05:13 (7.31 MB/s) - `OpenFOAM-2.1.0.tgz' saved [32518358/32518358]


[1]-  Done                    wget http://downloads.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz?r=http%3A%2F%2Fopenfoam.org%2Fdownload%2Fsource.php
[2]+  Done                    ts=1329690292
[Marco]snft3@cclix7$ tar -xzf OpenFOAM-2.1.0.tgz
[Marco]snft3@cclix7$ mkdir OpenFOAM
[Marco]snft3@cclix7$ mv OpenFOAM-2.1.0 ./OpenFOAM/
[Marco]snft3@cclix7$ cd OpenFOAM
[Marco]snft3@cclix7$ export FOAM_INST_DIR=/share/apps/snft/snft3/Marco/OpenFOAM
[Marco]snft3@cclix7$ foamDotFile=$FOAM_INST_DIR/OpenFOAM-2.1.0/etc/bashrc
[Marco]snft3@cclix7$ [ -f $foamDotFile ] && . $foamDotFile



[Marco]snft3@cclix7$ cd $WM_PROJECT_DIR
[OpenFOAM-2.0.1]snft3@cclix7$ chmod +x Allwmake
[OpenFOAM-2.0.1]snft3@cclix7$ ./Allwmake | tee ../compile.log

After about an hour I accidentally aborted the compiling process.
In the log there were some errors e.g.

make[2]: Entering directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
/bin/sh: /share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/wmake/platforms/linux64Gcc/wmkdep: No such file or directory
make[2]: Leaving directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
make[2]: Entering directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
SOURCE=buoyantPimpleFoam.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-100 -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/thermophysicalModels/basic/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/turbulenceModels/compressible/turbulenceModel -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude -IlnInclude -I. -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64GccDPOpt/buoyantPimpleFoam.o
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude/globalIndexAndTransformI.H: In member function 'Foam::labelList Foam::globalIndexAndTransform::transformIndicesForPatches(const Foam::labelHashSet&) const':
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude/globalIndexAndTransformI.H:515: warning: converting to 'Foam::label' from 'double'
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude/readTimeControls.H: In function 'int main(int, char**)':
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude/readTimeControls.H:38: warning: unused variable 'maxDeltaT'
g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-100 -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/thermophysicalModels/basic/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/turbulenceModels/compressible/turbulenceModel -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude -IlnInclude -I. -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OSspecific/POSIX/lnInclude   -fPIC -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64GccDPOpt/buoyantPimpleFoam.o -L/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib \
	     -lmeshTools -lbasicThermophysicalModels -lspecie -lcompressibleTurbulenceModel -lcompressibleRASModels -lcompressibleLESModels -lfiniteVolume -lOpenFOAM -ldl   -lm -o /share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/bin/buoyantPimpleFoam
/usr/bin/ld: cannot find -lmeshTools
collect2: ld returned 1 exit status
make[2]: *** [/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/bin/buoyantPimpleFoam] Error 1
make[2]: Leaving directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
make[1]: *** [buoyantPimpleFoam] Error 2

Other ones are very similar.
I'm now trying to compile again, I'll post the log.
I want to know:

1. what kind of errors are these?
2. are they critical?
3. how to solve them?

Thank you. Marco

[wyldckat]

Greetings Marco and welcome to the forum!

AFAIK, Rocks 5.4 is based on CentOS 5.4, which means that you probably don't have decent Gcc compiler... or in other words, have a version older than Gcc 4.3.3.

In these situations where CentOS and relatives are involved, I generally point people to CentFOAM: http://sourceforge.net/apps/mediawik...itle=Main_Page

But if you want to waste more time trying to diagnose the issue:

• gcc -v

  will give you the version you've got installed. Remember, anything below 4.3.3 will not work out-of-the-box.
• To really see what goes wrong:

  ./Allwmake > make.log 2>&1

  This way the whole compilation is logged into the file "make.log".

Best regards,
Bruno

[ciberbob1987]

Hi Bruno, thank you for the reply. I have downloaded CentFOAM and made some tests, but something related to MPI don't work well.

In the user guide of the cluster (http://dauin-hpc.polito.it/?p=user-guide) is reported that the path of mpirun is /opt/openmpi/bin, but if I use it with

### Defining openmpi executable
MPIRUN=/opt/openmpi/bin/mpirun
#
#
### Set run directory
DIR_FOAM_RUN=/share/apps/snft/snft3/Marco/OpenFOAM/run
#
## Solver and case
SOLVER=snappyHexMesh
CASO=turbineSiting
#
source /share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.0.x/etc/bashrc
cd $DIR_FOAM_RUN
$MPIRUN $SOLVER -case $CASO -parallel

without any changes in OF config the system returns an error:

[compute-0-0.local:32745] mca: base: component_find: unable to open /share/apps/snft/snft3/Marco/OpenFOAM/ThirdParty-2.0.x/platforms/linux64Gcc45/openmpi-1.5.3/lib/openmpi/mca_ess_slurmd: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
/opt/openmpi/bin/mpirun: symbol lookup error: /share/apps/snft/snft3/Marco/OpenFOAM/ThirdParty-2.0.x/platforms/linux64Gcc45/openmpi-1.5.3/lib/openmpi/mca_plm_rshd.so: undefined symbol: orte_plm_base_rsh_launch_agent_setup

So, I tried to set WM_MPLIB=SYSTEMOPENMPI in bashrc and edit the settings.sh file from:

SYSTEMOPENMPI)
    # Use the system installed openmpi, get library directory via mpicc
    export FOAM_MPI=openmpi-system

    # Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI
    export PINC="`mpicc --showme:compile`"
    export PLIBS="`mpicc --showme:link`"
    libDir=`echo "$PLIBS" | sed -e 's/.*-L\([^ ]*\).*/\1/'`

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    export MPI_ARCH_PATH="${libDir%/*}"

    if [ "$FOAM_VERBOSE" -a "$PS1" ]
    then
        echo "Using system installed MPI:"
        echo "    compile flags : $PINC"
        echo "    link flags    : $PLIBS"
        echo "    libmpi dir    : $libDir"
    fi

    _foamAddLib     $libDir
    unset libDir
    ;;

to

SYSTEMOPENMPI)
    # Use the system installed openmpi, get library directory via mpicc
    export FOAM_MPI=openmpi-system

    # Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI
    export PINC="`/opt/openmpi/bin/mpicc --showme:compile`"
    export PLIBS="`/opt/openmpi/bin/mpicc --showme:link`"
    libDir=/opt/openmpi/lib

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    export MPI_ARCH_PATH=/opt/openmpi

    if [ "$FOAM_VERBOSE" -a "$PS1" ]
    then
        echo "Using system installed MPI:"
        echo "    compile flags : $PINC"
        echo "    link flags    : $PLIBS"
        echo "    libmpi dir    : $libDir"
    fi

    _foamAddLib     $libDir
    unset libDir
    ;;

But also in this case the system reports some errors (one for each processor):

--> FOAM FATAL ERROR: 
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

    From function UPstream::init(int& argc, char**& argv)
    in file UPstream.C at line 37.

FOAM exiting

Finally, I restored the default bashrc and settings.sh to use the OpenMPI provided with CentFOAM. Now the program runs (snappyHexMesh), but I can't use more than one node and before the execution some warnings are shown (see attached logs).

[wyldckat]

Hi Marco,

Those errors almost look painful...

You were going in the right direction with "WM_MPLIB=SYSTEMOPENMPI". All that was missing was the dedicated Pstream and PTScotch libraries:

1. Pstream:

   foam
   wmake all src/Pstream

2. PTScotch:

   foam3rdParty
   ./Allwmake

Well, running Allwmake once more on the OpenFOAM folder should do the trick as well, perhaps even better:

foam
./Allwmake

Best regards,
Bruno

[ciberbob1987]

Hi Bruno,
I will try as soon as possible your solution, but I have gcc version 4.1.2 and flex is not installed. Is it sufficient to compile third party without errors?

[wyldckat]

Hi Marco,

You should have gotten a shiny new gcc 4.4 or 4.5 with CentFOAM! For using strictly with Cent/OpenFOAM... i.e., while you have OpenFOAM's environment active (a shell where you're able to run OpenFOAM applications).

Best regards,
Bruno

[linnemann]

Hi

Why do you want to change the openmpi location?

centFOAM comes with its own openmpi install.

Youre trying to mix a version of openmpi (most likely compiled with gcc 4.1.2) with a newer version compiled with gcc 4.5 or higher.

This is bound to give you errors :-). If you want to use system openmpi you have to recompile OF.

If your cluster dont have infiniband, the provided openmpi in centFOAM should be just as good as any system openmpi.

EDIT: Just saw that your cluster indeed have inifiniband so you will have to recompile as Brunos suggest using the system openmpi.

Best

[wyldckat]

Hi Linnemann,

Originally Posted by linnemann

EDIT: Just saw that your cluster indeed have inifiniband so you will have to recompile as Brunos suggest using the system openmpi.

If CentFOAM delivers a complete installation (including the "lnInclude" folders with the correct time stamps), then it should only be necessary to compile Pstream and PTScotch, due to OpenFOAM's building structure

Best regards,
Bruno

[ciberbob1987]

Hi everybody,I have re-compiled the PStream code and now all seems to work fine.
First simulation with posousSimpleFoam runned in parallel on 7 CPU with no errors.
Many thanks to all, you saved me!