[CFDnewbie147]

Hello together,

I'm trying to set up OpenFOAM 2.2.1 on a openSUSE 10.3 Cluster(x86 -64) to do some cfd-work for my study.

I have downloaded the Source Packs.tgz for OpenFOAM 2.2.1 and ThirdParty 2.2.1 and have, as descriped in the OF Source Pack Installation Guide, unpacked them in my $HOME/OpenFOAM directory.

Because it's running in tcsh I've added " alias opf 'source $HOME/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc' " to my .tcsh- file.

Then i want to use the Intel compiler(13.1.2), so I've changed the following in my cshrc- file in OpenFoam/etc:

#- Compiler:
# WM_COMPILER = Gcc | Gcc43 | Gcc44 | Gcc45 | Gcc46 | Clang | Icc (Intel icc)
setenv WM_COMPILER icc
setenv WM_COMPILER_ARCH # defined but empty
unsetenv WM_COMPILER_LIB_ARCH

I thought this shuld be all and i did ./Allwmake in my OpenFOAM directory. But after compiling in the "xterm window" I do the foamInstallationTest it looks like the picture "fehler"...

What's going wrong or what actions have i forgotten?
Please help me!!!

Thank u for ur replies...

[wyldckat]

Greetings CFDnewbie147 and welcome to the forum!

Run Allwmake like this:

./Allwmake > make.log 2>&1

It will log the build process into the file "make.log".
Then look for the first occurrences of "Error ", that is if your system is in English; don't forget to not use the quotes in the search, but make sure you do use the space after the word 'error'.

If you can't figure out what the problem is, search and replace any sensitive information on that "make.log" file. Then compress it and attach the compressed file to your next post.

Best regards,
Bruno

[CFDnewbie147]

Thank you for your answer Bruno.

I've given the job to my supervisior, he's also the admin of the cluster. We had to change alle the make files and had to update a lot programms like gcc, qt and so on...

Now it's running on the frontends of the cluster and now it's a bit tricki, to run it via PBS (Portable batch system) on the processors of each node. Have u any suggestions for this problem?

Best regards and thank u for your help!

[wyldckat]

Hi CFDnewbie147,

The problem with "PBS" is that there are several variants of PBS and each one has it's own way of setting up the number of cores and machines to be used.

The best you can do is:

1. Check with your systems administrator what is the conventional way to launch applications on your cluster, using the PBS you've got, in order to define the number of cores and machines to be assigned to each job.
2. Then search here on the forum and online for people that are using a similar PBS job nomenclature.
3. Among those, see if any of them are using OpenFOAM.

Best regards,
Bruno

[CFDnewbie147]

Hello,

I looked for the version of PBS we are using and it's TORQUE, i hope u know it.

The preprocessing is done on the frontend of the cluster and not on the nodes: the blockMesh and decomposePar that it should run on 8 processors, this means, that only one node is running with it's 8 processors on it.

I'm trying to parallize the cavity tutorial:

Here's my qsk.proc file which i push in the queu with qsub qsk.proc after doing the preprocessing:

#!/bin/sh
###############################################################
##
##  script qsub.opf for starting OpenFOAM within pbs-System on server...
##  
##
##  Author: ##  Date:   19.08.13
##
##  in this script: PBS-settings are done in script
##  start job with 'qsub qsk.opf'
##
##- Basic PBS-options are: ----------------
##  -l: ressources (ppn: procs/nodes; ....)
##  -q: queue to be used:
##  -N: for name of pbs-job:
##  -M,-m: for Email-notification:
##  -oe: log-files    (needs rcp/rsh?)
##  -S: is used due to a bug in the 'normal' shells
##
#######PBS-Definitions must'!preceed all other stuff!:
#
#PBS -N OpenFOAM_test
#PBS -l nodes=1:ppn=8
#PBS -q batch1
#PBS -S /bin/sh
#
#-----general user definitions:
#
#source OpenFOAM
#. /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc
#
PROG="Processing"
#
####################### end of user defs. #####
#
## look for no. of processors:
PROCS=`wc -l < $PBS_NODEFILE`
#
##### set intel-libs (needed if qsub -S... is executed)
##if [ -z "${LD_LIBRARY_PATH}" ]
##then
##  LD_LIBRARY_PATH="/opt/intel/cc_80/lib"
##  export LD_LIBRARY_PATH
##else
##  LD_LIBRARY_PATH="/opt/intel/cc_80/lib:$LD_LIBRARY_PATH"
##  export LD_LIBRARY_PATH
##fi
#
##if [ -z "${LD_LIBRARY_PATH}" ]
##then
##  LD_LIBRARY_PATH="/opt/intel/fc_80/lib"
##  export LD_LIBRARY_PATH
##else
##  LD_LIBRARY_PATH="/opt/intel/fc_80/lib:$LD_LIBRARY_PATH"
##  export LD_LIBRARY_PATH
##fi
#
##### set intel-libs (needed if qsub -S... is executed)
echo initial LD_LIBRARY_PATH was $LD_LIBRARY_PATH
## if gcc: comment it out
LD_LIBRARY_PATH="/opt/intel/composerxe/lib/intel64"
export LD_LIBRARY_PAT
#
##### set actual mpi-path:
PATH=/opt/openmpi-intel/bin:$PATH
export PATH
MPI_HOME=/opt/openmpi-intel
export MPI_HOME 
## for openmpi:
LD_LIBRARY_PATH="/opt/openmpi-intel/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH
#
#### run mpi 
cd $PBS_O_WORKDIR
echo "WorkingDirectory is: " $PBS_O_WORKDIR
echo "running:" `which mpirun` " with lib: " $LD_LIBRARY_PATH 
echo "Job is running on:" 
cat $PBS_NODEFILE 
## env > env-vars
#
##mpirun -nolocal -np $PROCS -machinefile $PBS_NODEFILE $PROG
##Note: 1) if complete Path is given for mpirun, MPI_HOME nad LD_LIBRARAY_PATH is not used!
##      2) don't use the machinefile-def., openmpi and Torque arrange this topic by intern comm.
/opt/openmpi-intel/bin/mpirun -np $PROCS $PROG xx

This file starts the Processing file which looks like:

#!/bin/sh
cd ${0%/*} || exit 1    # run from this directory
# Source tutorial run functions
. /opt/OpenFOAM/OpenFOAM-2.2.1/bin/tools/RunFunctions
# run icoFoam on front-nd
runParallel icoFoam 8

Then i get following logfiles:
log.icoFoam:

--------------------------------------------------------------------------
Failed to find the following executable:
Host:       magnus24
Executable: icoFoam
Cannot continue.
--------------------------------------------------------------------------

log OpenFOAM_test.e12936:

[magnus24:31966] [0,0,2] ORTE_ERROR_LOG: Not found in file odls_default_module.c at line 1191
[magnus24:31966] [0,0,2] ORTE_ERROR_LOG: Not found in file orted.c at line 626

log OpenFOAM_test.o12936:

initial LD_LIBRARY_PATH was
WorkingDirectory is:  /.../.../.../.../OpenFOAM/cavity_parallel_knoten
running: /opt/openmpi-intel/bin/mpirun  with lib:  /opt/openmpi-intel/lib:/opt/intel/composerxe/lib/intel64
Job is running on:
magnus24.....
magnus24.....
magnus24.....
magnus24....
magnus24....
magnus24....
magnus24....
magnus24....
Running icoFoam in parallel on /.../.../.../.../OpenFOAM/cavity_parallel_knoten using 8 processes
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run

It looks like, that each processor on the node "magnus24" tries to do the icoFoam completely and not only for the decomposed part. How can i fix this, that every processor only does the word for one part of the whole simulation???

Please help me!
Thank u for your replies!

[wyldckat]

Hi CFDnewbie147,

If you are using PBS and by looking at the way your script is made, you should not use the "Processing" script file like that.

If you Google for:

OpenFOAM PBS

you'll see that people call the solver directly. In your case, the last or near last line in "qsk.proc" should be like this:

/opt/openmpi-intel/bin/mpirun -np $PROCS icoFoam -parallel

This is because the runParallel function being used inside the "Processing" script, which also calls mpirun, leading to an attempt of launching 64 icoFoam processes, although only the first 8 (might) work.

Best regards,
Bruno

[CFDnewbie147]

Hello Bruno,

thanks for your answer.
I added to my qsk.proc:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc

and changed the last line to:

/opt/openmpi-intel/bin/mpirun -np $PROCS icoFoam -parallel xx

Now i don't start the Processing- script, but run the code icoFoam directly, as mentioned above.

But now i get the following error message in my OpenFOAM_test.e12942- file:

/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 35: setenv: command not found
/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 36: setenv: command not found
/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 61: syntax error near unexpected token `setenv'
/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 61: `if ( ! $?FOAM_INST_DIR ) setenv FOAM_INST_DIR $foamInstall'
[magnus16:16418] [0,0,1] ORTE_ERROR_LOG: Not found in file odls_default_module.c at line 1191
--------------------------------------------------------------------------
Failed to find the following executable:
Host:       magnus16
Executable: icoFoam
Cannot continue.
--------------------------------------------------------------------------
[magnus16:16418] [0,0,1] ORTE_ERROR_LOG: Not found in file orted.c at line 626

:/

What's wrong again?
I hope u can help me again...Thank You very much!

[wyldckat]

Quick answer - change:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc

To:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/bashrc

[CFDnewbie147]

Sorry, but doesn't work...

Then i get the following error message in my OpenFOAM_test.e12980:

[magnus16:14346] mca: base: component_find: unable to open paffinity hwloc: file not found (ignored)
[magnus16:14346] mca: base: component_find: unable to open errmgr default: file not found (ignored)
[magnus16:14346] mca: base: component_find: unable to open rml oob: file not found (ignored)
[magnus16:14346] [NO-NAME] ORTE_ERROR_LOG: Not found in file runtime/orte_init_stage1.c at line 182
[magnus16:14346] *** Process received signal ***
[magnus16:14346] Signal: Segmentation fault (11)
[magnus16:14346] Signal code: Address not mapped (1)
[magnus16:14346] Failing at address: 0xfffffff3
[magnus16:14346] [ 0] /lib64/libpthread.so.0 [0x2b6e5ba7bfb0]
[magnus16:14346] [ 1] /lib64/libc.so.6(strlen+0x20) [0x2b6e5bcfebb0]
[magnus16:14346] [ 2] /lib64/libc.so.6(_IO_vfprintf+0x37bd) [0x2b6e5bcce84d]
[magnus16:14346] [ 3] /lib64/libc.so.6 [0x2b6e5bccf8b3]
[magnus16:14346] [ 4] /lib64/libc.so.6(_IO_vfprintf+0x9e) [0x2b6e5bccb12e]
[magnus16:14346] [ 5] /opt/openmpi-intel/lib/libopen-pal.so.0(opal_show_help+0x485) [0x2b6e5ad85455]
[magnus16:14346] [ 6] /opt/openmpi-intel/lib/libopen-rte.so.0(orte_init_stage1+0x10b) [0x2b6e5aaf2b03]
[magnus16:14346] [ 7] /opt/openmpi-intel/lib/libopen-rte.so.0(orte_system_init+0x7) [0x2b6e5aaf672b]
[magnus16:14346] [ 8] /opt/openmpi-intel/lib/libopen-rte.so.0(orte_init+0x4f) [0x2b6e5aaf2843]
[magnus16:14346] [ 9] /opt/openmpi-intel/bin/mpirun(orterun+0x1ab) [0x4042e5]
[magnus16:14346] [10] /opt/openmpi-intel/bin/mpirun(main+0x20) [0x403d88]
[magnus16:14346] [11] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b6e5bca6b54]
[magnus16:14346] [12] /opt/openmpi-intel/bin/mpirun [0x403cd9]
[magnus16:14346] *** End of error message ***
/var/spool/torque/mom_priv/jobs/12980.karma.SC: line 70: 14346 Segmentation fault      /opt/openmpi-intel/bin/mpirun -np $PROCS icoFoam -parallel

My OpenFOAM_test.o12980 says:

Run started at
Wed Aug 28 07:30:13 CEST 2013
initial LD_LIBRARY_PATH was /opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/gperftools-svn/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/paraview-3.12.0/lib/paraview-3.12:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib64:/opt/intel/composerxe/lib/intel64:/opt/Python-2.7.2-infini-intel/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/qt-4.8.5/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/flex-2.5.37/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpc-1.0.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpfr-3.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gmp-5.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gcc-4.8.1/lib64:/home/zachjoer/OpenFOAM/zachjoer-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/site/2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/dummy
WorkingDirectory is:  /z/pro/cfdtmp03/zachjoer/OpenFOAM/cavity_parallel_knoten
running: /opt/openmpi-intel/bin/mpirun  with lib:  /opt/openmpi-intel/lib:/opt/intel/composerxe/lib/intel64
Job is running on:
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
Run completed at 
Wed Aug 28 07:30:13 CEST 2013

I don't know what's wrong, because if i make

echo $shell

i get

/bin/tcsh

So I've normally to use the cshrc of OpenFOAM i think...
My administrator told me, that there is on each node OpenFOAM or the link... so why does it not work???

[wyldckat]

Hi CFDnewbie147,

I didn't manage to look with much attention into this the other day and only now did I managed to look into this...

Basically the problem is that in the "qsk.proc" file, you have right at the start this line:

#!/bin/sh

which means that it will use either dash or bash (or similar) by default, which is why you were getting the error about "setenv" not existing.
If you change that line to:

#!/bin/csh

and use the original sourcing command you had:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc

then it should work!

Best regards,
Bruno

[CFDnewbie147]

Hello Bruno,

thanks for your answer but it doesn't work...I get the following error output:

PROCS=8: Command not found.
LD_LIBRARY_PATH=/opt/intel/composerxe/lib/intel64: Command not found.
export: Command not found.
PATH=/opt/openmpi-intel/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/paraview-3.12.0/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/flex-2.5.37/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gcc-4.8.1/bin:/home/zachjoer/OpenFOAM/zachjoer-2.2.1/platforms/linux64Gcc48DPOpt/bin:/opt/OpenFOAM/site/2.2.1/platforms/linux64Gcc48DPOpt/bin:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/bin:/opt/OpenFOAM/OpenFOAM-2.2.1/bin:/opt/OpenFOAM/OpenFOAM-2.2.1/wmake:/usr/local/bin/:/home/zachjoer/util:/opt/intel/composer_xe_2013.4.183/bin/intel64:/opt/intel/composer_xe_2013.4.183/mpirt/bin/intel64:/opt/intel/composer_xe_2013.4.183/bin/intel64_mic:/opt/intel/composer_xe_2013.4.183/debugger/gui/intel64:/opt/torque/bin:/opt/netcdf-gcc/bin:/opt/openmpi-gcc/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib64/jvm/jre/bin:/opt/ofed/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/ofed/sbin:/usr/lib/qt3/bin: Command not found.
export: Command not found.
MPI_HOME=/opt/openmpi-intel: Command not found.
export: Command not found.
LD_LIBRARY_PATH=/opt/openmpi-intel/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/paraview-3.12.0/lib/paraview-3.12:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib64:/opt/intel/composerxe/lib/intel64:/opt/Python-2.7.2-infini-intel/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/qt-4.8.5/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/flex-2.5.37/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpc-1.0.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpfr-3.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gmp-5.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gcc-4.8.1/lib64:/home/zachjoer/OpenFOAM/zachjoer-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/site/2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/dummy:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.4.183/compiler/lib/intel64:/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64:/opt/intel/composer_xe_2013.4.183/mpirt/lib/intel64:/opt/intel/composer_xe_2013.4.183/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.4.183/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.4.183/tbb/lib/intel64/gcc4.1:/opt/openmpi-gcc/lib:/opt/netcdf-gcc/lib: Command not found.
export: Command not found.
PROCS: Undefined variable.

What's wrong again?
Best regards :-)

[wyldckat]

Hi CFDnewbie147,

OK, so from what I re-read in an attempt to remember what was discussed here, what I can see is that when you followed my instruction from post #8, you got closer to the solution.
But then the other idea from #10 ended up in a worse situation.


Now that I reviewed the output from your post #9, there is one clear indication that one very important detail has been ignored: you built OpenFOAM with Open-MPI 1.6.3 that is provided in OpenFOAM's ThirdParty folder, when you should in fact have indicated to use the system's MPI.
In other words, the variable "WM_MPLIB" should be set to "SYSTEMOPENMPI" and not "OPENMPI". This is defined inside OpenFOAM's "etc/bashrc" file.

You need to change the setting for that variable in said file, start a new terminal and run Allwmake once again. It will re-build only the libraries that depend on the new MPI.

Best regards,
Bruno

[CFDnewbie147]

Hello Bruno,

sorry for this late answer. I solved the problem. As u said, i used the system open mpi, but i have to use the mpi of OF.

So i deleted the lines which get the system open mpi. In my script, there's only the source OF-cshrc and the last line, run mpi..... and now it works!

Thank you for your help!!!
Best regards
CFDNewbie147