[mahmoodn]

Is there any official tutorial or example regarding OpenFoam and PBS job scheduler (the qsub command)? I saw some pages on the web but it seems that they are trying to show how to solve a problem first.

I actually don't care about that since my job is to install (have already done) and check it users can submit their jobs to the cluster.

So, I need an example which shows how to submit a solved problem (meshing, ...) to the job scheduler.

Basically, I write a pbs script and put "ls" command in it and submit it to the scheduler to see if all nodes are working and communicating. So, I am searching for a similar simple command with respect to OpenFoam.

Any idea for that?

[wwzhao]

Hi mahmoodn,

You can use blockMesh and snappyHexMesh provided by OpenFOAM for meshing.

For testing PBS job scheduler, you can simply pick up a tutorial case from $FOAM_TUTORIALS directory and test with that case.

The $FOAM_TUTORIALS/incompressible/icoFoam/cavity is a good start. You need to copy it to your working directory, and run blockMesh inside the case directory. After meshing is done, run icoFoam to begin your simulation.

P.S.: don't forget to source OpenFOAM bashrc first

[mahmoodn]

Do you have any idea about this:

mahmood@cluster:of-test$ source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc
mahmood@cluster:of-test$ blockMesh
-bash: blockMesh: command not found
mahmood@cluster:of-test$ /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64Gcc48DPOpt/bin/blockMesh
/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64Gcc48DPOpt/bin/blockMesh: error while loading shared libraries: libblockMesh.so: cannot open shared object file: No such file or directory
mahmood@cluster:of-test$

I can test with LD_LIBRARY_PATH and PATH, but just want to know if I am on the correct track.

[wwzhao]

Hi mahmoodn,

Have you modified the bashrc from OpenFOAM? What's the value of your $FOAM_INST_DIR and $WM_PROJECT_INST_DIR?

[mahmoodn]

Yes, I modified it during the installation. Lines associated with that is

export WM_PROJECT=OpenFOAM
export WM_PROJECT_VERSION=2.3.0
foamInstall=/export/apps/mechanics/$WM_PROJECT

Therefore, I see

$ ls /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/
Allwmake      bin      doc  platforms    README.org  tutorials
applications  COPYING  etc  README.html  src         wmake

The values you asked are

$ echo $FOAM_INST_DIR
/export/apps/mechanics/OpenFOAM
$ echo $WM_PROJECT_INST_DIR
/export/apps/mechanics/OpenFOAM

[wwzhao]

Hmm. Seems the shell variables aren't exported correctly.

What's the output of export | grep -E '(FOAM_|WM_)'? What's the value of your $PATH and $LD_LIBRARY_PATH?

[mahmoodn]

Check it

$ export | grep -E '(FOAM_|WM_)'
declare -x FOAM_APP="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/applications"
declare -x FOAM_APPBIN="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/bin"
declare -x FOAM_EXT_LIBBIN="/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib"
declare -x FOAM_INST_DIR="/export/apps/mechanics/OpenFOAM"
declare -x FOAM_JOB_DIR="/export/apps/mechanics/OpenFOAM/jobControl"
declare -x FOAM_LIBBIN="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib"
declare -x FOAM_MPI="openmpi-1.6.5"
declare -x FOAM_RUN="/home/mahmood/OpenFOAM/mahmood-2.3.0/run"
declare -x FOAM_SETTINGS=""
declare -x FOAM_SIGFPE=""
declare -x FOAM_SITE_APPBIN="/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/bin"
declare -x FOAM_SITE_LIBBIN="/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/lib"
declare -x FOAM_SOLVERS="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/applications/solvers"
declare -x FOAM_SRC="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/src"
declare -x FOAM_TUTORIALS="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/tutorials"
declare -x FOAM_USER_APPBIN="/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/bin"
declare -x FOAM_USER_LIBBIN="/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/lib"
declare -x FOAM_UTILITIES="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/applications/utilities"
declare -x WM_ARCH="linux64"
declare -x WM_ARCH_OPTION="64"
declare -x WM_CC="gcc"
declare -x WM_CFLAGS="-m64 -fPIC"
declare -x WM_COMPILER="Gcc"
declare -x WM_COMPILER_LIB_ARCH="64"
declare -x WM_COMPILE_OPTION="Opt"
declare -x WM_CXX="g++"
declare -x WM_CXXFLAGS="-m64 -fPIC"
declare -x WM_DIR="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/wmake"
declare -x WM_LDFLAGS="-m64"
declare -x WM_LINK_LANGUAGE="c++"
declare -x WM_MPLIB="OPENMPI"
declare -x WM_OPTIONS="linux64GccDPOpt"
declare -x WM_OSTYPE="POSIX"
declare -x WM_PRECISION_OPTION="DP"
declare -x WM_PROJECT="OpenFOAM"
declare -x WM_PROJECT_DIR="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0"
declare -x WM_PROJECT_INST_DIR="/export/apps/mechanics/OpenFOAM"
declare -x WM_PROJECT_USER_DIR="/home/mahmood/OpenFOAM/mahmood-2.3.0"
declare -x WM_PROJECT_VERSION="2.3.0"
declare -x WM_THIRD_PARTY_DIR="/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0"



$ echo $PATH
/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/gperftools-svn/bin:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/ParaView-4.1.0/bin:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/openmpi-1.6.5/bin:/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/bin:/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/bin:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/bin:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/bin:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/wmake:/opt/openmpi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bio/ncbi/bin:/opt/bio/mpiblast/bin:/opt/bio/EMBOSS/bin:/opt/bio/clustalw/bin:/opt/bio/tcoffee/bin:/opt/bio/hmmer/bin:/opt/bio/phylip/exe:/opt/bio/mrbayes:/opt/bio/fasta:/opt/bio/glimmer/bin:/opt/bio/glimmer/scripts:/opt/bio/gromacs/bin:/opt/bio/gmap/bin:/opt/bio/tigr/bin:/opt/bio/autodocksuite/bin:/opt/bio/wgs/bin:/opt/eclipse:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/maven/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/opt/condor/bin:/opt/condor/sbin:/opt/gridengine/bin/linux-x64:/export/apps/mathematics/matlab2012a/bin:/home/mahmood/bin



$ echo $LD_LIBRARY_PATH
/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/gperftools-svn/lib:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/ParaView-4.1.0/lib/paraview-4.1:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/openmpi-1.6.5/lib:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/openmpi-1.6.5/lib64:/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/dummy:/opt/gridengine/lib/linux-x64:/opt/openmpi/lib

[wwzhao]

Seems your applications aren't installed to $FOAM_APPBIN.

Have you exported WM_COMPILER=Gcc48 before building OpenFOAM? This may be the issue. If so, just rebuild OpenFOAM without exporting that.

[mahmoodn]

I ran it again. Please see the variables first

root@cluster:ThirdParty-2.3.0# module load openmpi-x86_64 || export PATH=$PATH:/usr/lib64/openmpi/bin
root@cluster:ThirdParty-2.3.0# source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI
root@cluster:ThirdParty-2.3.0#

root@cluster:OpenFOAM-2.3.0# echo $WM_NCOMPPROCS
4
root@cluster:OpenFOAM-2.3.0# echo $WM_COMPILER
Gcc48
root@cluster:OpenFOAM-2.3.0# echo $WM_MPLIB
SYSTEMOPENMPI
root@cluster:OpenFOAM-2.3.0# echo $FOAM_APPBIN
/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64Gcc48DPOpt/bin

Then I ran ./Allwmake again and you can see the lengthy output at pastebin


This time, when I run blockMesh, I see its usage

root@cluster:OpenFOAM-2.3.0# blockMesh
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.3.0-f5222ca19ce6
Exec   : blockMesh
Date   : Mar 25 2016
Time   : 11:03:08
Host   : "cluster.scu.ac.ir"
PID    : 10972
Case   : /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Disallowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time



--> FOAM FATAL IO ERROR:
cannot find file

file: /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/system/controlDict at line 0.

    From function regIOobject::readStream()
    in file db/regIOobject/regIOobjectRead.C at line 73.

FOAM exiting

So it is working....
Question: What commands should I put in the .bashrc or /etc/profile (for all users)? These two?

root@cluster:ThirdParty-2.3.0# module load openmpi-x86_64 || export PATH=$PATH:/usr/lib64/openmpi/bin
root@cluster:ThirdParty-2.3.0# source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI

[wwzhao]

Hi mahmoodn,

Good to see it working

Question: What commands should I put in the .bashrc or /etc/profile (for all users)? These two?

Well, if your users do not need to switch between various versions of OpenFOAM, you can just add this two lines to /etc/profile.d/openfoam.sh (create new one if it does not exist).

Just keep the user variables same with those when you built OpenFOAM .


Regards,
Weiwen

[mahmoodn]

It seems that the following line has no effect in /etc/profile and ~/.bashrc

alias of230='module load openmpi-x86_64; source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI'

Instead, if I manually run

source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI

from the terminal, then I will be able to run blockMesh without full path

[wwzhao]

Originally Posted by mahmoodn

It seems that the following line has no effect in /etc/profile and ~/.bashrc

alias of230='module load openmpi-x86_64; source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI'

This line set alias for OpenFOAM, users who want to use OpenFOAM must execute of230 first.

The aliases are useful when there are multiple OpenFOAM versions as it wouldn't mess up the environment variables .


Regards,
Weiwen

[mahmoodn]

OK I got it... So the correct thing is to first run of230 and then blockMesh.

Thanks for that. I will move on to the tutorial for PBS.

[mahmoodn]

Hi again,
I ran blockMesh and then icoFoam in the $FOAM_TUTORIALS/incompressible/icoFoam/cavity. I saw some outputs without any error and it seems it is working.

Is there any multithreaded (needing more than core) example which I want to test with PBS?

[wwzhao]

Originally Posted by mahmoodn

Is there any multithreaded (needing more than core) example which I want to test with PBS?

Yes, for parallel computing, you need a more complex case, e.g, $FOAM_TUTORIALS/incompressible/pimpleDyMFoam/propeller

The following steps may be helpful.

1. Copy the propeller case to your working directory.

2. Run ./Allrun.pre to generate mesh and set initial conditions.

3. Run decomposePar to decompose the domain into 4 parts.

4. Prepare the PBS submit scripts. A simple example may look like this:

#!/bin/bash
#PBS -l nodes=1:ppn=4
#
#PBS -l walltime=00:30:00
#PBS -N myJobName
#PBS -A MyProjectName
#PBS -o outputFileName
#PBS -e errorFileName

module load openmpi-x86_64
source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI

mpirun -np 4 pimpleDyMFoam -parallel | tee log

5. Use qsub to submit your job.


Regards,
Weiwen

[mahmoodn]

Some good news and some bad news...

Good:
I am able to run the mpi command on master node as well as the compute node. In other words, the following commands works as expected.

of230
mpirun -np 4 pimpleDyMFoam -parallel | tee log

check the output for the server node (where OpenFoam is installed) and the compute node. 4 cores are utilized.

However, when I submit the job, the output file contains the following error:

mahmood@cluster:propeller$ qsub submit.tor
49.cluster.scu
mahmood@cluster:propeller$ qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
49.cluster                 myJobName        mahmood                0 R default
mahmood@cluster:propeller$ qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
49.cluster                 myJobName        mahmood         00:00:00 C default
mahmood@cluster:propeller$ cat outputFileName
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.3.0-f5222ca19ce6
Exec   : pimpleDyMFoam -parallel
Date   : Mar 25 2016
Time   : 20:20:19
Host   : "compute-0-0.local"
PID    : 2102
[0]
[0]
[0] --> FOAM FATAL ERROR:
[0] Cannot read "/home/mahmood/system/decomposeParDict"
[0]
FOAM parallel run exiting
[0]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 2102 on
node compute-0-0.local exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Something is wrong with the MPI communicator.

[mahmoodn]

Regarding the error

Cannot read "/home/mahmood/system/decomposeParDict"

I have to say, the file actually is available but in a different place. As suggested by you, I copied the tutorial folder to my home. So:

mahmood@cluster:propeller$ pwd
/home/mahmood/of-test/propeller

mahmood@cluster:propeller$ ls
0         Allrun.pre     log.createPatch            log.topoSet     processor1  system
0.org     constant       log.renumberMesh           outputFileName  processor2
Allclean  log            log.snappyHexMesh          postProcessing  processor3
Allrun    log.blockMesh  log.surfaceFeatureExtract  processor0      submit.tor

mahmood@cluster:propeller$ ls system/
controlDict                        forces      readFields                         surfaces
createInletOutletSets.topoSetDict  fvSchemes   snappyHexMeshDict
createPatchDict                    fvSolution  surfaceFeatureExtractDict
decomposeParDict                   Q           surfaceFeatureExtractDictDefaults

So, system/decomposeParDict is present. I suspect that someone (PBS or OpenFoam or MPI) wrongly creates $HOME/system/decomposeParDict which is considered as /home/mahmood/system/decomposeParDict

Any idea for that?

[mahmoodn]

OK. I think I found the problem. According to Basics: Set-up for different desktop machines (post #7 and after that), I found that my home has not been exported to the NFS! (have to say that I created my account on the server only).

So, let me test something and I will be back.

[wwzhao]

Hi mahmoodn,

You need to setup NIS server and client for sharing account info. Moreover, the home of compute nodes should be mounted via NFS.


Regards,
Weiwen

[mahmoodn]

OK I found the problem


In the PBS script, we have to add cd $PBS_O_WORKDIR abd this is an important thing.

Without that and by adding pwd before the mpi command in the PBS script, I saw that the working directory was /home/mahmood which is actually $HOME and for that reason it appended system/decomposeParDict to that.

With cd $PBS_O_WORKDIR, the working directory is correctly set to the location where propeller exists.

So, everything is OK now and I am able to run a OpenFoam job. Have to tell the guys for that.
Thanks.