[makaveli_lcf]

Hi all!

I have following problem while running OpenFOAM 1.6 on cluster with queuing system. I compiled OpenFOAM from scratch using GCC from OF distribution and MVAPICH installed on cluster. OpenFOAM runs perfect on login machine in serial, but when I start it with queuing system it fails with such error in error output:

--> FOAM FATAL ERROR in Foam::findEtcFile() : could not find mandatory file
'controlDict'

As I understand, OpenFOAM could not get proper environment variable while running on remote machine. How to provide proper environment for OpenFOAM?

Thank you in advance

[makaveli_lcf]

Problem solved!
Tommorow read about my expereance with OpenFOAM on MVAPICH+InfinityBand

[mrv4real]

Hi,

even if this is an very old thread, do you remember the solution? Do i have to set some environment variables and if yes, which are they? setting the -case unfortunately did not work.

Thanks in advance,
mrv4real

[wyldckat]

Greetings mrv4real and welcome to the forum!

There are at least two ways for doing this:

1. There is a script named "foamExec" that you can use to launch OpenFOAM applications. Run in your current terminal/console window the following command:

   which foamExec

   Now, use that script for calling your desired solver or application. As an example, for running icoFoam in parallel in my machine, instead of simply using:

   icoFoam -parallel

   I would use:

   /home/me/OpenFOAM/OpenFOAM-2.0.x/bin/foamExec icoFoam -parallel

   This should work on all OpenFOAM versions, but sadly it doesn't always work, due to some bugs that existed on some versions.
2. Or you can use a job file for the PBS, like the one shown here: http://www.cfd-online.com/Forums/ope...tml#post286124 - post #5

Best regards,
Bruno

[mrv4real]

Hi Bruno,

thanks a lot for the nice welcome message and your fast reply!

I tried your suggestions but am still having problems. I try to run it on a BlueGene/P and it works for serial code.

If I try it with mpirun in a loadleveler script, i had to pass the LD_LIBRARY_PATH environment variable and allthough that i specify the case directory with "-case <...>" he throws the error message about the missing controlDict.

Is there a way to give the directory of the simulation using an environment variable i can pass to the mpirun?

Thanks,
mrv4real

[mrv4real]

... maybe it is best, if I post the script for the loadleveler. Hoping for some sugesstions...

#!/bin/sh
# @ account_no = []
# @ job_name = test_damBreak
# @ output = $(job_name).out
# @ error = $(job_name).err
# @ environment = COPY_ALL
# @ wall_clock_limit = 00:05:00
# @ notification = always
# @ notify_user = my@address.net

# @ job_type = bluegene
# @ bg_size = 4


# @ queue

source ~/OpenFOAM/OpenFOAM-1.6/etc/bashrc

cd /home/my_account/damBreak

mpirun -np 4 -mode SMP -cwd /home/my_account/damBreak -exe /home/my_account/OpenFOAM/OpenFOAM-1.6/applications/bin/linuxGccDPOpt/interFoam -args "-case /home/my_account/damBreak -parallel" -env LD_LIBRARY_PATH=/home/my_account/OpenFOAM/OpenFOAM-1.6/lib/linuxGccDPOpt/mpi:/home/my_account/OpenFOAM/my_account-1.6/lib/linuxGccDPOpt:/home/my_account/OpenFOAM/site/1.6/lib/linuxGccDPOpt:/home/my_account/OpenFOAM/OpenFOAM-1.6/lib/linuxGccDPOpt


Thanks a lot,
mrv4real

[wyldckat]

Hi mrv4real,

I forgot to mention that the controlDict that the solver is complaining about is not the one in your case's system folder; it is complaining about not finding "OpenFOAM-1.6/etc/controlDict". This means that LD_LIBRARY_PATH is far from enough; there are a whole lot more variables that would need exporting.

Mmm... OpenFOAM 1.6 is getting old now... but OK, let's try this:

1. foamExec in 1.6 allows you to read OF's bashrc file from a personal folder, namely "$HOME/.OpenFOAM/$version/bashrc". So, the following commands will create the personal folder used by OpenFOAM and copy the bashrc file to the newly created folder:

   mkdir -p $HOME/.OpenFOAM/1.6
   cp $WM_PROJECT_DIR/etc/bashrc $HOME/.OpenFOAM/1.6/

2. Edit the copied file if necessary, such as hard coding some of the folders to use your "/home/my_account", instead of $HOME.
3. Now, as for the loadleveler script, this should do the trick:

   #!/bin/sh
   # @ account_no = []
   # @ job_name = test_damBreak
   # @ output = $(job_name).out
   # @ error = $(job_name).err
   # @ environment = COPY_ALL
   # @ wall_clock_limit = 00:05:00
   # @ notification        = always
   # @ notify_user        = my@address.net
   
   # @ job_type = bluegene
   # @ bg_size = 4
   
   
   # @ queue
   
   source $HOME/.OpenFOAM/1.6/bashrc
   
   cd /home/my_account/damBreak
   
   mpirun -np 4 -mode SMP -cwd $PWD -exe `which foamExec`  -args "interFoam -parallel"

   It's simpler this way and more easily adaptable when you need to change something. Otherwise there is another way, but it ain't pretty

Good luck!
Bruno

[mrv4real]

Hi,

i already tried it with a script, but then mpirun gives the error "Load failed on 172.16.100.69: Magic value in ELF header of executable file is invalid"

This comes from the fact, that foamExec is a script and not an executable. I found something about a "-m script" switch for mpirun, but the BlueGene doesn´t accept it...

Thanks,
mrv4real

[wyldckat]

I did a bit of googling and found this: http://bluegene.epfl.ch/mpirun.txt
There seems to be an option named "-env_all". This would simplify the script, therefore not needing foamExec nor to manually define each variable to be sent.

If that doesn't work, then it's not going to be pretty

If I'm not mistaken, the necessary variables are: PATH LD_LIBRARY_PATH WM_PROJECT_DIR WM_PROJECT_INST_DIR WM_OPTIONS FOAM_LIBBIN FOAM_APPBIN FOAM_USER_APPBIN MPI_BUFFER_SIZE
The last one depends on the MPI you are using in BlueGene. In bold are the ones that are reeeeally necessary. I think the other ones are sometimes necessary...

Something like this should work:

#!/bin/sh
# @ account_no = []
# @ job_name = test_damBreak
# @ output = $(job_name).out
# @ error = $(job_name).err
# @ environment = COPY_ALL
# @ wall_clock_limit = 00:05:00
# @ notification        = always
# @ notify_user        = my@address.net

# @ job_type = bluegene
# @ bg_size = 4


# @ queue

source ~/OpenFOAM/OpenFOAM-1.6/etc/bashrc

cd /home/my_account/damBreak

mpirun -np 4 -mode SMP -cwd $PWD -exe `which interFoam`  -args "-parallel" -exp_env LD_LIBRARY_PATH -exp_env PATH -exp_env WM_PROJECT_DIR -exp_env WM_PROJECT_INST_DIR -exp_env  WM_OPTIONS -exp_env  FOAM_LIBBIN -exp_env  FOAM_APPBIN -exp_env  FOAM_USER_APPBIN -exp_env MPI_BUFFER_SIZE

If exporting the current variables to the remote nodes can't be done, then... well, the only ones that can have problems are PATH and LD_LIBRARY_PATH, so for the remaining variables, you can use the "-exp_env" option.

Oh, about PWD: this is usually an environment variable that has the path for the current directory, but it might not work on the environment you are working on

Best regards,
Bruno

[mrv4real]

thanks a lot !!! now it works.

Best,
mrv4real

[tH3f0rC3]

Edit: sorry for posting.

[nandiganavishal]

Hello Foamers,

I was able to compile my test code (PNPFoam) and could run it in a serial mode in Kraken cluster. However, when I was trying to run a parallel case, I had an error similar to the ones discussed in the above message.

FOAM FATAL ERROR in Foam::findEtcFiles() : could not find mandatory file
    'controlDict--> FOAM FATAL ERROR in Foam::findEtcFiles() : could not find mandatory file

I tried to follow the above procedure, but none of them worked.

Below is my sample script file to run the parallel case.

#!/bin/bash
#PBS -A TG-CDA100010
#PBS -l size=72,walltime=00:02:00
#PBS -o a.out
#PBS -N  simout 

cd $PBS_O_WORKDIR

aprun -n 72 -N 12 /lustre/scratch/joonho/openfoamv/conc_10mM/2D_simulations/PNP_solve/PNP_Foam

also below are the export environments that are defined in the bashrc file in my home folder

export LD_LIBRARY_PATH=/lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/system-dynlib:/lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/OpenFOAM-2.1.1/platforms/linux64GccDPOpt/lib/mpich2-system:/lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/OpenFOAM-2.1.1/platforms/linux64GccDPOpt/lib/dummy:/opt/gcc/mpc/0.8.1/lib:/opt/gcc/mpfr/2.4.2/lib:/opt/gcc/gmp/4.3.2/lib:/opt/gcc/4.6.2/snos/lib64:/lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/OpenFOAM-2.1.1/platforms/linux64GccDPOpt/lib:/opt/cray/atp/1.4.1/lib

export PATH=/lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/OpenFOAM-2.1.1/wmake:/lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/OpenFOAM-2.1.1/platforms/linux64GccDPOpt/bin:/lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/OpenFOAM-2.1.1/bin:/opt/cray/atp/1.4.1/bin:/opt/cray/xt-asyncpe/5.11/bin:/opt/cray/pmi/2.1.4-1.0000.8596.15.1.ss/bin:/opt/gcc/4.6.2/bin:/nics/e/sw/xt-cle3.1/xdusage:/sw/xt/globus/5.0.4/binary/bin:/sw/xt/globus/5.0.4/binary/sbin:/sw/altd/bin:/usr/local/hsi/bin:/usr/local/gold/bin:/sw/xt/bin:/opt/moab/6.1.6/bin:/opt/torque/2.5.11/bin:/opt/cray/lustre-cray_ss_s/1.8.4_2.6.27.48_0.12.1_1.0301.5943.18.1-1.0301.27524.1.24/sbin:/opt/cray/lustre-cray_ss_s/1.8.4_2.6.27.48_0.12.1_1.0301.5943.18.1-1.0301.27524.1.24/bin:/opt/cray/MySQL/5.0.64-1.0301.2899.20.1.ss/sbin:/opt/cray/MySQL/5.0.64-1.0301.2899.20.1.ss/bin:/opt/cray/sdb/1.0-1.0301.25929.4.88.ss/bin:/opt/cray/nodestat/2.2-1.0301.25918.4.1.ss/bin:/opt/modules/3.2.6.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib64/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/cray/bin

Can you please let me know how to resolve this error.

Thanks

Regards,
Vishal

[wyldckat]

Greetings Vishal,

You should contact the administrator for the cluster you're using and/or read the instructions on their website. Because it seems that you're using a dedicated installation of OpenFOAM 2.1.1, which very likely will not work how we might expect, since we don't have access to the cluster you're using.

Because from your description, you seem to be confusing the path to your case "/lustre/scratch/joonho/openfoamv/conc_10mM/2D_simulations/PNP_solve/PNP_Foam" with the path to your application "PNPFoam".

My guess is that you should change your script to something like this:

#!/bin/bash
#PBS -A TG-CDA100010
#PBS -l size=72,walltime=00:02:00
#PBS -o a.out
#PBS -N  simout 

cd $PBS_O_WORKDIR

. /lustre/scratch/proj/sw/openfoam/2.1.1/cnl3.1_gnu4.6.2/OpenFOAM-2.1.1/etc/bashrc

aprun -n 72 -N 12 /full/path/to/your/PNPFoam -case /lustre/scratch/joonho/openfoamv/conc_10mM/2D_simulations/PNP_solve/PNP_Foam -parallel

Make sure to change the path "/full/path/to/your/PNPFoam" to the correct path for your solver. You can see its path if you build it again, as it's in the last 1 or 2 lines of the build process.

Best regards,
Bruno

[Mehrez]

Thanks it works !

Originally Posted by wyldckat

Greetings mrv4real and welcome to the forum!

There are at least two ways for doing this:

1. There is a script named "foamExec" that you can use to launch OpenFOAM applications. Run in your current terminal/console window the following command:

   which foamExec

   Now, use that script for calling your desired solver or application. As an example, for running icoFoam in parallel in my machine, instead of simply using:

   icoFoam -parallel

   I would use:

   /home/me/OpenFOAM/OpenFOAM-2.0.x/bin/foamExec icoFoam -parallel

   This should work on all OpenFOAM versions, but sadly it doesn't always work, due to some bugs that existed on some versions.
2. Or you can use a job file for the PBS, like the one shown here: http://www.cfd-online.com/Forums/ope...tml#post286124 - post #5

Best regards,
Bruno

[t.oliveira]

I received the message "FOAM FATAL ERROR in Foam::findEtcFile() : could not find mandatory file 'controlDict'" when running splitMeshRegions from the command line, but it worked when called through a script. After comparing the environment variables in each situation and doing some test, I found out that the culprit was the variable WM_PROJECT_DIR.

Could anyone explain the relationship between the variable WM_PROJECT_DIR being wrongly set (it should points to where OpenFOAM is installed) and splitMeshRegions not finding the controlDict file (whose path is specific to the case I am working in)?

Thanks,
Thomas

[t.oliveira]

After some more tests, I realised that the mandatory file 'controlDict' which was not found didn't refer to the system/controlDict file of my specific case. Instead, it referred to the $WM_PROJECT_DIR/etc/controlDict file.

Now the relationship between the variable WM_PROJECT_DIR and the error message makes sense.

Regards,
Thomas

[EDIT: and now I noticed that wyldckat had already pointed out to this fact in September 3, 2011].