[vishwa]

Hi,
I am pretty to new to OpenFoam seeing from the number of posts that i have posted in the last few days.. I had a lot of problems running LES of my case. And finally I got it running today.

out high performane cluster has 16 processors per node and i was successful in running the case on 1 node with 16 processors. However I couldn't run it on 2 nodes with 16 processors per node.

here are the details of the decomposeParDict

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.6                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 32;

method          metis;

simpleCoeffs
{
    n               ( 2 1 1 );
    delta           0.001;
}

hierarchicalCoeffs
{
    n               ( 2 2 1 );
    delta           0.001;
    order           xyz;
}

metisCoeffs
{
}

manualCoeffs
{
    dataFile        "";
}

distributed    no;

roots  ();


// ************************************************************************* //

here is the script that i have written

#!/bin/tcsh

#   1) Save this script as a file named myscript on hpc4
#   2) On hpc4, Issue                   
#       qsub myscript    to submit the job 
#        Use qstat -a to see job status, 
#         Use qdel jobname to delete one of your jobs
#         jobnames are of the form 1234.hpc4 

###########################################
# Output goes to file BATCH_OUTPUT.
# Error output goes to file BATCH_ERRORS.
# If you want the output to go to another file, change BATCH_OUTPUT 
# or BATCH_ERRORS in the following lines to the full path of that file. 


#PBS  -o BATCH_OUTPUT
#PBS  -e BATCH_ERRORS

#PBS -lmem=64Gb,nodes=2:ppn=16,cput=768:00:00,walltime=24:00:00

cd $PBS_O_WORKDIR


mpirun --hostfile $PBS_NODEFILE -np 32 pisoFoam -case /working004b/mech_eng1/vishwa/OpenFOAM/vishwa-1.6/CIJR/ -parallel > log

here is the error that i get

[0] 
[0] 
[0] number of processor directories = 16 is not equal to the number of processors = 32
[0] 
FOAM parallel run exiting
[0] 
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 23160 on
node node161 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
-

-------------------------------------------------------------------------

when i ran decomposePar, it did produce 32 processor directories..I am not sure why i get this error..

Any help would be greatly appreciated..

Thanks a tonne in advance..

Vishwa

[feijooos]

Vishwa,

You need to specify the metis coefficients in the decomposeParDict file, so there need to 32 coefficients. See below:

/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 1.6 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 32;

method metis;

simpleCoeffs
{
n ( 2 1 1 );
delta 0.001;
}

hierarchicalCoeffs
{
n ( 2 2 1 );
delta 0.001;
order xyz;
}

metisCoeffs
{
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
}

manualCoeffs
{
dataFile "";
}

distributed no;

roots ();

[flying]

vishwa and feijooos:

I think both of you have some experience to set up openfoam in cluster.

Please, really please give me some advice on it.

I am trying to set up openfoam in the cluster, but it is really sad that I do not know how to do it.

There is gcc4.4.1 and openmpi 1.3.3 in the cluster.


I should use all of the complier in the cluster. I am not allowed to use any complier in the thirdparty when the code is run in multi-nodes.


So, I need to compile the openfoam with the compile in cluster.


First question, Are gcc 4.4.1 and openmpi 1.3.3 enough for compile openfoam 1.5?
Second question, if it is enough , how do I do it?


I have changed the complier option and mpi setting up option. As follows


# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# WM_COMPILER_INST = OpenFOAM | System
# WM_COMPILER_INST=OpenFOAM (Orginal)
WM_COMPILER_INST=System (changed)

case "$WM_MPLIB" in
OPENMPI)
mpi_version=openmpi-1.2.6
# export MPI_HOME=$WM_THIRD_PARTY_DIR/$mpi_version
# export MPI_HOME=$WM_THIRD_PARTY_DIR/$mpi_version (orginal)
# export MPI_ARCH_PATH=$MPI_HOME/platforms/$WM_OPTIONS (orginal

export MPI_ARCH_PATH=/opt/openmpi/1.3.3/gcc-4.4.1 (changed)


It could compile. But there is some error information as follows:



Note: ignore spurious warnings about missing mpicxx.h headers
+ wmake libso mpi
Making dependency list for source file OPwrite.C
could not open file ompi/mpi/cxx/pmpicxx.h for source file OPwrite.C
could not open file ompi/mpi/cxx/constants.h for source file OPwrite.C
could not open file ompi/mpi/cxx/functions.h for source file OPwrite.C
could not open file ompi/mpi/cxx/datatype.h for source file OPwrite.C
could not open file ompi/mpi/cxx/exception.h for source file Opwrite.C


Who would like to give me some idea about it?

[feijooos]

sorry, I can't help you. I don't know much about installing OF.

[make]

dear flying,

it seems like you haven't set the path to the openmpi includes on your cluster correctly. The dependency builder is not able to find the header files. MPI_ARCH_PATH should point to the path where the include directory of openmpi is located. And don't forget to set WM_MPLIB in etc/bashrc within the OpenFOAM directory correctly.

regards,
Matthias

[flying]

Thank you for your reply!

I have changed the path and set the WM_MPLIB in etc/bashrc. But it still doesn't work.

Best wishes!

[make]

Dear flying,

this is what my settings.sh look like for OpenMPI:

OPENMPI)
mpi_version=1.3.3
export MPI_HOME=/software/all/openmpi/$mpi_version
export MPI_ARCH_PATH=$MPI_HOME

# Tell OpenMPI where to find its install directory
export OPAL_PREFIX=$MPI_ARCH_PATH

_foamAddPath $MPI_ARCH_PATH/bin
_foamAddLib $MPI_ARCH_PATH/lib

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/$mpi_version
unset mpi_version
;;

Within the /software/all/openmpi/1.3.3 directory is the include folder of openmpi /software/all/openmpi/1.3.3/include/openmpi. Your dependency builder is looking for the folder ompi in this directory.
E.g. one line in In your post looks like:
could not open file ompi/mpi/cxx/pmpicxx.h for source file OPwrite.C

So it seems like you still haven't set the path correctly. Check that these directories are within the path you specified for MPI_ARCH_PATH.
According to your specification the file

/opt/openmpi/1.3.3/gcc-4.4.1/include/openmpi/ompi/mpi/cxx/pmpicxx.h

should exist. I sligthly wonder about the gcc-4.4.1 folder in your path. Maybe try MPI_ARCH_PATH=/opt/openmpi/1.3.3.

Regards,
Matthias

[farhagim]

Hi there,

I am a beginner user of the OpenFoam...I really need to learn how to run openfoam with parallel processing. I read the all the available guideline in this site www.OpenFoam.com)...but I have basic problem with that. First of all I can not find "etc/hosts " in OpenFoam Directories where I should define the host names of the machines. Since I am not familiar with that kind of file
, I would be so grateful if anybody provide me one of these kind of files.

2. I read somewhere that I have to create a "file that contains the
host names of the machines. The file can be given any name and located at any path. " I could not understand what does it means ...or what kind of the files do i have to create....can anybody give a hint or example about that.

3. In decompose Dict, the roots entry is a list of root paths, <root0>, <root1>, . . . , for each node:
roots
<nRoots>
(
"<root0>"
"<root1>"
...
);

I need somebody help me throough this too..
AS you can see , I do not know anything about the configuration of running the case in parallel. I would be so grateful if you can help me

Thanks alot
Farhangi

Originally Posted by make

Dear flying,

this is what my settings.sh look like for OpenMPI:

OPENMPI)
mpi_version=1.3.3
export MPI_HOME=/software/all/openmpi/$mpi_version
export MPI_ARCH_PATH=$MPI_HOME

# Tell OpenMPI where to find its install directory
export OPAL_PREFIX=$MPI_ARCH_PATH

_foamAddPath $MPI_ARCH_PATH/bin
_foamAddLib $MPI_ARCH_PATH/lib

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/$mpi_version
unset mpi_version
;;

Within the /software/all/openmpi/1.3.3 directory is the include folder of openmpi /software/all/openmpi/1.3.3/include/openmpi. Your dependency builder is looking for the folder ompi in this directory.
E.g. one line in In your post looks like:
could not open file ompi/mpi/cxx/pmpicxx.h for source file OPwrite.C

So it seems like you still haven't set the path correctly. Check that these directories are within the path you specified for MPI_ARCH_PATH.
According to your specification the file

/opt/openmpi/1.3.3/gcc-4.4.1/include/openmpi/ompi/mpi/cxx/pmpicxx.h

should exist. I sligthly wonder about the gcc-4.4.1 folder in your path. Maybe try MPI_ARCH_PATH=/opt/openmpi/1.3.3.

Regards,
Matthias

[maysmech]

Originally Posted by farhagim

Hi there,

I am a beginner user of the OpenFoam...I really need to learn how to run openfoam with parallel processing. I read the all the available guideline in this site www.OpenFoam.com)...but I have basic problem with that. First of all I can not find "etc/hosts " in OpenFoam Directories where I should define the host names of the machines. Since I am not familiar with that kind of file
, I would be so grateful if anybody provide me one of these kind of files.

2. I read somewhere that I have to create a "file that contains the
host names of the machines. The file can be given any name and located at any path. " I could not understand what does it means ...or what kind of the files do i have to create....can anybody give a hint or example about that.

3. In decompose Dict, the roots entry is a list of root paths, <root0>, <root1>, . . . , for each node:
roots
<nRoots>
(
"<root0>"
"<root1>"
...
);

I need somebody help me throough this too..
AS you can see , I do not know anything about the configuration of running the case in parallel. I would be so grateful if you can help me

Thanks alot
Farhangi

I have same problems.
Any help will be appreciated.

[fsaltara]

I had a similar problem, and solved it simply running the Allwmake script in the thirdparty folder.

[maysmech]

I have used two machines (laptop and desktop) which are connected via cross cable.

maysam-desktop cpu = 6
maysam-laptop cpu = 2

I added to /etc/hosts of both machines:
192.168.1.1 maysam-desktop
192.168.1.2 maysam-laptop
Also set ip same as above for both of them.

After trying to run, this error is seen:

maysam@maysam-desktop:~/OpenFOAM/maysam-1.7.0/run/icotest$ mpirun --hostfile host.txt -np 8 icoFoam -parallel
maysam@maysam-laptop's password:
--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not find an executable:

Executable: icoFoam
Node: maysam-laptop

while attempting to start process rank 6.
--------------------------------------------------------------------------
maysam@maysam-desktop:~/OpenFOAM/maysam-1.7.0/run/icotest$

Any help will be appreciated.

[longamon]

I don't know if this comes too late, but heres a sample of my decomposeParDict:


numberOfSubdomains 6; //number of cpus you'll use

method scotch; //method of decomposition

simpleCoeffs
{
n ( 2 2 2 );
delta 0.001;
}

hierarchicalCoeffs
{
n ( 1 1 1 );
delta 0.001;
order xyz;
}

// in my case I use scotch decomposition, so I specify the processsor weight for each cpu

scotchCoeffs
{
processorWeights
(
1
1
1
1
1
1
);
}

manualCoeffs
{
dataFile "";
}

distributed yes; //distribute the data on diferent harddrives

roots 5
(
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
);

since there are 6 cpus, there is one master and 5 slaves. the master will be processor 0, so there's no need to specify a root to it, the other 5 cpus are slave. note that one machine can have more than one cpu. in my case, the case file is located under /home/davidal for all my cpus.

see
http://www.cfd-online.com/Forums/ope...am-solved.html
for more info

In my case i will have 6 folders of info in each cpu, all named /processor0, /processor1, /processor2, /processor3, /processor4 and /processor5. Since I have 2 cpu's per machines i will be saving data on only 2 of these folders on each machine.
machine1 will save data on processor0 and processor1
machine2 will save data on processor2 and processor3
machine3 will save data on processor4 and processor5

hope this helps, I'm quite new to OF and struggled with this for a bit, but now its working fine

[maysmech]

Originally Posted by longamon

I don't know if this comes too late, but heres a sample of my decomposeParDict:


numberOfSubdomains 6; //number of cpus you'll use

method scotch; //method of decomposition

simpleCoeffs
{
n ( 2 2 2 );
delta 0.001;
}

hierarchicalCoeffs
{
n ( 1 1 1 );
delta 0.001;
order xyz;
}

// in my case I use scotch decomposition, so I specify the processsor weight for each cpu

scotchCoeffs
{
processorWeights
(
1
1
1
1
1
1
);
}

manualCoeffs
{
dataFile "";
}

distributed yes; //distribute the data on diferent harddrives

roots 5
(
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
);

since there are 6 cpus, there is one master and 5 slaves. the master will be processor 0, so there's no need to specify a root to it, the other 5 cpus are slave. note that one machine can have more than one cpu. in my case, the case file is located under /home/davidal for all my cpus.

see
http://www.cfd-online.com/Forums/ope...am-solved.html
for more info

In my case i will have 6 folders of info in each cpu, all named /processor0, /processor1, /processor2, /processor3, /processor4 and /processor5. Since I have 2 cpu's per machines i will be saving data on only 2 of these folders on each machine.
machine1 will save data on processor0 and processor1
machine2 will save data on processor2 and processor3
machine3 will save data on processor4 and processor5

hope this helps, I'm quite new to OF and struggled with this for a bit, but now its working fine

Dear David,
Thanks for your post.
My problem has been solved. you can see its solution at:
http://www.cfd-online.com/Forums/ope...tml#post297100

Regards,
Maysam

[pfhan]

Thanks, David, it is of great help to me.

[samiam1000]

Hi All,

I would run a parallel case on a server.

I read that I should give this command:

mpirun -np 4 -hostname FILE buoyantPimpleFoam -parallel

where FILE is a text file where I write the name of the machine I am running my case.

Does this file need any header or it is enough a single line with the name of the machine?

Thanks,
Samuele

[wyldckat]

Hi Samuele,

I'm going to quote (minor changes to make it more legible):

Originally Posted by Lexx

2- In file machines (which can be created in the tutorial) put the machine name followed by how many processors are used on this machine.

Example:

ubuntu1
 ubuntu2 cpu = 2

In this case, the FILE is named "machines".

Best regards,
Bruno

[samiam1000]

Thanks a lot Bruno.

I solved it.

I opened another thread (maybe you can check it or delete it and continue here) to understand how to put my process in background, when running on the server.

Thanks a lot,
Samuele

[wyldckat]

Hi Samuele,

OK... I guess the thread you speak of is this: http://www.cfd-online.com/Forums/ope...el-server.html

I'll answer it there...

Best regards,
Bruno

[samiam1000]

Perfect.

Thanks a lot!

[arunsmec]

I am trying to do an LES of roundjet using parallel computing. I used the decomposePar utility to decompose the domain. My workstation is equipped with 2 quadcore processors with multithreading (2 threads/core). Does decomposition into more than 8 subdomains make any difference?