[gabrielfelix]

Hi folks,

I've recently gained access to my university cluster, HPC or server (call it as you prefer) and I wanted to run my OpenFOAM simulations on it.

In these clusters you usually do not have any administrator rights so you can't install precompiled builds of OF as you do in your personal computer. Therefore, you will need to download and compile the OpenFOAM source code on the server.

I took almost two weeks to succesfully compile OF on the cluster. I have read almost every thread about the subject here on CFD-Online comunity and just a few of them were actually useful to apply in my case. I'll try to compile here in this thread all the information I could gather among these threads and I hope that it helps you if you find yourself in the same situation I was.

What I did initially was to follow the OpenFoam.org official instructions constained here. However, my compilation crashed/broke on a mpi.h error.

When I set the ./Allwmake -k flag (according to this thread
[OpenFOAM.com] Compiling v2006, mpicc -- broken), the compilation skipped the mpi.h, and succesfully compiled the other packages. I could actually run a test case such as pitzDaily with simpleFoam, but I could not run it in parallel as it needs the library OpenMPI.

I started then to research about the OpenMPI installation and it seems that it is the error that most people get. The clusters usually have different versions of OpenMPI or it can even not be intalled on the machine. In most of the threads people said to modify the WM_MPLIB flag on etc/bashrc from SYSTEMOPENMPI to OPENMPI and other ones. However you will always get the same error because the MPI might not be running on the server. You will need to discover which version is installed on the server and then you will have to load it.

After loading the OpenMPI module, I stopped getting the mpi.h error and compilation advanced until it crashed again in a ld.gold error, which I managed to solve through steps 1 to 4 of this thread Compiling OpenFoam from git : problem with ld. Compilation finally finished. However, when I tried to decompose the case with decomposePar, I got errors again.

After many tries, I decided to delete this compilation and start a fresh-new one because I have done many modifications in etc/bashrc and Allwmake and these modifications could have introduced some user-induced errors to the process. To delete the compilation you just got to delete the opt/OpenFoam folder and unpack the .tgz file again to the same directory.

Until here I was always using the default "system" compiler and a user in this thread
[OpenFOAM.org] OpenFOAM 2.3.0 on CentOS 6.5 suggested to use the "ThirdParty" compiler and modify the WM_MPLIB to OPENMPI. I loaded the OpenMPI module and made the modifications, however I still got errors. Then I only modified the compiler and not the WM_MPLIB and it worked! There finally were no errors during compilation and it succesfully finished compiling all packages.

So, in order to succesfully compile OF on the server I followed the steps above. I hope that this set up works for you!

1. Download the source code in your client PC and transfer to the server

https://dl.openfoam.com/source/v2012/OpenFOAM-v2012.tgz

scp -P <port> ~/OpenFOAM-v2012-windows10.tgz <your-user>@XXX.XX.XX.XX:$HOME/

You might not need to specify the port as I did.

2. Unpack

tar -xvzf -C OpenFOAM-v2012.tgz $HOME/opt/

3. Identify which OpenMPI version is/are installed on the cluster

module spider openmpi

If there is any OpenMPI installed in the machine you will probably need to compile it yourself. There are some threads on the forum to help you.

$ module spider openmpi

--------------------------------------------------------------------------------------------------------------------
  OpenMPI:
--------------------------------------------------------------------------------------------------------------------
    Description:
      The Open MPI Project is an open source MPI-3 implementation.

     Versions:
        OpenMPI/3.1.1-GCC-7.3.0-2.30
        OpenMPI/3.1.3-GCC-8.2.0-2.31.1
        OpenMPI/3.1.4-GCC-8.3.0
        OpenMPI/4.0.5-GCC-10.2.0

--------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "OpenMPI" module (including how to load the modules) use the module's full name.
  For example:

     $ module spider OpenMPI/4.0.5-GCC-10.2.0
--------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------
  openmpi/3.1.3: openmpi/3.1.3/2019
--------------------------------------------------------------------------------------------------------------------

    This module can be loaded directly: module load openmpi/3.1.3/2019

4. Load the version you want. I chose the latest that was shown in step 3

Copy exactly the name of the version that appears in the message returned in step 3.

module load OpenMPI/4.0.5-GCC-10.2.0 ||  export PATH=$PATH:/usr/lib64/openmpi/bin

5. Edit etc/bashrc parameters

WM_COMPILER_TYPE=ThirdParty

Do not change the WM_MPLIB=SYSTEMOPENMPI parameter

6. Source etc/bashrc

source etc/bashrc

7. Execute compilator

You can run it in parallel with the -j flag. Change the number of processors according to your machine. You don't need to use the maximum number of processors available. Use just a few.

./Allwmake -j 6

8. Test the installation

foamInstallationTest

Run a case such as pitzDaily

cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily $HOME/
cd $HOME/pitzDaily
blockMesh
simpleFoam

Run in parallel.
Create a decomposeParDict similar to the other tutorials

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 5;

method          hierarchical;

coeffs
{
    n           (1 5 1);
}

blockMesh
decomposePar
mpirun -np 5 simpleFoam -parallel
or
mpiexec -np 5 simpleFoam -parallel
reconstructPar

Change the number of processors according to your machine

9. Load OpenMPI and OF automatically

You will need to modify the server bashrc to load the OF and the OpenMPI whenever you login in to it so you won't have to load it manually every time.

Add these lines to the end of ~/.bashrc

alias of2012='alias of2012='module load OpenMPI/4.0.5-GCC-10.2.0 || export PATH=$PATH:/usr/lib64/openmpi/bin; source /home70/gfelix/opt/OpenFOAM-v2012/etc/bashrc $FOAM_SETTINGS'

When you login again just type

of2012

to load it.

Now the OF is ready to run your cases. You just gotta transfer your run files with scp as done in step 1.

NOTE: You might want to set up a passwordless login in order to avoid having to type your user and password everytime you login to the server. Just follow this guide How to Setup Passwordless SSH Login

NOTE 2: You can check the information about the cluster's nodes and number of processors with the following commands:

Number of processors (CPU)
	grep -c ^processor /proc/cpuinfo
Infos about the nodes
	cat /proc/cpuinfo

[mikulo]

Originally Posted by gabrielfelix

Hi folks,

I've recently gained access to my university cluster, HPC or server (call it as you prefer) and I wanted to run my OpenFOAM simulations on it.

In these clusters you usually do not have any administrator rights so you can't install precompiled builds of OF as you do in your personal computer. Therefore, you will need to download and compile the OpenFOAM source code on the server.

I took almost two weeks to succesfully compile OF on the cluster. I have read almost every thread about the subject here on CFD-Online comunity and just a few of them were actually useful to apply in my case. I'll try to compile here in this thread all the information I could gather among these threads and I hope that it helps you if you find yourself in the same situation I was.

What I did initially was to follow the OpenFoam.org official instructions constained here. However, my compilation crashed/broke on a mpi.h error.

When I set the ./Allwmake -k flag (according to this thread
[OpenFOAM.com] Compiling v2006, mpicc -- broken), the compilation skipped the mpi.h, and succesfully compiled the other packages. I could actually run a test case such as pitzDaily with simpleFoam, but I could not run it in parallel as it needs the library OpenMPI.

I started then to research about the OpenMPI installation and it seems that it is the error that most people get. The clusters usually have different versions of OpenMPI or it can even not be intalled on the machine. In most of the threads people said to modify the WM_MPLIB flag on etc/bashrc from SYSTEMOPENMPI to OPENMPI and other ones. However you will always get the same error because the MPI might not be running on the server. You will need to discover which version is installed on the server and then you will have to load it.

After loading the OpenMPI module, I stopped getting the mpi.h error and compilation advanced until it crashed again in a ld.gold error, which I managed to solve through steps 1 to 4 of this thread Compiling OpenFoam from git : problem with ld. Compilation finally finished. However, when I tried to decompose the case with decomposePar, I got errors again.

After many tries, I decided to delete this compilation and start a fresh-new one because I have done many modifications in etc/bashrc and Allwmake and these modifications could have introduced some user-induced errors to the process. To delete the compilation you just got to delete the opt/OpenFoam folder and unpack the .tgz file again to the same directory.

Until here I was always using the default "system" compiler and a user in this thread
[OpenFOAM.org] OpenFOAM 2.3.0 on CentOS 6.5 suggested to use the "ThirdParty" compiler and modify the WM_MPLIB to OPENMPI. I loaded the OpenMPI module and made the modifications, however I still got errors. Then I only modified the compiler and not the WM_MPLIB and it worked! There finally were no errors during compilation and it succesfully finished compiling all packages.

So, in order to succesfully compile OF on the server I followed the steps above. I hope that this set up works for you!

1. Download the source code in your client PC and transfer to the server

https://dl.openfoam.com/source/v2012/OpenFOAM-v2012.tgz

scp -P <port> ~/OpenFOAM-v2012-windows10.tgz <your-user>@XXX.XX.XX.XX:$HOME/

You might not need to specify the port as I did.

2. Unpack

tar -xvzf -C OpenFOAM-v2012.tgz $HOME/opt/

3. Identify which OpenMPI version is/are installed on the cluster

module spider openmpi

If there is any OpenMPI installed in the machine you will probably need to compile it yourself. There are some threads on the forum to help you.

$ module spider openmpi

--------------------------------------------------------------------------------------------------------------------
  OpenMPI:
--------------------------------------------------------------------------------------------------------------------
    Description:
      The Open MPI Project is an open source MPI-3 implementation.

     Versions:
        OpenMPI/3.1.1-GCC-7.3.0-2.30
        OpenMPI/3.1.3-GCC-8.2.0-2.31.1
        OpenMPI/3.1.4-GCC-8.3.0
        OpenMPI/4.0.5-GCC-10.2.0

--------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "OpenMPI" module (including how to load the modules) use the module's full name.
  For example:

     $ module spider OpenMPI/4.0.5-GCC-10.2.0
--------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------
  openmpi/3.1.3: openmpi/3.1.3/2019
--------------------------------------------------------------------------------------------------------------------

    This module can be loaded directly: module load openmpi/3.1.3/2019

4. Load the version you want. I chose the latest that was shown in step 3

Copy exactly the name of the version that appears in the message returned in step 3.

module load OpenMPI/4.0.5-GCC-10.2.0 ||  export PATH=$PATH:/usr/lib64/openmpi/bin

5. Edit etc/bashrc parameters

WM_COMPILER_TYPE=ThirdParty

Do not change the WM_MPLIB=SYSTEMOPENMPI parameter

6. Source etc/bashrc

source etc/bashrc

7. Execute compilator

You can run it in parallel with the -j flag. Change the number of processors according to your machine. You don't need to use the maximum number of processors available. Use just a few.

./Allwmake -j 6

8. Test the installation

foamInstallationTest

Run a case such as pitzDaily

cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily $HOME/
cd $HOME/pitzDaily
blockMesh
simpleFoam

Run in parallel.
Create a decomposeParDict similar to the other tutorials

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 5;

method          hierarchical;

coeffs
{
    n           (1 5 1);
}

blockMesh
decomposePar
mpirun -np 5 simpleFoam -parallel
or
mpiexec -np 5 simpleFoam -parallel
reconstructPar

Change the number of processors according to your machine

9. Load OpenMPI and OF automatically

You will need to modify the server bashrc to load the OF and the OpenMPI whenever you login in to it so you won't have to load it manually every time.

Add these lines to the end of ~/.bashrc

alias of2012='alias of2012='module load OpenMPI/4.0.5-GCC-10.2.0 || export PATH=$PATH:/usr/lib64/openmpi/bin; source /home70/gfelix/opt/OpenFOAM-v2012/etc/bashrc $FOAM_SETTINGS'

When you login again just type

of2012

to load it.

Now the OF is ready to run your cases. You just gotta transfer your run files with scp as done in step 1.

NOTE: You might want to set up a passwordless login in order to avoid having to type your user and password everytime you login to the server. Just follow this guide How to Setup Passwordless SSH Login

NOTE 2: You can check the information about the cluster's nodes and number of processors with the following commands:

Number of processors (CPU)
	grep -c ^processor /proc/cpuinfo
Infos about the nodes
	cat /proc/cpuinfo

Hi,

I encounter the same problem as you are. I followed almost the same route as what you did, however, if I'll run parallel processing it seems like all the cores in our cluster are hosting. What I am saying is, for example, if I'll ask 1 node which has 20 cores, then all these cores are hosting when in fact 19 cores must be slaves while one is the master. Have you encountered these issues?

Regards,
Mike