[gwierink]

Dear all,

I am trying to compile OpenFOAM-1.6 on a Cray XT5 machine and so far managed to compile it with help of this document. I also tried a static build, after this thread, but that didn't work out.
Now solvers etc work, but not in parallel. That kind of defeats the purpose of installation on a cluster, doesn't it? The trouble is with Pstream, it does not compile properly. I went back to ThirdParty and found out that I overlooked that metis-parmetis did not compile. When I compile $WM_THIRD_PARTY_DIR/ParMetis-3.1 ld gives an error that it can't find lpmi while compiling METISLib (which should build metis-parmetis.so). Would anyone have an idea how to fix this? Many thanks in advance!

[gwierink]

Ok, an update:

I found out that libpmi.so lives in /opt/cray/pmi/default/lib64, so I changed $WM_PROJECT_DIR/wmake/rules/crayxt/mplibMPT from

PFLAGS     = 
PINC       = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L$(MPICH_PATH)/pmi/lib -L$(MPICH_PATH)/util/lib -lmpich -lpmi -lalpslli -lalpsutil -lrt

to

PFLAGS     =
PINC       = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L/opt/cray/pmi/default/lib64 -L/opt/cray/stat/1.1.0/lib/alps -lmpich -lpmi -lalpslli -lalpsutil -lrt

to reflect the right paths (I hope). Now ThirdParty and Pstream compiled without error!! Whoohoo!

The problem I have now is that there is no mpirun, only mpigammarun and mpirunDebug. Perhaps a noob question, but does anyone know if this is normal or not and how to use mpigammarun/mpirunDebug? I mean, debug is slower, right?

[wyldckat]

Greetings Gijsbert,

I don't have any experience with Crays (but I do wish I had ), but... pmi? "Process Manager Interface"? According to this presentation and also indicated by your posted code snippets, you're looking for MPICH, not pmi

So, if I'm not mistaken:

1. You should take out all references in your mplibMPT file to pmi.
2. Find where your system's MPICH lives, probably by running:

   echo $MPICH_PATH

3. Edit the file $WM_PROJECT_DIR/etc/settings.sh. Search for "mpich" and update the definitions it has for it.
4. Edit the file $WM_PROJECT_DIR/etc/bashrc and make sure you set properly the variable WM_MPLIB to MPICH.
5. Source the bashrc file:

   . $WM_PROJECT_DIR/etc/bashrc

   or start a new terminal. I personally always start a new terminal... seems to be the most efficient away to make sure that the changes a properly implemented.
6. Rebuild the Third Party tools/libraries and OpenFOAM, just to make sure things are set into place (note: if you feel the need to backup before this step, you should!):

   cd $WM_THIRD_PARTY_DIR
   wclean all
   cd $WM_PROJECT_DIR
   wclean all
   Allwmake > make.log 2>&1

   This way you keep a log of the build process. It'll make your life easier to assert what went wrong
   NOTE: if while running "wclean all" in the ThirdParty folder there are some errors about "gmake not found", it's not really important either so you can ignore those errors

Note: the link for that presentation at the top of my post also has some interesting optimization tweaks for MPICH in CrayXT!

Good luck! Best regards,
Bruno

[gwierink]

Hi Bruno,

Thank you for your reply! The presentation you mentioned looks very useful, I wil definitely have a more detailed look at it.

you're looking for MPICH, not pmi

Actually, I was looking for pmi, not MPICH . When I compiled ParMetis in ThirdParty and Pstream, the compiler complained that ld could not find -lpmi. The reason is that I needed to set up a new compiler in wmake/rules, called crayxt (well, of course I could call it banana, but crayxt seemed more appropriate). According to a friend of mine at CSC and this document a new MPI needs to be defined in settings.sh, namely:

MPT)
    export MPICH_PATH=$MPICHBASEDIR
    export MPI_ARCH_PATH=$MPICH_DIR
    export MPICH_ROOT=$MPI_ARCH_PATH
    _foamAddLib $MPI_ARCH_PATH/lib
    _foamAddLib $MPICH_PATH/pmi/lib
    _foamAddLib $MPICH_PATH/util/lib
    export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpt
    ;;

Now, the trouble is that on the system I was installing OF the pmi and alpsutil libraries moved to a different place. Therefore, they did not live under $MPICH_PATH anymore. The solution was to adjust the paths above to reflect the ones on the local machine and also edit $WM_PROJECT_DIR/wmake/rules/crayxt/mplibMPT to look like:

PFLAGS     =
PINC       = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L/opt/cray/pmi/default/lib64 -L/opt/cray/stat/1.1.0/lib/alps -lmpich -lpmi -lalpslli -lalpsutil -lrt

With these adjustments OF ThirdParty and Pstream compiled without any error .

Then, the other issue I mentioned above was how to use mpirun. After successfully compiling Pstream, there was still no mpirun. It turned out that after decomposing with decomposePar I could just write a qsub script to submit the job, where the executing line looks like

aprun -n <nr_of_processors> <solver> -parallel > log.txt

As simple as that! Now it's crunching time .

[namCFD]

Hello Gijs,

I am trying to install 1.7.x on Cray XT4 but running into trouble: /usr/bin/lid cannot find lOpenFOAM. Is it possible to for you to list what you have done so I can follow to install?

I am using c-shell (csh) instead of bash and PrgEnv-gnu instead of PrgEnv-pgi. gcc is 4.4.1

I was able to install 1.7.1 but it is not working in parallel with aprun. In 1.7.1, I can use mpirun.

Thank you for your help.

[gwierink]

Hi Nam,

I made a short report on how I compiled OpenFOAM-1.6.x on Cray XT, you can find it here. It lists all the changes etc that worked for me, hope it's helpful for you too.

[namCFD]

Hello Gijs,

After modifying your write-up to fit my system, I got the following error while running a batch job using "aprun": aprun -n 8 icoFoam -parallel with cavity tutorial. The error message is:

icoFoam: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

Do you have any hints on how to solve this error? I tried including the directory in the LD_LIBRARY_PATH in my batch script but it is not working.

Thanks,

Nam

[gwierink]

Hi Nam,

I tried including the directory in the LD_LIBRARY_PATH in my batch script but it is not working.

Ok, so what did you do exactly? You know where libstdc++.so.6 lives and you added it to LD_LIBRARY_PATH?

[namCFD]

Hi Gijs,

Update: I contacted the system administrator and indeed libstdc++.so.6 does not exist in the compute node. Do you have any ideas on how to solve this issue?
----------------------

First of all, thank you for your time and help.

I could not get OpenFOAM-1.7.x compiled correctly on Cray XT4 but OpenFOAM-1.7.1 is ok after using foamInstallationTest command. I don't know why.

I know that the libstdc++.so.6 lives in /usr/lib64 in my system. So, I did this in my batch script: export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH.

Do you think /usr/lib64 might only exist in the log-in node but not on compute nodes?

Thanks,

[namCFD]

Hello Gijs,

Can you help me with this error message?

bool Pstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

From function Pstream::init(int& argc, char**& argv)
in file Pstream.C at line 73.

Thanks,

[gwierink]

Hi Nam,

bool Pstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

This looks like a run time error, is that correct? If so, did you decompose the case with decomposePar?

[namCFD]

Hi Gijs,

I did decompose the case before submitting the case to HPC with aprun. However, we compiled OpenFOAM using openMPI that is in the ThirdParty folder. I think this is the problem because the compute nodes do not have openMPI, only MPICH2. Do you think this is the cause? We are trying to recompile with MPICH2 but as you already know, it is not a simple process. We are now trying to install OpenFOAM on Cray XE6. Do you have any pointers?

Thanks,

[SMatt]

Originally Posted by gwierink

Hi Nam,

I made a short report on how I compiled OpenFOAM-1.6.x on Cray XT, you can find it here. It lists all the changes etc that worked for me, hope it's helpful for you too.

I know this is an old thread, but I am struggling with getting OpenFOAM 2.1.1 to work in parallel on a Cray XE6. The links in the above post are broken.

Can anybody point me to any other threads on how to make the install work, or should I just follow the steps described above by gwierink?

Thanks.
Silvia