[deji]

Hey there FOAMers. I have been struggling trying to compile OpenFOAM 1.6 on an HPC computer and in need of some serious feedback .
The machine is 64 bit and is tcsh shell. Has anyone done anything remotely close to this, compiling OpenFOAM on an HPC machine?

Deji

[arkangel]

where exactly is the problem ? I have it in altix4700 altix3700 and SGI-ICE

[deji]

Third party software
-------------------------------------------------------------------------------
Software Version Location
-------------------------------------------------------------------------------
WARNING: gcc version does not match gcc supplied with this release of OpenFOAM
Supplied version: 4.3.3
User version : 4.4.3
Minimum required: 4.3.1

gcc 4.4.3
WARNING: Conflicting installations:
OpenFOAM settings : /home/fabio/OpenFOAM/ThirdParty-1.6/gcc-4.3.3/platforms/linux64/bin/gcc
current path : /usr/bin/gcc
CRITICAL ERROR

gzip 1.3.12 /bin/gzip
tar 1.22 /bin/tar
icoFoam 1.6 .../OpenFOAM-1.6/applications/bin/linux64GccDPOpt/icoFoam


I found this error on the forum and it is exact same critical that I received when I attempted to compile the code this morning. I am not using a Bash shell and just not sure how to get around this error on the Altix machine.

[deji]

And I must say that I am compiling OpenFOAM 1.6.x.

[fs82]

We have installed OpenFOAM 1.6.x on out SGI Altix 4700 and it was really time consuming. SGI has some special features, e.g. a SGI-MPI. Also you have to switch your compiler to the intel c++ compiler because of the Itanium2 CPU. This CPU has some features which are not supported by the gcc compiler. The performance of OpenFOAM on an Altix with gcc is horrible.
Please have a look here: http://www.cfd-online.com/Forums/ope...ation-icc.html
The first post of me discribes the changes you have to make for the SGI-MPI library and the switch of the compiler. May be you have to adapt the pathes.
Also interesting, but due to the fact you are allready using the GIT version of OpenFOAM, not important: http://www.cfd-online.com/Forums/ope...1-6-x-icc.html

kind regards,
fabian

[deji]

Thanks much for your reply. I should say that I was able to compile OpenFOAM 1.6.x with Intel 11.1 compiler. However, since I will be running jobs with mpi on an HPC machine, are my to compile the code with an mpi compiler? I was going to use intelmpi-4.0.0, but I do not know how to compile the code with mpi and how do I add mpiCC to my Allwmake files?

Thanks

[fs82]

Allright this is very easy because if I understand you right, you do not have to take care about the MPI compiler, this does OpenFOAM for you ... roughly speaking. Make sure that your $WM_PROJECT_DIR/etc/bashrc contains a section for the IA64 architecture and set the compiler to Icc (be sure that the intel compiler is installed correctly). Also you have to set the MPI which should be used. But this name is just a placeholder. You have to ensure that a file exits in $WM_PROJECT_DIR/wmake/rules/linuxIA64Icc/ which is named like: mplib<name you have used in $WM_MPLIB variable>. I used SGIMPI, but you could use MPI if you like or MPICC. The options of the MPI should be set in the mpilibMPI or mpilibSGIMPI or mpilibMPICC file. For options please have a look to the other files contained in this directory and ask you admin about this. Than compile Openfoam and set up a parallel case.
You have to use decompasePar first and than start the job with e.g. "mpirun (on our mechine its named pamrun) -n 4 pisoFoam -parallel" thats it. This worked for me perfectly and I am using up to 400 Cpus without problems and surprisingly good efficiency.

kind regards,
Fabian

[arkangel]

to use your own mpi you have to identify in etc/wmake/rules the rules you are using and basically tell wmake to use your own mpi

depending which MPI was loaded e.g echo $WM_MPLIB if MPI then edit mplibMPI if OPENMPI edit mplibOPENMPI, etc

mpi intel is based on mpich so you might force in you etc/bashrc or etc/setting to set WM_MPLIB = MPI, INTELMPI or whatever and use this settings


PFLAGS = -DMPICH_IGNORE_CXX_SEEK
PINC = -I/opt/intel.../include
PLIBS = -L/opt/inte/...lib -lmpi

[deji]

Thanks for the feedback guys, I will certainly give that a try.

Deji

[sebastian]

Hi Guys!

I am struggling with an annoying problem, while compiling OF-1.6-ext or OF-1.6.x on SGI Altix 4700 with the intel's compiler version 11.1.

Can't find anything helping me neither here in the threads nor on the internet.
No matter what I have tried so far, I can not get rid of these errors.

Hope anybody of you has an idea.

First, here are my changes in the settings (pretty much that what Fabian Schlegel did):

etc/bashrc:

# Compiler: set to Gcc, Gcc43 or Icc (for Intel's icc)
: ${WM_COMPILER:=Icc}; export WM_COMPILER

# WM_MPLIB = | OPENMPI | MPICH | MPICH-GM | HPMPI | GAMMA | MPI | QSMPI
# Compilation options
: ${WM_MPLIB:=MPI}; export WM_MPILIB (MPI für generic MPI, z.B. MPT)

# Detect system type and set environment variables appropriately
    ia64)
      export WM_CC='icc'
      export WM_CXX='icpc'
      export WM_CFLAGS='-fPIC-O2-fz'
      export WM_CXXFLAGS='-fPIC-O2-fz'
      WM_ARCH=linuxIA64
      export WM_COMPILER=Icc
;;

etc/settings.sh

# compilerInstall = OpenFOAM | System
compilerInstall=System

# Communications library
unset MPI_ARCH_PATH

export MPI_ARCH_PATH=/opt/sgi/mpt/mpt-1.26
export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpt-1.26

_foamAddLib $FOAM_MPI_LIBBIN

In the folder wmake/rules/linuxIA64Icc, I changed the content of mplibMPI to

PFLAGS	= -DSGIMPI -DMPI_NO_CPPBIND
PINC = -I/opt/sgi/mpt/mpt-1.26/include
PLIBS = -L/opt/sgi/mpt/mpt-1.26/lib –lmpi 

And now here, I always geht these kind of errors:

could not open file scotch.h for source file scotchDecomp.C
scotchDecomp.H(39): catastrophic error: could not open source file "decompositionMethod.H"
  #include "decompositionMethod.H"
compilation aborted for scotchDecomp.C (code 4)
make: *** [Make/linuxIA64IccDPOpt/scotchDecomp.o] Error 4

...
metisDecomp.H(39): catastrophic error: could not open source file "decompositionMethod.H"
  #include "decompositionMethod.H"

...
parMetisDecomp.H(38): catastrophic error: could not open source file "decompositionMethod.H"
  #include "decompositionMethod.H"
compilation aborted for parMetisDecomp.C (code 4)
make: *** [Make/linuxIA64IccDPOpt/parMetisDecomp.o] Error 4

...
ld: cannot find -lscotchDecomp
make: *** [/home/la2bsesa/OpenFOAM/OpenFOAM-1.6-ext/lib/linuxIA64IccDPOpt/libdecompositionMethods.so] Error 1

...
ld: cannot find -ldecompositionMethods
make: *** [/home/la2bsesa/OpenFOAM/OpenFOAM-1.6-ext/lib/linuxIA64IccDPOpt/libmeshTools.so] Error 1

...
ld: cannot find -lmeshTools
make: *** [/home/la2bsesa/OpenFOAM/OpenFOAM-1.6-ext/lib/linuxIA64IccDPOpt/libfiniteVolume.so] Error 1

...

and so on and so on.
The files mentioned here which are not possible to be opened are existing there, so no idea about whats going wrong.

I would by appreciated for every hint for what causes this problem or to solve it.
Thanks guys!

Sebastian

[deji]

I recently successfully compiled OpenFOAM 1.6.x on the SGI Altix machine that I have access to, and it took me roughly 2 months to do so. I do not think I utilized an Intel compiler for my compilation though. I can look into my files and if so send them to you to look at.

Deji

[fs82]

As far as I remember right, there was a problem with one decomposition method on 64 bit systems like the Altix 4700. I think this was the scotch library but I am not sure. Try the Allwmake script inside the ThirdParty directory and check for errors. If there are some concerning a specific decomposition method I would comment it out in the Allwmake script as a first guess and try it again :-D