[elisabet]

Hi all,

I'm used to work with OF using OpenMPI, but now I have access to a cluster that uses MPICH2. Hence, I guess some specific changes have to be done in both bashrc and settings.sh files. However, among the available options I only find MPICH, is it ok to keep MPICH but in settings.sh introduce the right MPICH2 version and location?

Moreover, should I have something else in mind? That is, are there special treats for Pstream compilation with PMICH2? I have not been able to find the information I need in previous posts.

Any information is welcome!

Regards,

elisabet

[elisabet]

Still with this...

When compiling Pstream I get the following error:

/usr/bin/ld: /shared/source/mpich2-1.0.6p1/lib/libmpich.a(allreduce.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/shared/source/mpich2-1.0.6p1/lib/libmpich.a: could not read symbols: Bad value
collect2: ld returned 1 exit status

I guess it has something to do with the rules in my mplibMPICH file, as stated in the wiki, but I don't know how to correct it. Any ideas?

Regards,

elisabet

[aunola]

I have a cluster running OF-1.5 in parallel using MPICH2. I had no problems rebuilding PStream. It's been a while since I did it, and I don't remember the exact steps. However, I did take notes. I don't have them at hand right now, but I will try to dig them out.

[elisabet]

That would be perfect!

Thanks Martin,

elisabet

[aunola]

Turns out my notes are for OF-1.4.1. I had a quick look at OF-1.5.x. The names and contents of sources files is different, but this is what I believe you should do, assuming your shell is bash:

1) Open $WM_PROJECT_DIR/etc/bashrc

Uncomment the line (put a # in front of it)

: ${WM_LIB:=OPENMPI}; export WM_LIB

and add this line

: ${WM_LIB:=MPICH2}; export WM_LIB

This will make MPICH2 the default MPI selection.


2) In $WM_PROJECT_DIR/etc/settings.sh find where all the MPI stuff is and add this (your paths and version may differ):

MPICH2)

export MPICH2_VERSION=1.0.5
export MPICH2_ARCH_PATH=/home/mar/mpich2-install
export MPICH2_PATH=$MPICH2_ARCH_PATH
export MPICH2_ROOT=$MPICH2_ARCH_PATH
_foamAddLib $MPICH2_ARCH_PATH/lib
_foamAddLib $MPICH2_ARCH_PATH/bin
export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpich2-$MPI_VERSION


3) In $WM_PROJECT_DIR/wmake/rules/linuxGcc add the new file mplibMPICH2 with contents:

PFLAGS =
PINC = -I$(MPICH2_ARCH_PATH)/include
PLIBS = -L$(MPICH2_ARCH_PATH)/lib

This file tells the compiler and linker where to find the include files and MPICH2 libraries respectively.


4) In $WM_PROJECT_DIR/src/Pstream/Allwmake add MPICH2 to the case statement:

... | MPICH2 | ...)



5) In $WM_PROJECT_DIR/src/Pstream

./Allwmake


Let me know if there are problems. Later this year I am planning on building a new cluster with OF running MPICH2-MX on top of OpenMX, so I have a vested interest in any observations you make.

[elisabet]

Hi Martin,

Sorry for the delay, I've been quite busy with other stuff.

OF-1.5.x, with MPICH2, is now compiled in the cluster and works. The steps I carried out are:

1. In etc/bashrc

# : ${WM_MPLIB:=OPENMPI

; export WM_MPLIB
: ${WM_MPLIB:=MPICH}; export WM_MPLIB # I kept MPICH for simplicity, I'm too lazy...
}

2. In etc/settings.sh (following the internal architecture of the cluster)

MPICH)
mpi_version=mpich2-1.0.6p1
export MPI_HOME=/shared/source/$mpi_version
export MPI_ARCH_PATH=/shared
export MPICH_ROOT=$MPI_ARCH_PATH

_foamAddPath $MPI_ARCH_PATH/bin
_foamAddLib $MPI_ARCH_PATH/lib

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/$mpi_version
unset mpi_version
;;

3. Pstream library compilation
My old error related with fPIC was solved when I asked the cluster administrator to recompile MPICH2 in a shared way. Once this was changed, everything complied directly.

4. others
I got a strange error when compiling decompositionAgglomeration that was solved, as suggested by the error message, changing the order of the include mpi.h in the top of the file decompositionAgglomeration/parMetisDecomp/parMetisDecomp.C. The error was related to redefinition of SEEK_SET, SEEK_CUR and SEEK_END in mpi.h

Nowadays the code works perfectly in parallel using MPICH2. Thanks to everyone!

Best regards,

elisabet

[lakeat]

This is how I configure the MPICH2:

./configure \
    --without-mpe \
    --disable-f77 \
    --disable-f90 \
    --disable-f90modules \
    --disable-cxx \
    --disable-mpedbg \
    --disable-devdebug \
    --disable-debug \
    --enable-fast \
    --enable-sharedlibs=gcc \
    --with-device=ch3:nemesis \
    -prefix=$MPI_ARCH_PATH

According to the installation guide of mpich2, enable-fast and with-device=ch3:nemesis options are said to perform better, and enable-sharedlibs=gcc option is the only option for linux, I'm afraid, and it would solve the problem of fPIC complaining.