[nzy102]

Hi guys:

I have been stuck with a parallel processing on openfoam 1.4.1. I tested a case with 2.3 million cells. After tested several different node numbers, I found that a lot of time has been wasted by using openmpi. Then I decided to switch to other mpich packages on my school cluster. Here is what I did:

=================================================
export MPICH_PATH=/usr/global/mpich-infinicon
export MPICH_ARCH_PATH=$MPICH_PATH/platforms/$WM_OPTIONS
export MPICH_ROOT=$MPICH_ARCH_PATH

AddLib $MPICH_ARCH_PATH/lib
AddPath $MPICH_ARCH_PATH/bin

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpich-infinicon
=================================================

The problem is I don't have permission to write in /global/mpich-infinicon. I am wondering if I really have to put "platforms/$WM_OPTIONS", "lib" and "bin" in the global directory or there is other way around. Thank you for help.

Ning

[mattijs]

It is the AddLib & AddPath I assume that give the problems since they do an 'mkdir' under the hood. Remove the mkdir from the AddLib and AddPath and see if that works.

[nzy102]

I tried ./Allwmake in ..src/Pstream directory. I got an error:
=================================================
+ '[' MPICH = LAM -o MPICH = OPENMPI -o MPICH = MPI -o MPICH = MPICH -o MPICH = MPICH-GM -o MPICH = MPIGAMMA ']'
+ export WM_OPTIONS=linux64GccDPOptMPICH
+ WM_OPTIONS=linux64GccDPOptMPICH
+ wmake libso mpi
/usr/bin/ld: cannot find -lmpich
collect2: ld returned 1 exit status
=================================================
Then I followed the link (http://openfoamwiki.net/non_cdl/Open...mpile_OpenFOAM)and installed binutil-2.18. I got the error as follows:
=================================================
+ '[' MPICH = LAM -o MPICH = OPENMPI -o MPICH = MPI -o MPICH = MPICH -o MPICH = MPICH-GM -o MPICH = MPIGAMMA ']'
+ export WM_OPTIONS=linux64GccDPOptMPICH
+ WM_OPTIONS=linux64GccDPOptMPICH
+ wmake libso mpi
/home1/nzy102/OpenFOAM/linux64/binutils-2.18/bin/ld: cannot find -lmpich
collect2: ld returned 1 exit status
make: *** [/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPs tream.so] Error 1
=================================================

[msrinath80]

Check whether /usr/global/mpich-infinicon/lib contains *.a (static) or *.so (shared) libraries. I am guessing that it contains only *.a files. To build libPstream.so successfully, you need *.so files in there. Mattijs, please correct me if I am wrong.

[nzy102]

Thanks,Srinath. I changed my .bashrc and now I made a new libPstream.so. This is what I used in my .bashrc:

=================================================

# export MPICH_VERSION=infinicon-3.1.0.0.35G
export MPICH_PATH=/usr/global/mpich-infinicon
export MPICH_ARCH_PATH=$MPICH_PATH
# export MPICH_ARCH_PATH=$MPICH_PATH/platforms/$WM_OPTIONS
export MPICH_ROOT=$MPICH_ARCH_PATH


# AddLib $MPICH_ARCH_PATH/lib
# AddPath $MPICH_ARCH_PATH/bin

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpich-infinicon
# export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mvapich-$MPICH_VERSION

=================================================

But when I submit my jobs using pbs file, I got an error:

=================================================
icoFoam: symbol lookup error: /home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined symbol: FStatusText
icoFoam: symbol lookup error: /home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined symbol: FStatusText
=================================================

Anybody knows what this means? For the new mpich, what command I should use for parallel?
Should I still use something like:

mpirun -np 2 icoFoam /home1/nzy102/OpenFOAM/nzy102-1.4.1/run/tutorials/icoFoam lvad_90_0_5_grid2_np2 -parallel

Thank you.

Ning

[msrinath80]

Replace icoFoam with `which icoFoam` and it should work. MPICH usually needs the full path to icoFoam specified when using the mpirun command. e.g:

mpirun -np 2 `which icoFoam` /home1/nzy102/OpenFOAM/nzy102-1.4.1/run/tutorials/icoFoam lvad_90_0_5_grid2_np2 -parallel

[msrinath80]

You may also need to specify a machinefile:

mpirun -np 2 -machinefile mymachinefile `which icoFoam_1` . mycase -parallel

Where mymachinefile contains:
hostname:number of processors

e.g.:
ning:2

Or you can give $PBS_NODEFILE instead of mymachinefile. e.g.:

mpirun -np 2 -machinefile $PBS_NODEFILE `which icoFoam_1` . mycase -parallel

[msrinath80]

Oh, and after you change to a different MPI library, go to your application source and recompile the solver. For icoFoam, you need to go to ~/OpenFOAM/OpenFOAM-1.4/applications/solvers/incompressible/icoFoam/

and execute wclean && wmake there. That should take care of your symbol lookup errors, I think?

[nzy102]

Thank you for quick replied, Srinath. I compiled my icoFoam and got:
=================================================
Making dependency list for source file icoFoam.C
SOURCE=icoFoam.C ; g++ -m64 -Dlinux64 -DDP -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -march=opteron -O3 -DNoRepository -ftemplate-depth-40 -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/finiteVolume/lnInclude -IlnInclude -I. -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/OpenFOAM/lnInclude -fPIC -c $SOURCE -o Make/linux64GccDPOpt/icoFoam.o
In file included from icoFoam.C:105:
computeMassFlux.H:17:2: warning: no newline at end of file
g++ -m64 -Dlinux64 -DDP -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -march=opteron -O3 -DNoRepository -ftemplate-depth-40 -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/finiteVolume/lnInclude -IlnInclude -I. -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/OpenFOAM/lnInclude -fPIC Make/linux64GccDPOpt/icoFoam.o -L/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt \
-lfiniteVolume -lOpenFOAM -liberty -ldl -lm -o /home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/applications/bin/linux64GccDPOpt/icoFoam
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `FStatusText'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_prepare'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_wc_status_msg'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_clear_async_event_handler'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_get_iba_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_create_qp'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `FCmStatusText'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_event_syndrome_sym'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_clear_comp_eventh'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_deregister_mr'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_query_qp'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_set_async_event_handler'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_alloc_pd'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_strerror'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_disconnect'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_poll_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_poll_cq_wait'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_finalize'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_create_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_destroy_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_post_recv2'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_post_send2'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_destroy_qp'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_register_mr'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_set_comp_eventh'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_wc_type_msg'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_event_record_sym'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_dealloc_pd'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_init'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_connect'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_get_iba_qp'
=================================================
Something is messed up with my new libpstream.so.

Ning

[msrinath80]

computeMassFlux.H is not a standard part of icoFoam is it? Are you sure that it has been parallelized properly?

Try this. Recompile the original icoFoam solver and see if you get these errors. I'm sorry I cannot offer more guidance. I don't know what could be wrong. Please wait for Mattijs to respond. He is the parallel expert

[nzy102]

Srinath:

ComputeMassFlux works with openmpi. You can ignore the computemassflux. Thank you for all the suggestions.

Mattijs:

Can you take a look at this case when you get a chance?

Thank you.

[mbeaudoin]

Hello Ning,

1: By commenting out
# AddLib $MPICH_ARCH_PATH/lib

you won't be able to run your MPI application because your LD_LIBRARY_PATH environment variable will not be set properly to find the MPI runtime libraries needed.

2: By commenting out
# AddPath $MPICH_ARCH_PATH/bin

you won't be able to use the proper version of mpirun associate with your local version of MPI. You might (and probably will) run into problems.

3: About your compilation problems and the numerous error messages associated to libPstream:

You are probably missing one or more libraries that need to be specified with libmpich.

Find someone at your location who has already compiled an application with your local version of MPICH, and look at the list of libraries that need to be specified for linking such an application.

Martin

[nzy102]

Hi Martin:

I linked the library required to icofoam solver and now it works fine. I still have addlib and addpath commented out and hasn't seen any problem so far. Since I don't have permission to write in the global directory, I am gonna leave it there. Thank you.

Ning

[mbeaudoin]

Hello Ning,

For your information, you don't need write access to your global /global/mpich-infinicon directory.

I am using a local openmpi version on my cluster, and I don't have write access there either. Don't need to.

Addlib() and AddPath() are defined in the file $WM_PROJECT_DIR/.bashrc.

Take a look at their definition:

AddPath(){
export PATH=$1:$PATH
if [ ! -d $1 ]; then
mkdir -p $1
fi
}

AddLib(){
export LD_LIBRARY_PATH=$1:$LD_LIBRARY_PATH
if [ ! -d $1 ]; then
mkdir -p $1
fi
}

See the lines in red? What this means is that if the path used as parameter to the functions AddLib and AddPath don't exists, then mkdir will try to create it.

So if you use a valid path when calling AddLib and AddPath, mkdir will never ever get called in the first place.

Check the validity of your paths when using AddLib and AddPath.

From your example, the paths you are using are probably wrong in the first place because
$MPICH_ARCH_PATH/lib would expand to
$MPICH_PATH/platforms/$WM_OPTIONS/lib which would probably looks like
/usr/global/mpich-infinicon/platforms/linuxGccDPOpt/lib.

I doubt this is a valid path on your system, hence the problem with mkdir when using AddLib.

What you probably want instead in your .bashrc is :

export MPICH_ROOT=$MPICH_PATH
AddLib $MPICH_PATH/lib
AddPath $MPICH_PATH/bin

Your cluster sysadmin probably set up your initial user environment properly for using the /global/mpich-infinicon libraries and binaries by default, this is why your setup does work even though the two important lines Addlib and AddPath are commented out.

Martin

[nzy102]

Now everything is clear to me. Thank you so much.

Ning