System OpenMPI on RHEL Cluster

pizzaice · October 12, 2011, 08:07

Hi all,

I am trying to get OF 2.0.1 to run on our RHEL cluster. However, I am facing issues when trying to compile OF with the SYSTEMOPENMPI setting.

I have changed to SYSTEMOPENMPI in the bashrc. However, when I source it I get:

Code:

/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status

I then adjusted preferences.sh with adding
export MPI_HOME=/opt/platform_mpi/
export MPI_ARCH_PATH=/opt/platform_mpi/

But I am getting the following error when compiling:

Code:

========================================
Done ThirdParty Allwmake
========================================

+ wmakePrintBuild -check
no git description found
+ /bin/rm -f OpenFOAM/Make/linux64Gcc44DPOpt/global.C OpenFOAM/Make/linux64Gcc44DPOpt/global.o
+ wmakeLnInclude OpenFOAM
+ wmakeLnInclude OSspecific/POSIX
+ Pstream/Allwmake
+ wmake libso dummy
'/opt/OpenFOAM/OpenFOAM-2.0.1/platforms/linux64Gcc44DPOpt/lib/dummy/libPstream.so' is up to date.
+ case "$WM_MPLIB" in
+ set +x

Note: ignore spurious warnings about missing mpicxx.h headers

wmake libso mpi
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status

Can anyone help please?

Thanks

wyldckat · October 12, 2011, 17:34

Hi pizzaice,

I don't know any better way to say this, but that doesn't look good... not good at all. According to your description and the error messages, it would seem that you have incompatible binaries running in your environment. My first guess would be the MPI binaries, namely mpirun that isn't built properly for your system.

What happens when you run mpirun in a terminal? Does it give out the same error you've gotten?

Another big problem is that Gcc 4.1.2 does not work with OpenFOAM! At least not without some serious tinkering...
You'll have to request your system administrator to install another Gcc version more recent than 4.3.2.

Another possibility, since you are using RHEL, is the CentFOAM project: http://sourceforge.net/apps/mediawik...itle=Main_Page

Best regards and good luck!
Bruno

pizzaice · November 16, 2011, 12:57

Hi Bruno,

Thanks very much for the advice. Yes, you were right, it was an issue with openmpi.. I have now used gcc45 to build the sources and a newly compiled openmpi .. This seems to have fixed the issue!

Thanks again and happy foaming :-)

Chris

kjmaki · October 26, 2012, 11:25

I have a similar problem compiling openfoam 2.1.x on a cluster using the SYSTEMOPENMPI. Everything compiles except for the Pstream. Here is the output from the Allwmake in the src/Pstream directory:

login2$ ./Allwmake
+ wmake libso dummy
'/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/dummy/libPstream.so' is up to date.
+ case "$WM_MPLIB" in
+ set +x

Note: ignore spurious warnings about missing mpicxx.h headers

wmake libso mpi
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
SOURCE=UOPwrite.C ; g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3 -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -IlnInclude -I. -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude -fPIC -c $SOURCE -o Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o
UOPwrite.C:29:17: error: mpi.h: No such file or directory
In file included from UOPwrite.C:32:
PstreamGlobals.H:55: error: ‘MPI_Request’ was not declared in this scope
PstreamGlobals.H:55: error: template argument 1 is invalid
PstreamGlobals.H:55: error: invalid type in declaration before ‘;’ token
UOPwrite.C: In static member function ‘static bool Foam::UOPstream::write(Foam::UPstream::commsTypes, int, const char*, std::streamsize, int)’:
UOPwrite.C:61: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:64: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:65: error: ‘MPI_Bsend’ was not declared in this scope
UOPwrite.C:81: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:84: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:85: error: ‘MPI_Send’ was not declared in this scope
UOPwrite.C:97: error: ‘MPI_Request’ was not declared in this scope
UOPwrite.C:97: error: expected ‘;’ before ‘request’
UOPwrite.C:103: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:106: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:107: error: ‘request’ was not declared in this scope
UOPwrite.C:108: error: ‘MPI_Isend’ was not declared in this scope
UOPwrite.C:115: error: request for member ‘size’ in ‘Foam::PstreamGlobals:

utstandingRequests_’, which is of non-class type ‘int’
UOPwrite.C:119: error: request for member ‘append’ in ‘Foam::PstreamGlobals:

utstandingRequests_’, which is of non-class type ‘int’
make: *** [Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o] Error 1

I have tried to customize the settings.sh file to properly set the PLIBS and PINC variables. My system-version of mpicc does not have --showme:compile and --showme:link options, so I have used the mpicc -compile-info to generate the paths to include and library directories and compiler flags. This is how my settings.sh file appears:

export FOAM_MPI=openmpi-system

# Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI
#kjm export PINC="`mpicc --showme:compile`"
#kjm export PLIBS="`mpicc --showme:link`"
export PINC=-I/opt/apps/gcc4_4/mvapich2/1.6/include
export PLIBS='-L/opt/apps/gcc4_4/mvapich2/1.6/lib
-L/opt/apps/limic2/0.5.4//lib
-L/opt/ofed/lib64/
-lmpich -lopa -llimic2 -lpthread
-lrdmacm -libverbs -libumad -ldl -lrt'

#kjm libDir=`echo "$PLIBS" | sed -e 's/.*-L$[^ ]*$.*/\1/'`
libDir=/opt/apps/gcc4_4/mvapich2/1.6/lib

# Bit of a hack: strip off 'lib' and hope this is the path to openmpi
# include files and libraries.
export MPI_ARCH_PATH="${libDir%/*}"
export MPI_HOME=MPI_ARCH_PATH

Any help is greatly appreciated!

Thanks!

Kevin

wyldckat · October 26, 2012, 19:43

Greetings Kevin,

From the output you've provided, it looks like you didn't update the shell environment for it to reflect the changes you've made to "settings.sh". I say this because the following line of output does not show your custom settings:

Quote:

Code:

SOURCE=UOPwrite.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra  -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3   -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX  -IlnInclude -I.  -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude  -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude    -fPIC -c $SOURCE -o Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o

The simplest way is to start a new terminal and try again. You can confirm if the settings you've made are active by running:

Code:

echo $PINC
echo $PLIBS

Best regards,
Bruno

kjmaki · October 27, 2012, 09:29

Thanks for your interest Bruno,

I have checked that the values of the shell env variables are as I set them in the settings.sh file. It still doesnt work. When I look in the Pstream/mpi/Make/options file, I see that I need a PFLAGS variable. When I echo $PFLAGS, it is empty. I grep for PFLAGS in the etc directory, and I don't see where it is set. Do you think this is what is causing my problem?

wyldckat · October 27, 2012, 12:57

Hi Kevin,

"PFLAGS" is loaded via these two lines on the "Make/options" file:

Code:

sinclude $(GENERAL_RULES)/mplib$(WM_MPLIB)
sinclude $(RULES)/mplib$(WM_MPLIB)

This means that this flag comes from "wmake/rules/General/mplibSYSTEMOPENMPI", which has got this:

Code:

# use the system installed openmpi - mpicc must be installed

PFLAGS     = -DOMPI_SKIP_MPICXX
PINC       = $(shell mpicc --showme:compile)
PLIBS      = $(shell mpicc --showme:link)

Well that explains the problems you're having!! Try commenting out or removing the "PINC" and "PLIBS" lines and you should be good to go!

Best regards,
Bruno

kjmaki · October 28, 2012, 09:13

Thanks for the hand Bruno,

Your suggestion was necessary for me to get it to work. I also had the issue that the -showme:compile option is not available for the mpicc on this cluster. The -compile-info option is a close alternative, although the output with this flag includes extra info. Also, the 'mpicc -compile-info' and 'mpicc -link-info' commands produce the exact same output. Is this normal?

I was able to test a parallel case, and it ran, although it has error messages on exit that I suspect may be due to me incorrectly specifying the PINC and PLIBS variables.

End

Finalising parallel run
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.

================================================== ===================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
================================================== ===================================
APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)

wyldckat · October 28, 2012, 13:57

Hi Kevin,

I suggest that you check the other files for ideas on how to configure "PINC" and "PLIBS":

Code:

ls -l $WM_PROJECT_DIR/wmake/rules/General/mplib*

By the way, I reported about the duplicate entries of the two variables in "settings.sh" and in the "mplibSYSTEMOPENMPI" files: http://www.openfoam.org/mantisbt/view.php?id=673 - the official fix is to remove these two variables from "settings.sh" and define them only in the "mplibSYSTEMOPENMPI" file.
Said fix is this one: https://github.com/OpenFOAM/OpenFOAM...10f50d61d5793e

Best regards,
Bruno

October 12, 2011, 17:34		#2
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,981 Blog Entries: 45 Rep Power: 128	Hi pizzaice, I don't know any better way to say this, but that doesn't look good... not good at all. According to your description and the error messages, it would seem that you have incompatible binaries running in your environment. My first guess would be the MPI binaries, namely mpirun that isn't built properly for your system. What happens when you run mpirun in a terminal? Does it give out the same error you've gotten? Another big problem is that Gcc 4.1.2 does not work with OpenFOAM! At least not without some serious tinkering... You'll have to request your system administrator to install another Gcc version more recent than 4.3.2. Another possibility, since you are using RHEL, is the CentFOAM project: http://sourceforge.net/apps/mediawik...itle=Main_Page Best regards and good luck! Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

October 26, 2012, 11:25	compile 2.1.x with SYSTEMOPENMPI	#4
kjmaki Member Kevin Maki Join Date: Mar 2009 Location: Ann Arbor, MI, USA Posts: 43 Rep Power: 17	I have a similar problem compiling openfoam 2.1.x on a cluster using the SYSTEMOPENMPI. Everything compiles except for the Pstream. Here is the output from the Allwmake in the src/Pstream directory: login2$ ./Allwmake + wmake libso dummy '/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/dummy/libPstream.so' is up to date. + case "$WM_MPLIB" in + set +x Note: ignore spurious warnings about missing mpicxx.h headers wmake libso mpi /usr/lib/../lib64/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' collect2: ld returned 1 exit status /usr/lib/../lib64/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' collect2: ld returned 1 exit status /usr/lib/../lib64/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' collect2: ld returned 1 exit status SOURCE=UOPwrite.C ; g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3 -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -IlnInclude -I. -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude -fPIC -c $SOURCE -o Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o UOPwrite.C:29:17: error: mpi.h: No such file or directory In file included from UOPwrite.C:32: PstreamGlobals.H:55: error: ‘MPI_Request’ was not declared in this scope PstreamGlobals.H:55: error: template argument 1 is invalid PstreamGlobals.H:55: error: invalid type in declaration before ‘;’ token UOPwrite.C: In static member function ‘static bool Foam::UOPstream::write(Foam::UPstream::commsTypes, int, const char, std::streamsize, int)’: UOPwrite.C:61: error: ‘MPI_PACKED’ was not declared in this scope UOPwrite.C:64: error: ‘MPI_COMM_WORLD’ was not declared in this scope UOPwrite.C:65: error: ‘MPI_Bsend’ was not declared in this scope UOPwrite.C:81: error: ‘MPI_PACKED’ was not declared in this scope UOPwrite.C:84: error: ‘MPI_COMM_WORLD’ was not declared in this scope UOPwrite.C:85: error: ‘MPI_Send’ was not declared in this scope UOPwrite.C:97: error: ‘MPI_Request’ was not declared in this scope UOPwrite.C:97: error: expected ‘;’ before ‘request’ UOPwrite.C:103: error: ‘MPI_PACKED’ was not declared in this scope UOPwrite.C:106: error: ‘MPI_COMM_WORLD’ was not declared in this scope UOPwrite.C:107: error: ‘request’ was not declared in this scope UOPwrite.C:108: error: ‘MPI_Isend’ was not declared in this scope UOPwrite.C:115: error: request for member ‘size’ in ‘Foam::PstreamGlobals:utstandingRequests_’, which is of non-class type ‘int’ UOPwrite.C:119: error: request for member ‘append’ in ‘Foam::PstreamGlobals:utstandingRequests_’, which is of non-class type ‘int’ make: ** [Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o] Error 1 I have tried to customize the settings.sh file to properly set the PLIBS and PINC variables. My system-version of mpicc does not have --showme:compile and --showme:link options, so I have used the mpicc -compile-info to generate the paths to include and library directories and compiler flags. This is how my settings.sh file appears: export FOAM_MPI=openmpi-system # Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI #kjm export PINC="`mpicc --showme:compile`" #kjm export PLIBS="`mpicc --showme:link`" export PINC=-I/opt/apps/gcc4_4/mvapich2/1.6/include export PLIBS='-L/opt/apps/gcc4_4/mvapich2/1.6/lib -L/opt/apps/limic2/0.5.4//lib -L/opt/ofed/lib64/ -lmpich -lopa -llimic2 -lpthread -lrdmacm -libverbs -libumad -ldl -lrt' #kjm libDir=`echo "$PLIBS" \| sed -e 's/.-L\([^ ]\)./\1/'` libDir=/opt/apps/gcc4_4/mvapich2/1.6/lib # Bit of a hack: strip off 'lib' and hope this is the path to openmpi # include files and libraries. export MPI_ARCH_PATH="${libDir%/}" export MPI_HOME=MPI_ARCH_PATH Any help is greatly appreciated! Thanks! Kevin

October 27, 2012, 12:57		#7
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,981 Blog Entries: 45 Rep Power: 128	Hi Kevin, "PFLAGS" is loaded via these two lines on the "Make/options" file: Code: sinclude $(GENERAL_RULES)/mplib$(WM_MPLIB) sinclude $(RULES)/mplib$(WM_MPLIB) This means that this flag comes from "wmake/rules/General/mplibSYSTEMOPENMPI", which has got this: Code: # use the system installed openmpi - mpicc must be installed PFLAGS = -DOMPI_SKIP_MPICXX PINC = $(shell mpicc --showme:compile) PLIBS = $(shell mpicc --showme:link) Well that explains the problems you're having!! Try commenting out or removing the "PINC" and "PLIBS" lines and you should be good to go! Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

October 28, 2012, 13:57		#9
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,981 Blog Entries: 45 Rep Power: 128	Hi Kevin, I suggest that you check the other files for ideas on how to configure "PINC" and "PLIBS": Code: ls -l $WM_PROJECT_DIR/wmake/rules/General/mplib* By the way, I reported about the duplicate entries of the two variables in "settings.sh" and in the "mplibSYSTEMOPENMPI" files: http://www.openfoam.org/mantisbt/view.php?id=673 - the official fix is to remove these two variables from "settings.sh" and define them only in the "mplibSYSTEMOPENMPI" file. Said fix is this one: https://github.com/OpenFOAM/OpenFOAM...10f50d61d5793e Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
CFX11 + Fortran compiler ?	Mohan	CFX	20	March 30, 2011 19:56
What is the best cluster file system to go with OF	sampaio	OpenFOAM Installation	4	March 17, 2011 05:10
How to install OpenFOAM15dev on RHEL AS3 cluster	waynezw0618	OpenFOAM Installation	0	February 26, 2009 11:10
How to install OpenFOAM15dev on RHEL AS3 cluster	waynezw0618	OpenFOAM Installation	0	February 26, 2009 11:04
Installation on a cluster running a queueing system PBS	sampaio	OpenFOAM Installation	12	June 20, 2005 18:16

November 16, 2011, 12:57		#3
pizzaice New Member Join Date: Nov 2009 Posts: 15 Rep Power: 17	Hi Bruno, Thanks very much for the advice. Yes, you were right, it was an issue with openmpi.. I have now used gcc45 to build the sources and a newly compiled openmpi .. This seems to have fixed the issue! Thanks again and happy foaming :-) Chris

October 27, 2012, 09:29		#6
kjmaki Member Kevin Maki Join Date: Mar 2009 Location: Ann Arbor, MI, USA Posts: 43 Rep Power: 17	Thanks for your interest Bruno, I have checked that the values of the shell env variables are as I set them in the settings.sh file. It still doesnt work. When I look in the Pstream/mpi/Make/options file, I see that I need a PFLAGS variable. When I echo $PFLAGS, it is empty. I grep for PFLAGS in the etc directory, and I don't see where it is set. Do you think this is what is causing my problem?

October 28, 2012, 09:13		#8
kjmaki Member Kevin Maki Join Date: Mar 2009 Location: Ann Arbor, MI, USA Posts: 43 Rep Power: 17	Thanks for the hand Bruno, Your suggestion was necessary for me to get it to work. I also had the issue that the -showme:compile option is not available for the mpicc on this cluster. The -compile-info option is a close alternative, although the output with this flag includes extra info. Also, the 'mpicc -compile-info' and 'mpicc -link-info' commands produce the exact same output. Is this normal? I was able to test a parallel case, and it ran, although it has error messages on exit that I suspect may be due to me incorrectly specifying the PINC and PLIBS variables. End Finalising parallel run interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed. interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed. interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed. interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed. ================================================== =================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 6 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ================================================== =================================== APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)