[ZKM]

Hello all,
It is a while that I am trying to install OpenFOAM-1.5-dev on a cluster without any success. I use gcc-4.3.5 by loading it as a module because system’s default gcc is 3.4.1. I set WM_COMPILER_DIR accordingly. We have HPMPI and set that in cshrc file. I install in a directory other than the standard ~ directory due to speed limitation of home. I cleaned before reinstalling. Applications are not made but foamInstallationTest does not report any problem except:

 
Checking the OpenFOAM env variables set on the LD_LIBRARY_PATH...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory                Valid Path Crit
-------------------------------------------------------------------------------
$MPI_ARCH_PATH       /opt/hpmpi                                    yes   no  yes
-------------------------------------------------------------------------------

And its check for icoFoam apparently complains that there is no icoFoam.

I really appreciate if someone could have a look on the attached log file and advise me how to solve the problem.
Many thanks in advance
ZKM

P.S. Sorry I had to delete parts of the file due to file size restriction.

[gwierink]

Hi ZKM,

On line 113 of your log it reads

make: *** [/libPstream.so] Error 1

OpenFOAM can compile, even if PStream did not. Then, parallel runs are not possible. It may also be possible that some paths are not set correctly, so icoFoam is not found. Make sure that the MPI parths (like MPI_HOME) are set right, so that also the parallelization part of ThirdParty (metis, parmetis etc).
I have ported OpenFOAM to a large Cray cluster recently. The cluster is a bit different than yours I guess, but you may find my report on it useful. There it is explained what to change and how. Hope it helps!

[ZKM]

Hello all and thanks Gijs for your help

I am still in battle for installing it. I managed to find the source of the libPstream.so problem. For some reasons that I can't understand FOAM_MPI_LIBBIN is not set inside the switch ("$WM_MPLIB") in settings.csh. I set it manually again:

setenv FOAM_MPI_LIBBIN $FOAM_LIBBIN/hpmpi

It solves this problem but I still have this typical errors/warnings for all solvers:

+ cd applications
+ ./Allwmake
+ cd solvers
+ wmake all
make[2]: Nothing to be done for `application'.
make[2]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[2]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
Making dependency list for source file conjugateHeatFoam.C
Making dependency list for source file potentialFoam.C
SOURCE=potentialFoam.C ;  g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC -c $SOURCE -o Make/linux64GccDPOpt/potentialFoam.o
SOURCE=conjugateHeatFoam.C ;  g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/coupledMatrix/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC -c $SOURCE -o Make/linux64GccDPOpt/conjugateHeatFoam.o
g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/coupledMatrix/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC Make/linux64GccDPOpt/conjugateHeatFoam.o -L/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt \
     -lcoupledLduMatrix -lfiniteVolume -lOpenFOAM -liberty -ldl /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so   -lm -o /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/conjugateHeatFoam
g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC Make/linux64GccDPOpt/potentialFoam.o -L/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt \
     -lfiniteVolume -llduSolvers -lOpenFOAM -liberty -ldl /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so   -lm -o /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/potentialFoam
/usr/bin/ld: warning: libPstream.so, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libmpi.so.1, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so, not found (try using -rpath or -rpath-link)
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Finalize'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Buffer_detach'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlockattr_setkind_np'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_create'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_rdlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Bsend'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Probe'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutexattr_destroy'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Send'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_destroy'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_unlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Test'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Allreduce'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Buffer_attach'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Comm_size'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Recv'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Isend'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Get_count'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Abort'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Waitall'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Get_processor_name'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_wrlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlockattr_init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Comm_rank'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutexattr_settype'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Irecv'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_sum'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlockattr_destroy'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_trywrlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_tryrdlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutexattr_init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_byte'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_double'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutex_trylock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_packed'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_comm_world'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_join'
collect2: ld returned 1 exit status
make[2]: *** [/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/conjugateHeatFoam] Error 1
make[1]: *** [conjugateHeatFoam] Error 2
make[1]: Target `application' not remade because of errors.
make: *** [conjugate] Error 2
/usr/bin/ld: warning: libPstream.so, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libmpi.so.1, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so, not found (try using -rpath or -rpath-link)

I have no idea how to use -rpath or -rpath-link as suggested in the warning message.

I found out that libmpi.so.1 is located in two locations /opt/hpmpi/lib/linux_amd64 and /opt/hpmpi/lib/linux_amd64/nomalloc/

I am going to do something like this manually:

setenv LD_LIBRARY_PATH /opt/hpmpi/lib/linux_amd64:$LD_LIBRARY_PATH

but I am not sure which libmpi.so.1 is required.

As for the other warning, that I think ends in those errors, again I don't know which libPstream.so is needed (in lib/linux64GccDPOpt/dummy or in lib/linux64GccDPOpt/hpmpi) and how to tell it to use from that path.

Our admin advised me to add -lpthread after the "-ldl" in:

g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC Make/linux64GccDPOpt/potentialFoam.o -L/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt \
     -lfiniteVolume -llduSolvers -lOpenFOAM -liberty -ldl /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so   -lm -o /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/potentialFoam

because he thinks that the pthread library was not linked in, but again I don't know where this command is read from to edit it.

Any help and comments are appreciated.

[gwierink]

Hi ZKM,

Probably it is best to create your own wmake rules for your cluster compiler. What kind of cluster are you building on actually? I built OpenFOAM on different clusters and needed to do that. I guess you could export every rules, as you describe, but that'll become a mess quite easily I'm afraid. Here you can find a manual I wrote. It's for OpenFOAM-1.6.x on Cray cluster, but the general recipe should be useful.

[ZKM]

So I am not the only one who works on Christmas

I have already read your document but couldn't solve the problem. Here is some information about the cluster:
http://users.encs.concordia.ca/~cirrus/1.html

Yes very true and I must implement all changes in wmake rules once I really understand how this thing works.

How threading is done in OpenFoam? Is it through Pstream? If not I guess I should follow that advice to link pthread by putting -lpthread after -ldl in the "general" rule file (is the file correct?)

ZKM