[syavash]

Dear All,

I have faced some issues in installing OpenFOAM 2.3.1 on a HPC cluster with CentOS 7, non-root access, and Intel Compiler.

In the first place, I did the following:

export FOAM_INST_DIR=$HOME/ch_openfoa/Tetralith/OpenFOAM
mkdir -p $FOAM_INST_DIR
cd $FOAM_INST_DIR

I downloaded the packages of OpenFOAM and ThirdParty and then extracted them.
Then I did the following modifications:

cp -r $FOAM_INST_DIR/OpenFOAM-2.3.1/wmake/rules/linux64Icc $FOAM_INST_DIR/OpenFOAM-2.3.1/wmake/rules/linux64IccHani

sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xCORE-AVX512"/g $FOAM_INST_DIR/OpenFOAM-2.3.1/wmake/rules/linux64IccHani/c++

I added the following alias to the bashrc in my home folder:

alias OF231T='export FOAM_INST_DIR=/home/x_vache/ch_openfoam/Tetralith/OpenFOAM; module load buildenv-intel/2018a-eb; export MPI_ROOT=$I_MPI_ROOT; . /home/x_vache/ch_openfoam/Tetralith/OpenFOAM/OpenFOAM-2.3.1/etc/bashrc WM_COMPILER=IccHani WM_MPLIB=INTELMPI'

Then I opened a new terminal and entered the above alias: OF231T.
I checked the outputs of the following commands:

mpirun --version
Intel(R) MPI Library for Linux* OS, Version 2018 Update 1 Build 20171011 (id: 17941)
Copyright (C) 2003-2017, Intel Corporation. All rights reserved.


which mpirun
/software/sse/easybuild/prefix/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/bin64/mpirun


mpicc --version
icc (ICC) 18.0.1 20171018
Copyright (C) 1985-2017 Intel Corporation.  All rights reserved.


which mpicc
/software/sse/easybuild/prefix/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/bin64/mpicc

At the end, I run the ./Allwmake but I get several errors which I suspect are related to MPI...

I have attached the log file to the following link to further investigate the problem.

https://www.dropbox.com/s/6lshhxyj0tybv6a/log.make?dl=1

I would appreciate if you help me to do proper installation.

Regards,
Syavash

[wyldckat]

Quick answer: The very first error message is this:

icpc ERROR: Unsupported compiler mpiicpc. Try -Nhelp.

which means that ICC is not use configured mpcc...


My guess is that in whichever way you load in the ICC environment, you also need to load in the Intel-MPI environment as well, so that mpiicpc is ready to be used.... no wait... what? The compiler mpiicpc is indeed found, it's just that it's not compatible with the compiler options you've chosen?!


OK, new guess: "-Nmpi" is not supported, hence the error.


Side note: You better edit the file "$FOAM_INST_DIR/OpenFOAM-2.3.1/wmake/rules/linux64IccHani/c++" manually and remove the duplicate entries, along with removing "-Nmpi".

[syavash]

Originally Posted by wyldckat

Quick answer: The very first error message is this:

icpc ERROR: Unsupported compiler mpiicpc. Try -Nhelp.

which means that ICC is not use configured mpcc...


My guess is that in whichever way you load in the ICC environment, you also need to load in the Intel-MPI environment as well, so that mpiicpc is ready to be used.... no wait... what? The compiler mpiicpc is indeed found, it's just that it's not compatible with the compiler options you've chosen?!


OK, new guess: "-Nmpi" is not supported, hence the error.


Side note: You better edit the file "$FOAM_INST_DIR/OpenFOAM-2.3.1/wmake/rules/linux64IccHani/c++" manually and remove the duplicate entries, along with removing "-Nmpi".

Dear Bruno,


Thank you very much for your reply. Actually, I tried to do a fresh install using the general guide provided here (Neglect first three steps):


https://openfoamwiki.net/index.php/I...HEL#CentOS_7.0


with replacing the command in step 8 with the following:

module load buildenv-intel/2018a-eb || export PATH=$PATH:/software/sse/easybuild/prefix/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/intel64/bin

and defining FOAM_INST_DIR as follows:

FOAM_INST_DIR=$HOME/ch_openfoam/Tetralith/OpenFOAM

Then I did some minor modifications as follows:

cp -r $FOAM_INST_DIR/OpenFOAM-2.3.1/wmake/rules/linux64Icc $FOAM_INST_DIR/OpenFOAM-2.3.1/wmake/rules/linux64IccHani


sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xCORE-AVX512"/g $FOAM_INST_DIR/OpenFOAM-2.2.x/wmake/rules/linux64IccHani/c++

Then I did

source $FOAM_INST_DIR/OpenFOAM-2.3.1/etc/bashrc WM_NCOMPPROCS=4 WM_COMPILER=IccHani WM_MPLIB=INTELMPI


echo "alias OF231T='export FOAM_INST_DIR=/home/x_vache/ch_openfoam/Tetralith/OpenFOAM; module load buildenv-intel/2018a-eb; export MPI_ROOT=$I_MPI_ROOT; source $HOME/ch_openfoam/Tetralith/OpenFOAM/OpenFOAM-2.3.1/etc/bashrc $FOAM_SETTINGS'" >> $HOME/.bashrc

Also, I modified the mplibINTELMPI according to the following posts:


Compile OF2.1 with Intel Icc and Intel MPI



Compile OF2.1 with Intel Icc and Intel MPI


And modified etc/config/settings.sh according to the following post:


Failed to compile OpenFOAM 3.0.1 with icc


The settings.sh now looks like the following:

INTELMPI)
    # no trailing slash
    [ "${MPI_ROOT%/}" = "${MPI_ROOT}" ] || MPI_ROOT="${MPI_ROOT%/}"

    export FOAM_MPI="${MPI_ROOT##*/}"
    export MPI_ARCH_PATH=$MPI_ROOT

    if [ ! -d "$MPI_ROOT" -o -z "$MPI_ARCH_PATH" ]
    then
        echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
        echo "    MPI_ROOT not a valid mpt installation directory or ending in a '/'." 1>&2
        echo "    Please set MPI_ROOT to the mpt installation directory." 1>&2
        echo "    MPI_ROOT currently set to '$MPI_ROOT'" 1>&2
    fi

    if [ "$FOAM_VERBOSE" -a "$PS1" ]
    then
        echo "Using INTEL MPI:" 1>&2
        echo "    MPI_ROOT : $MPI_ROOT" 1>&2
        echo "    FOAM_MPI : $FOAM_MPI" 1>&2
    fi

    _foamAddPath    $MPI_ARCH_PATH/bin64
    _foamAddLib     $MPI_ARCH_PATH/lib64
    ;;
*)
    export FOAM_MPI=mpi  #export FOAM_MPI=dummy
    ;;

The I opened a new terminal and entered the alias OF231T and went through, but in step 11 when I execute ./Allwmake I get some errors. I have attached the new log file below. I had a hard time but still the errors are there!



https://www.dropbox.com/s/k2coamml4s....make_new?dl=1



Regards,
Syavash

[wyldckat]

Greetings Syavash,

From what I can see, the changes you've made to "settings.sh" were not necessary... but if it's partially working now, I guess you may have changed something else too.

Based on the log file you provided, my guess is that the problem is related to the "libPstream.so" that was built to use Intel-MPI is being picked up when it should not. I suspect that the following trick might work, so with the OpenFOAM 2.3.1 environment activated, run the following commands:

cd $WM_PROJECT_DIR
mv $FOAM_LIBBIN/$FOAM_MPI  $FOAM_LIBBIN/${FOAM_MPI}_tmp

applications/Allwmake > log.make.applications 2>&1

mv $FOAM_LIBBIN/${FOAM_MPI}_tmp $FOAM_LIBBIN/$FOAM_MPI

The trick used here is that the folder where the "libPstream.so" file for Intel-MPI was compiled to, is renamed to another name, so that it is not found when building the applications. Then we only build the applications for the respective folder. Then when it's done, we restore the folder we had renamed.

Hopefully this will solve the problem.

Best regards,
Bruno

[syavash]

Originally Posted by wyldckat

Greetings Syavash,

From what I can see, the changes you've made to "settings.sh" were not necessary... but if it's partially working now, I guess you may have changed something else too.

Based on the log file you provided, my guess is that the problem is related to the "libPstream.so" that was built to use Intel-MPI is being picked up when it should not. I suspect that the following trick might work, so with the OpenFOAM 2.3.1 environment activated, run the following commands:

cd $WM_PROJECT_DIR
mv $FOAM_LIBBIN/$FOAM_MPI  $FOAM_LIBBIN/${FOAM_MPI}_tmp

applications/Allwmake > log.make.applications 2>&1

mv $FOAM_LIBBIN/${FOAM_MPI}_tmp $FOAM_LIBBIN/$FOAM_MPI

The trick used here is that the folder where the "libPstream.so" file for Intel-MPI was compiled to, is renamed to another name, so that it is not found when building the applications. Then we only build the applications for the respective folder. Then when it's done, we restore the folder we had renamed.

Hopefully this will solve the problem.

Best regards,
Bruno

Dear Bruno,


Thanks for your reply. I did as you suggested and attached the log file at the following link:


https://www.dropbox.com/s/cf7x1buwjk...lications?dl=1


I also removed changes in settings.sh but made no difference!

I really appreciate your help.


Regards,
Syavash

[wyldckat]

Quick answer: How did the file "platforms/linux64IccHaniDPOpt/lib/libPstream.so" come into existence? It should not exist... please delete that file, which can be done with the following commands:

cd $WM_PROJECT_DIR
rm platforms/linux64IccHaniDPOpt/lib/libPstream.so

Then try again the commands from my previous post.

[syavash]

Originally Posted by wyldckat

Quick answer: How did the file "platforms/linux64IccHaniDPOpt/lib/libPstream.so" come into existence? It should not exist... please delete that file, which can be done with the following commands:

cd $WM_PROJECT_DIR
rm platforms/linux64IccHaniDPOpt/lib/libPstream.so

Then try again the commands from my previous post.

Dear Bruno,


Thank you! I removed that file and OpenFOAM was installed without problem. However, after I executed the following command to move back MPI_FOAM I received an error when I entered "pimpleFoam" in the terminal.

mv $FOAM_LIBBIN/${FOAM_MPI}_tmp $FOAM_LIBBIN/$FOAM_MPI

and the error I received when I executed pimpleFoam in the terminal:

pimpleFoam: symbol lookup error: /home/x_vache/ch_openfoam/Tetralith/OpenFOAM/OpenFOAM-2.3.1/platforms/linux64IccHaniDPOpt/lib/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/libPstream.so: undefined symbol: ompi_mpi_op_sum

I tried to move the FOAM_MPI to FOAM_MPI_tmp. This way pimpleFoam was executed in serial mode but when I tried to submit a job in parallel it complained about MPI with the following error:

mpprun info: Starting impi run on 3 node ( 96 rank X 1 th ) for job ID 101262


--> FOAM FATAL ERROR:
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

    From function UPstream::init(int& argc, char**& argv)
    in file UPstream.C

--> FOAM FATAL ERROR:
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

    From function UPstream::init(int& argc, char**& argv)
    in file UPstream.C at line 37.

FOAM exiting



--> FOAM FATAL ERROR:
Trying to use the dummy Pstream library.

I guess that Pstream is again causing the issue. Would you please check the issue?


P.S. I had removed changes in settings.sh prior to installation.

[wyldckat]

Quick answer:

1. Undo the rename again, by running:
   

   mv $FOAM_LIBBIN/${FOAM_MPI}_tmp $FOAM_LIBBIN/$FOAM_MPI

2. The "libmpi.so" file seems to be missing, but it's strange that it's missing, because "libPstream.so" should have complained that it's missing...
3. Try running:
   

   cd $FOAM_SRC/Pstream
   wclean all
   ./Allwmake > log.make 2>&1

   The attach the file "log.make". This will tell us if "libmpi.so" is being linked to or not.

[syavash]

Originally Posted by wyldckat

Quick answer:

1. Undo the rename again, by running:
   

   mv $FOAM_LIBBIN/${FOAM_MPI}_tmp $FOAM_LIBBIN/$FOAM_MPI

2. The "libmpi.so" file seems to be missing, but it's strange that it's missing, because "libPstream.so" should have complained that it's missing...
3. Try running:
   

   cd $FOAM_SRC/Pstream
   wclean all
   ./Allwmake > log.make 2>&1

   The attach the file "log.make". This will tell us if "libmpi.so" is being linked to or not.

Dear Bruno,


Thank you for the reply. As I was thinking I had messed with the settings and addresses, I did again a fresh install and this time did not change any things. Before installing OpenFOAM I also moved back ${FOAM_MPI}_tmp to $FOAM_MPI. I followed the instructions given in WIKI again and OpenFOAM was installed successfully without a problem and did not complain about libPstream either! I tried a parallel run and it was OK. Though paraFoam was not installed (which I guess it could be due to qmake!) it should be all rights for now.


Thank you for your time and patience. I have posted the alias in the .bashrc file as somebody may find it helpful:

alias OF231T='export FOAM_INST_DIR=/home/.../OpenFOAM; module load buildenv-intel/2018a-eb; export MPI_ROOT=/.../software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28; source /home/.../OpenFOAM/OpenFOAM-2.3.1/etc/bashrc WM_NCOMPPROCS=4 WM_COMPILER=Icc WM_MPLIB=INTELMPI'

Regards,
Syavash