CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Installation

[OpenFOAM.org] Custom solver not running in parallel

Register Blogs Community New Posts Updated Threads Search

Like Tree2Likes
  • 1 Post By fertinaz
  • 1 Post By fertinaz

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   June 4, 2019, 04:54
Default Custom solver not running in parallel
  #1
Senior Member
 
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18
syavash is on a distinguished road
Hi All,

I am trying to run a custom solver in parallel on OpenFOAM 5. OF is installed on a cluster (root) and I have successfully managed to run one of its tutorial in parallel. However, when I try to run the custom solver in parallel I get the following error:

Code:
mpprun info: Starting impi run on 1 node ( 4 rank X 1 th ) for job ID 3202240



--> FOAM FATAL ERROR:
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

    From function static bool Foam::UPstream::init(int &, char **&)
    in file UPstream.C at line 37.

FOAM exiting
I have already added some flags to the Make/option to link the solver to the mpi library, according to the following thread:

wmake - options for parallel code using omp.h

Code:
ptFLAGS += -Nmpi -Nmkl -xCORE-AVX512
Could any body help me to resolve this issue?

Regards,
Syavash
syavash is offline   Reply With Quote

Old   June 4, 2019, 09:09
Default
  #2
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 650
Rep Power: 22
jherb is on a distinguished road
What happens, if you start a standard OpenFOAM solver in parallel, e.g. simpleFoam in the motorbike tutorial?
jherb is offline   Reply With Quote

Old   June 4, 2019, 09:14
Default
  #3
Senior Member
 
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18
syavash is on a distinguished road
Quote:
Originally Posted by jherb View Post
What happens, if you start a standard OpenFOAM solver in parallel, e.g. simpleFoam in the motorbike tutorial?
Hi Joachim,

As I stated in the original post, it goes well when running a tutorial in parallel. The problem is when I build a modified solver based on a native solver, it can't be run in parallel.
syavash is offline   Reply With Quote

Old   June 5, 2019, 00:25
Default
  #4
Member
 
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15
fertinaz is on a distinguished road
Hello Ehsan

I could not find any information in the link regarding the options you added to your makefiles. That guy in the link is probably using openmp related stuff in his code, therefore he wants to add its flag. I don't think you need to add AVX or MKL to ptFlags unless your modified solver taking advantage of vectorization or uses MKL.

By the way imposing vectorization on OF might even slow down your computations according to https://www.ixpug.org/images/docs/IX...g-OpenFOAM.pdf. See pages 27 to 32.

Anyway to solve the issue, I'd suggest you checking dependencies of your solver using ldd tool -- ldd `which yourSolverName`. It sounds like binaries from FOAM_USER_APPBIN are somehow linked against the dummy mpi which is located under FOAM_LIB. If that's the case, ldd will tell you, updating LD_LIBRARY_PATH should resolve your problem.

// Fatih
fertinaz is offline   Reply With Quote

Old   June 5, 2019, 03:37
Default
  #5
Senior Member
 
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18
syavash is on a distinguished road
Dear Fatih,

Thanks for your reply.
Quote:
Originally Posted by fertinaz View Post
Hello Ehsan

I could not find any information in the link regarding the options you added to your makefiles. That guy in the link is probably using openmp related stuff in his code, therefore he wants to add its flag. I don't think you need to add AVX or MKL to ptFlags unless your modified solver taking advantage of vectorization or uses MKL.
Yes, the flags I added to my case were not included in that link. The -Nmpi flags was, however, necessary when I compiled non-root OpenFOAM and wanted to link it to the mpi library of the Cluster. Also, I thought that ptFlags is the proper way of adding the flags to the Make/options. At least, it did not complain when I compiled the code!

Please see the following link:
Non-root installation of OpenFOAM 2.4.x, parallel issue

Quote:
By the way imposing vectorization on OF might even slow down your computations according to https://www.ixpug.org/images/docs/IX...g-OpenFOAM.pdf. See pages 27 to 32.
Thanks for the info. I added MKL and AVX according to the recommendation of the HPC administration, based on the HPC architecture. I have not yet examined the impact of AVX on the speed.

Quote:
Anyway to solve the issue, I'd suggest you checking dependencies of your solver using ldd tool -- ldd `which yourSolverName`. It sounds like binaries from FOAM_USER_APPBIN are somehow linked against the dummy mpi which is located under FOAM_LIB. If that's the case, ldd will tell you, updating LD_LIBRARY_PATH should resolve your problem.
I will certainly try that. I will post the results here, and kindly ask for your opinion on the solution.

Kind Regards,
Syavash
syavash is offline   Reply With Quote

Old   June 5, 2019, 03:44
Default
  #6
Senior Member
 
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18
syavash is on a distinguished road
Quote:
Originally Posted by fertinaz View Post
Anyway to solve the issue, I'd suggest you checking dependencies of your solver using ldd tool -- ldd `which yourSolverName`. It sounds like binaries from FOAM_USER_APPBIN are somehow linked against the dummy mpi which is located under FOAM_LIB. If that's the case, ldd will tell you, updating LD_LIBRARY_PATH should resolve your problem.

// Fatih
Dear Fatih,

I tried to use ldd as you suggested:

Code:
ldd 'which myInterFoam'
However, I get the following error:

Code:
ldd: ./which myInterFoam: No such file or directory
Is this the correct way of using ldd?

Kind Regards,
Syavash
syavash is offline   Reply With Quote

Old   June 5, 2019, 07:10
Default
  #7
Member
 
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15
fertinaz is on a distinguished road
Nope, you need to use backticks and you don't need ./ since which is already in your PATH:
Code:
ldd `which myIcoFoam`
fertinaz is offline   Reply With Quote

Old   June 5, 2019, 08:29
Default
  #8
Senior Member
 
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18
syavash is on a distinguished road
Quote:
Originally Posted by fertinaz View Post
Nope, you need to use backticks and you don't need ./ since which is already in your PATH:
Code:
ldd `which myIcoFoam`
Thanks for the correction. I executed the command,
Code:
ldd `which myInterFoam`
I get the following lines which I cannot figure out!

Code:
        linux-vdso.so.1 =>  (0x00007ffee71a3000)
        libimmiscibleIncompressibleTwoPhaseMixture.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libimmiscibleIncompressibleTwoPhaseMixture.so (0x00007fd5f631f000)
        libturbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libturbulenceModels.so (0x00007fd5f5d97000)
        libincompressibleTurbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libincompressibleTurbulenceModels.so (0x00007fd5f571f000)
        libfiniteVolume.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfiniteVolume.so (0x00007fd5f2e55000)
        libfvOptions.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfvOptions.so (0x00007fd5f29f2000)
        libmeshTools.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libmeshTools.so (0x00007fd5f2238000)
        libsampling.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsampling.so (0x00007fd5f1a69000)
        libOpenFOAM.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libOpenFOAM.so (0x00007fd5f0c81000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fd5f0a7d000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fd5f077b000)
        libmpicxx.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpicxx.so.12 (0x00007fd5f055b000)
        libmpifort.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpifort.so.12 (0x00007fd5f01b2000)
        libmpi.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpi.so.12 (0x00007fd5ef52d000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fd5ef325000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd5ef109000)
        libstdc++.so.6 => /.../.../.../.../.../GCCcore/6.4.0/lib64/libstdc++.so.6 (0x00007fd5f659a000)
        libgcc_s.so.1 => /.../.../.../.../.../GCCcore/6.4.0/lib64/libgcc_s.so.1 (0x00007fd5f6582000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fd5eed3c000)
        libincompressibleTransportModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libincompressibleTransportModels.so (0x00007fd5eea61000)
        libinterfaceProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libinterfaceProperties.so (0x00007fd5ee7b2000)
        libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000)
        libtwoPhaseMixture.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtwoPhaseMixture.so (0x00007fd5ee383000)
        libtwoPhaseProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtwoPhaseProperties.so (0x00007fd5ee109000)
        libimf.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libimf.so (0x00007fd5edb7b000)
        libsvml.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libsvml.so (0x00007fd5ec4c8000)
        libirng.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libirng.so (0x00007fd5ec154000)
        libintlc.so.5 => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libintlc.so.5 (0x00007fd5ebee7000)
        libtriSurface.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtriSurface.so (0x00007fd5ebc1a000)
        libcompressibleTurbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libcompressibleTurbulenceModels.so (0x00007fd5eb585000)
        libfluidThermophysicalModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfluidThermophysicalModels.so (0x00007fd5ead77000)
        libsurfMesh.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsurfMesh.so (0x00007fd5ea9ec000)
        libfileFormats.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfileFormats.so (0x00007fd5ea74a000)
        liblagrangian.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/liblagrangian.so (0x00007fd5ea519000)
        libdynamicMesh.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libdynamicMesh.so (0x00007fd5e9c37000)
        libconversion.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libconversion.so (0x00007fd5e998b000)
        libz.so.1 => /.../.../.../.../.../zlib/1.2.11-GCCcore-6.4.0/lib/libz.so.1 (0x00007fd5e9973000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd5f652e000)
        libcompressibleTransportModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libcompressibleTransportModels.so (0x00007fd5e976d000)
        libsolidThermo.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsolidThermo.so (0x00007fd5e9409000)
        libsolidSpecie.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsolidSpecie.so (0x00007fd5e91c1000)
        libspecie.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libspecie.so (0x00007fd5e8afc000)
        libthermophysicalProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libthermophysicalProperties.so (0x00007fd5e8821000)
        libextrudeModel.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libextrudeModel.so (0x00007fd5e85fd000)
        libreactionThermophysicalModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libreactionThermophysicalModels.so (0x00007fd5e779c000)
Could be something wrong here??

Kind Regards,
Syavash
syavash is offline   Reply With Quote

Old   June 5, 2019, 10:22
Default
  #9
Member
 
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15
fertinaz is on a distinguished road
Yes something is wrong there. When you look carefully you'll see that your solver is linked against the dummy libPstream - the library that implements MPI comm in OF. See:
Code:
        libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000)
There must be another libPstream.so under an mpi directory in the same FOAM_LIB. Try to prepend that path to your LD_LIBRARY_PATH:
Code:
export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:LD_LIBRARY_PATH
${FOAM_LIBBIN}/mpi could be wrong, to make sure go to your OF installation path and run sth. like "find . -name libPstream.so" and check results.

Hope this helps

// Fatih
fertinaz is offline   Reply With Quote

Old   June 5, 2019, 11:02
Default
  #10
Senior Member
 
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18
syavash is on a distinguished road
Quote:
Originally Posted by fertinaz View Post
Yes something is wrong there. When you look carefully you'll see that your solver is linked against the dummy libPstream - the library that implements MPI comm in OF. See:
Code:
        libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000)
There must be another libPstream.so under an mpi directory in the same FOAM_LIB. Try to prepend that path to your LD_LIBRARY_PATH:
Code:
export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:LD_LIBRARY_PATH
${FOAM_LIBBIN}/mpi could be wrong, to make sure go to your OF installation path and run sth. like "find . -name libPstream.so" and check results.

Hope this helps

// Fatih
Thanks for the tip. I used the find command and I got the following results:

Code:
./platforms/linux64IccDPInt32Opt/lib/mpi/libPstream.so
./platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so
Could you tell me according to the results above what I should export as LD_LIBRARY_PATH now?

Kind Regards,
Syavash
syavash is offline   Reply With Quote

Old   June 5, 2019, 13:04
Default
  #11
Member
 
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15
fertinaz is on a distinguished road
So according to those results ${FOAM_LIBBIN}/mpi is correct. If you export dummy you'll have the same error again. Just type the command with export I've written in my previous reply before you call your solver or if you're using a job script add it to that script after you load OF modules. Do not add it to your bashrc.


Hope this helps
syavash likes this.
fertinaz is offline   Reply With Quote

Old   June 5, 2019, 13:08
Default
  #12
Senior Member
 
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18
syavash is on a distinguished road
Quote:
Originally Posted by fertinaz View Post
So according to those results ${FOAM_LIBBIN}/mpi is correct. If you export dummy you'll have the same error again. Just type the command with export I've written in my previous reply before you call your solver or if you're using a job script add it to that script after you load OF modules. Do not add it to your bashrc.


Hope this helps
I really appreciate your help. I could get it to work! Thank you.

Only one point. I had to recompile my solver for this correction to work. Hope this help others with the same issue.

Kind Regards,
Syavash

P. S. It is a basic question, but what is LD_LIBRARY_PATH doing?
syavash is offline   Reply With Quote

Old   June 6, 2019, 16:02
Default
  #13
Member
 
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15
fertinaz is on a distinguished road
Ehsan

That's a basic question but it is in fact quite complicated to explain

I can recommend two references:
-- http://tldp.org/HOWTO/Program-Librar...libraries.html
-- "An Introduction to GCC" Book by Brian Gough

In short; when you use a shared library while writing your code, you need to tell the compiler that those libraries should be dynamically linked during the compilation. After you execute your code, parts of those libraries used in your code will be invoked during runtime.

This mechanism has a few advantages;
1- You don't need to integrate your libraries into the executables, so it keeps binaries in a bare minimum.
2- When those libs are updated, you don't need to compile everything from scratch - for instance if you change a boundary condition code in OF stack, you don't compile solvers from scratch because BCs are shared objects.
3- If a single shared lib is used by multiple executables, OS still keeps a single copy of that library in the memory which means reduced memory consumption.

How does this relate to LD_LIBRARY_PATH? It is the environment variable that keeps a list of those libraries checked by Linux OS runtime.

By the way, primary reason I'm writing this message is that I've seen a typo in the export command. Correct version should be:
Code:
export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:$LD_LIBRARY_PATH
Hope this is useful and I'd highly suggest reading the references above for further details.

// Fatih
syavash likes this.
fertinaz is offline   Reply With Quote

Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Radiation Modeling Using Discrete Ordinates Method and Parallel Solver malicemethods FLUENT 3 May 25, 2018 15:25
Poisson equation in custom solver diverges for large boundary values babakflame OpenFOAM Running, Solving & CFD 2 May 23, 2017 21:11
Fluent 14.0 file not running in parallel mode in cluster tejakalva FLUENT 0 February 4, 2015 08:02
Running CFX parallel distributed Under linux system with loadleveler queuing system ahmadbakri CFX 1 December 21, 2014 05:19
Coupled problem running in parallel liu OpenFOAM Running, Solving & CFD 1 June 24, 2005 06:57


All times are GMT -4. The time now is 21:27.