|
[Sponsors] |
[OpenFOAM.org] Custom solver not running in parallel |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
June 4, 2019, 04:54 |
Custom solver not running in parallel
|
#1 |
Senior Member
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18 |
Hi All,
I am trying to run a custom solver in parallel on OpenFOAM 5. OF is installed on a cluster (root) and I have successfully managed to run one of its tutorial in parallel. However, when I try to run the custom solver in parallel I get the following error: Code:
mpprun info: Starting impi run on 1 node ( 4 rank X 1 th ) for job ID 3202240 --> FOAM FATAL ERROR: Trying to use the dummy Pstream library. This dummy library cannot be used in parallel mode From function static bool Foam::UPstream::init(int &, char **&) in file UPstream.C at line 37. FOAM exiting wmake - options for parallel code using omp.h Code:
ptFLAGS += -Nmpi -Nmkl -xCORE-AVX512 Regards, Syavash |
|
June 4, 2019, 09:09 |
|
#2 |
Senior Member
Joachim Herb
Join Date: Sep 2010
Posts: 650
Rep Power: 22 |
What happens, if you start a standard OpenFOAM solver in parallel, e.g. simpleFoam in the motorbike tutorial?
|
|
June 4, 2019, 09:14 |
|
#3 | |
Senior Member
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18 |
Quote:
As I stated in the original post, it goes well when running a tutorial in parallel. The problem is when I build a modified solver based on a native solver, it can't be run in parallel. |
||
June 5, 2019, 00:25 |
|
#4 |
Member
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15 |
Hello Ehsan
I could not find any information in the link regarding the options you added to your makefiles. That guy in the link is probably using openmp related stuff in his code, therefore he wants to add its flag. I don't think you need to add AVX or MKL to ptFlags unless your modified solver taking advantage of vectorization or uses MKL. By the way imposing vectorization on OF might even slow down your computations according to https://www.ixpug.org/images/docs/IX...g-OpenFOAM.pdf. See pages 27 to 32. Anyway to solve the issue, I'd suggest you checking dependencies of your solver using ldd tool -- ldd `which yourSolverName`. It sounds like binaries from FOAM_USER_APPBIN are somehow linked against the dummy mpi which is located under FOAM_LIB. If that's the case, ldd will tell you, updating LD_LIBRARY_PATH should resolve your problem. // Fatih |
|
June 5, 2019, 03:37 |
|
#5 | |||
Senior Member
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18 |
Dear Fatih,
Thanks for your reply. Quote:
Please see the following link: Non-root installation of OpenFOAM 2.4.x, parallel issue Quote:
Quote:
Kind Regards, Syavash |
||||
June 5, 2019, 03:44 |
|
#6 | |
Senior Member
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18 |
Quote:
I tried to use ldd as you suggested: Code:
ldd 'which myInterFoam' Code:
ldd: ./which myInterFoam: No such file or directory Kind Regards, Syavash |
||
June 5, 2019, 07:10 |
|
#7 |
Member
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15 |
Nope, you need to use backticks and you don't need ./ since which is already in your PATH:
Code:
ldd `which myIcoFoam` |
|
June 5, 2019, 08:29 |
|
#8 | |
Senior Member
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18 |
Quote:
Code:
ldd `which myInterFoam` Code:
linux-vdso.so.1 => (0x00007ffee71a3000) libimmiscibleIncompressibleTwoPhaseMixture.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libimmiscibleIncompressibleTwoPhaseMixture.so (0x00007fd5f631f000) libturbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libturbulenceModels.so (0x00007fd5f5d97000) libincompressibleTurbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libincompressibleTurbulenceModels.so (0x00007fd5f571f000) libfiniteVolume.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfiniteVolume.so (0x00007fd5f2e55000) libfvOptions.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfvOptions.so (0x00007fd5f29f2000) libmeshTools.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libmeshTools.so (0x00007fd5f2238000) libsampling.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsampling.so (0x00007fd5f1a69000) libOpenFOAM.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libOpenFOAM.so (0x00007fd5f0c81000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fd5f0a7d000) libm.so.6 => /lib64/libm.so.6 (0x00007fd5f077b000) libmpicxx.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpicxx.so.12 (0x00007fd5f055b000) libmpifort.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpifort.so.12 (0x00007fd5f01b2000) libmpi.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpi.so.12 (0x00007fd5ef52d000) librt.so.1 => /lib64/librt.so.1 (0x00007fd5ef325000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd5ef109000) libstdc++.so.6 => /.../.../.../.../.../GCCcore/6.4.0/lib64/libstdc++.so.6 (0x00007fd5f659a000) libgcc_s.so.1 => /.../.../.../.../.../GCCcore/6.4.0/lib64/libgcc_s.so.1 (0x00007fd5f6582000) libc.so.6 => /lib64/libc.so.6 (0x00007fd5eed3c000) libincompressibleTransportModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libincompressibleTransportModels.so (0x00007fd5eea61000) libinterfaceProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libinterfaceProperties.so (0x00007fd5ee7b2000) libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000) libtwoPhaseMixture.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtwoPhaseMixture.so (0x00007fd5ee383000) libtwoPhaseProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtwoPhaseProperties.so (0x00007fd5ee109000) libimf.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libimf.so (0x00007fd5edb7b000) libsvml.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libsvml.so (0x00007fd5ec4c8000) libirng.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libirng.so (0x00007fd5ec154000) libintlc.so.5 => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libintlc.so.5 (0x00007fd5ebee7000) libtriSurface.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtriSurface.so (0x00007fd5ebc1a000) libcompressibleTurbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libcompressibleTurbulenceModels.so (0x00007fd5eb585000) libfluidThermophysicalModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfluidThermophysicalModels.so (0x00007fd5ead77000) libsurfMesh.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsurfMesh.so (0x00007fd5ea9ec000) libfileFormats.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfileFormats.so (0x00007fd5ea74a000) liblagrangian.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/liblagrangian.so (0x00007fd5ea519000) libdynamicMesh.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libdynamicMesh.so (0x00007fd5e9c37000) libconversion.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libconversion.so (0x00007fd5e998b000) libz.so.1 => /.../.../.../.../.../zlib/1.2.11-GCCcore-6.4.0/lib/libz.so.1 (0x00007fd5e9973000) /lib64/ld-linux-x86-64.so.2 (0x00007fd5f652e000) libcompressibleTransportModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libcompressibleTransportModels.so (0x00007fd5e976d000) libsolidThermo.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsolidThermo.so (0x00007fd5e9409000) libsolidSpecie.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsolidSpecie.so (0x00007fd5e91c1000) libspecie.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libspecie.so (0x00007fd5e8afc000) libthermophysicalProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libthermophysicalProperties.so (0x00007fd5e8821000) libextrudeModel.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libextrudeModel.so (0x00007fd5e85fd000) libreactionThermophysicalModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libreactionThermophysicalModels.so (0x00007fd5e779c000) Kind Regards, Syavash |
||
June 5, 2019, 10:22 |
|
#9 |
Member
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15 |
Yes something is wrong there. When you look carefully you'll see that your solver is linked against the dummy libPstream - the library that implements MPI comm in OF. See:
Code:
libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000) Code:
export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:LD_LIBRARY_PATH Hope this helps // Fatih |
|
June 5, 2019, 11:02 |
|
#10 | |
Senior Member
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18 |
Quote:
Code:
./platforms/linux64IccDPInt32Opt/lib/mpi/libPstream.so ./platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so Kind Regards, Syavash |
||
June 5, 2019, 13:04 |
|
#11 |
Member
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15 |
So according to those results ${FOAM_LIBBIN}/mpi is correct. If you export dummy you'll have the same error again. Just type the command with export I've written in my previous reply before you call your solver or if you're using a job script add it to that script after you load OF modules. Do not add it to your bashrc.
Hope this helps |
|
June 5, 2019, 13:08 |
|
#12 | |
Senior Member
Ehsan Asgari
Join Date: Apr 2010
Posts: 473
Rep Power: 18 |
Quote:
Only one point. I had to recompile my solver for this correction to work. Hope this help others with the same issue. Kind Regards, Syavash P. S. It is a basic question, but what is LD_LIBRARY_PATH doing? |
||
June 6, 2019, 16:02 |
|
#13 |
Member
Fatih Ertinaz
Join Date: Feb 2011
Location: Istanbul
Posts: 64
Rep Power: 15 |
Ehsan
That's a basic question but it is in fact quite complicated to explain I can recommend two references: -- http://tldp.org/HOWTO/Program-Librar...libraries.html -- "An Introduction to GCC" Book by Brian Gough In short; when you use a shared library while writing your code, you need to tell the compiler that those libraries should be dynamically linked during the compilation. After you execute your code, parts of those libraries used in your code will be invoked during runtime. This mechanism has a few advantages; 1- You don't need to integrate your libraries into the executables, so it keeps binaries in a bare minimum. 2- When those libs are updated, you don't need to compile everything from scratch - for instance if you change a boundary condition code in OF stack, you don't compile solvers from scratch because BCs are shared objects. 3- If a single shared lib is used by multiple executables, OS still keeps a single copy of that library in the memory which means reduced memory consumption. How does this relate to LD_LIBRARY_PATH? It is the environment variable that keeps a list of those libraries checked by Linux OS runtime. By the way, primary reason I'm writing this message is that I've seen a typo in the export command. Correct version should be: Code:
export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:$LD_LIBRARY_PATH // Fatih |
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Radiation Modeling Using Discrete Ordinates Method and Parallel Solver | malicemethods | FLUENT | 3 | May 25, 2018 15:25 |
Poisson equation in custom solver diverges for large boundary values | babakflame | OpenFOAM Running, Solving & CFD | 2 | May 23, 2017 21:11 |
Fluent 14.0 file not running in parallel mode in cluster | tejakalva | FLUENT | 0 | February 4, 2015 08:02 |
Running CFX parallel distributed Under linux system with loadleveler queuing system | ahmadbakri | CFX | 1 | December 21, 2014 05:19 |
Coupled problem running in parallel | liu | OpenFOAM Running, Solving & CFD | 1 | June 24, 2005 06:57 |