[Nicolas Périnet]

Hi,

I am writing a code to compute periodic solutions in fluid dynamics using a arclength continuation method that involves OpenFOAM functions as subroutines. At each iteration of a main loop, OpenFOAM is used to integrate the Navier-Stokes equations. The obtained solution is post-processed and modified and this new result is then used for the next integration by OpenFOAM and so on.

How can I parallelize such a code?
I've read some topics and documents about parallel programming with OpenFOAM and apparently, the software uses some functions decomposeParDict and recomposeParDict that are in general run separately, and a library PStream. Can I use decomposeParDict/recomposeParDict inside the code? Do I have to parallelize everything with MPI without PStream because of the structure of my code?

Thanks in advance for your answers.

Nicolas Périnet

[akidess]

Typically, you use decompose at the start of the simulation, and reconstruct at the end of it. I believe parallelization is done using PStream to scatter and gather data on processor boundary patches, and PStream is built on top of MPI functions. You should be able to use PStream if it makes your life easier (I can't tell if it will). If not, you can surely use the underlying MPI functions directly.

[Nicolas Périnet]

Thanks a lot Anton for your advises.

I continue on this thread after a long time of wandering.

I've tried to make interact Openfoam with MPI. The useful objects are defined in the file UPStream.H. for instance:
-number of processes : nProcs,
-index of the process [from 0 (masterNo) to nprocs-1]: myProcNo,
-index of master process: masterNo,
etc...
typing for instance

Foam::UPstream::myProcNo()

I've got a test example based on icoFOAM and, on a single file, that I can post there if somebody finds it useful.

Now, I'm trying to use Openfoam in a loop initialized in a fortran subroutine which is itself part of a main code in C++ involving OpenFOAM functions.
It compiles but when I'm running the code I get the error message

Calling MPI_Init or MPI_Init_thread twice is erroneous.

once I call Openfoam at the first step of my loop. I've dug for a while and found it was just after

#include SetRootCase

because it uses itself UPstream that initializes MPI (setRootCase.H is included in my main program and also in the OpenFOAM function inside the loop). How can I avoid this error?

If I comment the call to setRootCase inside the loop, I also get error messages.

Apparently it is possible to use the function MPI_Initialized to avoid initializing MPI again, but I don't know yet how to use it. I'm afraid there will be a lot of work to do and I'll have to modify a bunch a files to achieve this.

I would be grateful to anybody that could help to get rid of this error.

[Nicolas Périnet]

after having searched for a while, I've found that mpi_initialize seems to be indeed a good solution since the code is now working better, I don't know yet on what extent, but it doesn't stop at the beginning of the loop with the previous error message.

I replaced in UPstream.C the line

    MPI_Init(&argc, &argv);

by

    int is_initialized;
    MPI_Initialized(&is_initialized);
	if(!is_initialized)
	{
        MPI_Init(&argc, &argv);
    }

Now, it checks whether mpi_init has already been performed each time setRootCase is run (and subsequently each time mpi_init was called), avoiding the previous error.

The library containing UPstream.C has to be recompiled.

UPstream.C is in

$WM_PROJECT_DIR/src/Pstream/mpi

the library is recompiled by typing

wmake libso

After this it should work fine.