[matteoL]

Hello,
when running in parallel, for some reasons I have some actions that have to be done only by the master core and I would like the other partitions to wait until the master has finished.
(I call those action doing:
If (Pstream::master()){ ...} )


I think the solution in a general c++ code using MPI would be to add a line like:
MPI_BARRIER(MPI_COMM_WORLD)

Is there an already implemented function in OF available?
OR,
how could i get acces to the MPI class/function/settings?

Thank you very much,
best regards,
Matteo

[niklas]

does the master node information need to be distibuted to the other processors?
(it usually does)

in which case you can just

if (master)
   loop i over number of procs except master
       send info to proc i
else
    receive info from master
end if

which would have the same effect

[matteoL]

Hello Nicklas, thanks a lot for your quick answer.
Not sure I have understood what you meant.

1)Easy case
Let's say that the master is doing some calculations and then writes down those info on a file.

AFTER that the file have been written, I want all procs to read the file and go on ...

How do i do that?
If I simply do:
If (Pstream::master()){
...cacl...
write file
}

read file

then a non-master node will skip the first and will try to read the file BEFORE the file has even been written...

How would you do?

2)Same thing as before, but the data are not written to a file but, after been computed are shared by the master to all other node.
-Once again,I do i tell the other proc to wait for the master to do the calculations?
-how do i send data from the master to all other proc? (using "reduce"?)

thanks again,
Matteo

[niklas]

First off, I think that communicating to other processors via files is a really really bad idea
and should be abandoned. If you want to have that info on file, sure write it to a file, but dont make the other processors read it in via the file, it will drastically reduce your performance...just think of all the processors asking for i/o access to the disk.
I think you should view your processors as the file instead, what you want to do is stream the info to the other processors directly, it is just as easy as streaming it to a file.

below is a very simple example of a piece of code sending a vector to the main processor.
it should be easy enough to figure out how to do the reverse, i.e send all the info to
the other processors from the master

    // get the number of processors
    label n=Pstream::nProcs();

    vector localV = vector::zero;

    if (Pstream::myProcNo() == 0)
    {
        Random rnd(0);
        localV = rnd.vector01();
        List<vector> allV(n, vector::zero);

        allV[0] = localV;
        for(label i=1; i<n; i++)
	{
            // create the input stream from processor i
 	    IPstream vStream(Pstream::blocking, i);
            vStream >> allV[i];
	}
        // print the list of all vectors on the main proc
        Info << allV << endl;
    }
    else
    {
        Random rnd(0);
        localV = rnd.vector01();
        // print the vector on the processor (so we can compare it later when we print it on the main proc)
        Sout << "[" << Pstream::myProcNo() << "] V = " << localV << endl;

        // create the stream to send to the main proc
        OPstream vectorStream
	(
	    Pstream::blocking, 0
        );
        vectorStream << localV;
    }

since the streaming is 'blocking' the processor will wait until it has received the information

i didnt really answer any of your questions cause I hope that the above code will help you
solve the problem yourself, otherwise post them again

[matteoL]

I see very well what you mean and I think I may use some of the information you have sent me.
Although in my case I think i am forced to use file for I/O.

Here is the explanation:
Mainly, I am coupling OF with an external structural solver written in Fortran.
Thus I use as OF as main code and the structural solver is written as Fortran functions that are called inside OF thanks to the " extern "C" options".

Since I don't want(can not, at the moment) pass vectors from OF to the Fortran code, I decided to pass the needed informations by file.

Do you see my problem?how would you tackle it?
Thanks again,
matteo

[niklas]

ah, I see.

If it is possible I would have the master processor read all the info from the file and then send it to each processor. (it will also solve your sync problem)

This should be alooot faster than having each processor reading the file #proc number of times.

N

[cliffoi]

I know this thread is no longer active but in case anyone is interested in interfacing OpenFOAM and Fortran I felt like adding my own two cents worth: I agree with Niklas, there should be no need to use file IO. I've passed fields of data in OpenFOAM to and from Fortran very happily in the past. Fortran 90 and above is easy. If you're still using Fortran 77, I'd first suggest you seriously think about upgrading, and then write a F90 wrapper around your F77 interface functions. If you're looking at passing fields of any data type to an external Fortran program you simply need to pass the memory address of the start of the field and then tell the Fortran function what data type it should expect.

SUBROUTINE fortranSub(vf, size)
    USE ISO_C_BINDING, ONLY : C_DOUBLE, C_INT
    IMPLICIT NONE

    REAL(C_DOUBLE), INTENT(INOUT)   :: vf(size)
    INTEGER(C_INT), INTENT(IN)      :: size

    vf(:) = 100D0

END SUBROUTINE

and in your C++ header

extern"C"
{
    void __fortransub
    (
        double* vf,
        const int& size
    );
}

You can then call it from OpenFOAM like follows by explicitly passing the array address and size.

scalarField sf(mesh.nCells());

__fortransub(&sf[0], sf.size());

Info << sf << endl;

[jherb]

Just if somebody arrives here from google in this very old thread (I didn't find a newer one): You could use something like this to synchronize your processes:

    if (Pstream::master())
    {
        /* what ever */
    }
    int dummy;
    Pstream::scatter(dummy, Pstream::blocking);

See also http://www.cfd-online.com/Forums/ope...tml#post440958

[zhengzh5]

Originally Posted by jherb

Just if somebody arrives here from google in this very old thread (I didn't find a newer one): You could use something like this to synchronize your processes:

    if (Pstream::master())
    {
        /* what ever */
    }
    int dummy;
    Pstream::scatter(dummy, Pstream::blocking);

See also http://www.cfd-online.com/Forums/ope...tml#post440958

Hey Joachim,

what version of OpenFOAM are you using (i assume 2.2)? I tried it on OpenFOAM 2.0.x and there's no match for the function. Do you know if there's anyways to sync the processors before a reduce call in 2.0.x?

thanks for your help in advance!

Jace

[jherb]

Hi Jace,

actually I am using OpenFOAM 2.2.x but it should not matter. The source files for Pstream::scatter are also in the repository of OpenFOAM 2.0.x:
https://github.com/OpenFOAM/OpenFOAM...eams/Pstream.H
https://github.com/OpenFOAM/OpenFOAM...atherScatter.C

So what are the error message(s) during compiling/running? Perhaps you have to add the correct header file (probably Pstream.H) and perhaps you have to specify additional libraries for linking.

Joachim

Originally Posted by zhengzh5

Hey Joachim,

what version of OpenFOAM are you using (i assume 2.2)? I tried it on OpenFOAM 2.0.x and there's no match for the function. Do you know if there's anyways to sync the processors before a reduce call in 2.0.x?

thanks for your help in advance!

Jace

[zhengzh5]

Originally Posted by jherb

Hi Jace,

actually I am using OpenFOAM 2.2.x but it should not matter. The source files for Pstream::scatter are also in the repository of OpenFOAM 2.0.x:
https://github.com/OpenFOAM/OpenFOAM...eams/Pstream.H
https://github.com/OpenFOAM/OpenFOAM...atherScatter.C

So what are the error message(s) during compiling/running? Perhaps you have to add the correct header file (probably Pstream.H) and perhaps you have to specify additional libraries for linking.

Joachim

Hi, the following are the error message:

In file included from spray/sprayInject.C:124:0:
spray/findInjectorCell.H: In member function ‘void Foam::spray::inject()’:
spray/findInjectorCell.H:23:45: error: no matching function for call to ‘Foam::Pstream::scatter(int&, Foam::UPstream::commsTypes)’
spray/findInjectorCell.H:23:45: note: candidates are:
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:109:25: note: template<class T> static void Foam::Pstream::scatter(const Foam::List<Foam::UPstream::commsStruct>&, T&)
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:113:25: note: template<class T> static void Foam::Pstream::scatter(T&)
make: *** [Make/linuxGccDPOpt/sprayInject.o] Error 1

I am able to access other functions belonging to Pstream, such as Pstream::nProcs, Pstream::master(). To me, it seems like the foam 2.0.x version can't take in the blocking argument. what do you think?

thanks for quick response!

Jace

[jherb]

I guess you are right: This function declaration is new:
https://github.com/OpenFOAM/OpenFOAM...source=cc#L124

The source code is probably one of these:
https://github.com/OpenFOAM/OpenFOAM...Scatter.C#L193
https://github.com/OpenFOAM/OpenFOAM...Scatter.C#L218

Perhaps you can backport it?

Originally Posted by zhengzh5

Hi, the following are the error message:

In file included from spray/sprayInject.C:124:0:
spray/findInjectorCell.H: In member function ‘void Foam::spray::inject()’:
spray/findInjectorCell.H:23:45: error: no matching function for call to ‘Foam::Pstream::scatter(int&, Foam::UPstream::commsTypes)’
spray/findInjectorCell.H:23:45: note: candidates are:
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:109:25: note: template<class T> static void Foam::Pstream::scatter(const Foam::List<Foam::UPstream::commsStruct>&, T&)
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:113:25: note: template<class T> static void Foam::Pstream::scatter(T&)
make: *** [Make/linuxGccDPOpt/sprayInject.o] Error 1

I am able to access other functions belonging to Pstream, such as Pstream::nProcs, Pstream::master(). To me, it seems like the foam 2.0.x version can't take in the blocking argument. what do you think?

thanks for quick response!

Jace

[klausb]

I want to gather the distributed upperAddr(), lowerAddr(), psi.size() and number of matrix coefficients on the master thread to do some checks on the global matrix - something like:

if (Pstream::master())
{
  // gather the parts of upperAddr(), lowerAddr(), psi.size(), n_coefficients
   global_upperAddr() = ?; // OF or c++ list or vector

   global_upperAddr() = ?; // OF or c++ list or vector

   global_n = gSum(psi.size()) ?; // matrix size

   global_n_coefficients = gSum(matrix().lower().size() + matrix().upper().size() + matrix().diag().size()) ?;

}

How can I gather the distributed data?