[mrRichField]

Dear All,

in the course of a university project we are trying to adapt the basic simpleFoam solver to our needs by adding some lines of code. Everything works fine performing single core, but as soon as we go parallel, we encountered problems.

Right before the simple algorithm we perform a loop over all cells, where the elements of vector A are initialized with the average of vector B, using something like...

forAll(mesh.cells(), i)
{
    A[i] = B.average().value();
    ...
}

In parallel we receive an MPI_ERROR, but we soon found, that we can proceed using

temp=B.average().value();
forAll(mesh.cells(), i)
{
    A[i] = tmp;
    ...
}

but unfortunately we are not aware of the reason for this behaviour!??

Furthermore, we thought, that as with 'conventional' MPI parallelization we know from pure C programming, we used mpi_reduce statements to sum up and count, and calculate an average of a quantity over all workers. Please, could someone point us in the right direction in the context of C++ & OpenFOAM programming, since we are "just beginners"

We are already struggling with getting some debug information from each worker, i.e. how can we put something like

Info << "worker no. " << i << " | value = " << v << endl;

in the code, in order to get a feedback from each worker?

We've checked the manuals and tutorials available throughout the internet, but ... Is there any good programming tutorial focusing on parallel?

We would be grateful for your feedback,
Kind Regards,

George

[floquation]

A very incomplete answer, but this is what I know:

The following statement I use for parallel debugging (that way, each processor shows its own message with the processor ID appended):

Sout << "[" << Pstream::myProcNo() << "] " << "Debug message" << nl;

[mrRichField]

Thanks a lot floquation!

In the meantime I also found that

Sout

is the output stream which provides an answer from each worker. Furthermore I found that the

.average()

method uses

gaverage

(something I think of as 'global' average), which ensures that the respective

mpi_reduce

formalism is performed. Ok, thus I can use the average method on a field in a straight forward way for single proc or parallel applications, respectively.

However, I still don't understand, what is the difference between using the average method inside the 'forall' scope or outside? Well, my code does it's job right now, but I'm really interested in what is going on behind the scenes!? Please, if anyone can put a light on this, I would be grateful!

Cheers,
George

BTW: Could anyone provide me a link on a more comprehensive programming guideline (sorry, even though I do programming for a while, I'm an engineer and not an IT specialist) that goes a bit further than the official OF programming guide, and which provides a clarifying picture of the underlying structure and hierarchy of OF?

[jherb]

Why do you call B.average() inside the loop? It looks like it is not depending on i (or the cell), so you are calculating the same value again and again. Can't you put it into a temporary variable defined before the loop? (Ok, the compiler might be optimizing it this way, but who knows?)

Originally Posted by mrRichField

forAll(mesh.cells(), i)
{
    A[i] = B.average().value();
    ...
}

[mrRichField]

Hi jherb,

indeed, you#re right, and that's what I already did (cf. my first post). But, even though it will be optimized by either the compiler (if...) or by developer (if she is aware of it...), it is not clear for me, why we receive an mpi_error in the first case (average in loop)...

So finally, it is clear, that one should keep the calculation of a non-changing value outside of the loop with respect to performance (regardless of compiler optimization), but with regards to efficiently do parallel programming in openFoam, we'd like to know more about the source of the problem.

Kind Regards,
George

[jherb]

The gaverage function (see also e.g. the gSum function) automatically reduce the results from all CPUs, i.e. collect the results from all CPUs and does the averaging. Then the result is sent to all CPUs.

Just speculating, but I think the call to such a reducing function has to be called on all CPUs at the same point in the code. But the loop of the mesh cells is for different cells on the different CPUs. So if there are different number of cells on different CPUs this sounds like trouble