[xiao]

The same message but posted under a more illustrative name.

By Jaswinder Pal Singh on Monday, November 10, 2008 - 02:17 am: Edit Post

Dear Forum

Good Morning

I have some doubts related to parallelization of an OpenFOAM application or solver.

A modiication done to interFoam works fine on single processor but crashes, when run in parallel. A simple test such as :

mesh.ncells() or field.size() give different results when run on single and in parallel.

Also the changes to the solver do some explicit operation. For example :

Look for a cell with a particular value of the volume fraction field gamma and take their neighbours and do some calculation. How to make such operations parallel aware. As far as my understanding goes one has to explicitly take care of it. In code this might look:

forAll(gamma, cellI)
{
if (0.01 < gamma < 0.99)
{
perform calcs ....

}


}



I request the users with experience of this aspect to please share the information. It would be great if we could compile a list of guidelines one may follow to make sure that their solver is completely compatible to parallel runs.

Hope that I will find some contributors

Kind Regards
Jaswi

By Jaswinder Pal Singh on Monday, November 10, 2008 - 07:23 am: Edit Post

Some more search shows that a reduce operation has to be done whereever something processor dependent is being done. I tried the following:

label cellCount = mesh.nCells();
reduce(cellCount, sumOp<label>());

Info << "Mesh cell count : " << cellCount << endl;

returns the same value for single and parallel, but when I do the same for faces:

label faceCount = mesh.nFaces();
reduce(faceCount, sumOp<label>());

Info << "Mesh face count : " << faceCount << endl;

the output is different as the faceCount now includes the number of faces for the processor patches as well so be aware of that fact. I do not how yet how to get the correct number of faces using reduction.

It seems that the reduce operation can be done for various operations such as

reduce(foundLabel, orOp<bool>())

Still digging....

Any body having a clear understanding please contribute

Kind Regards
Jaswi

By Sandeep Menon on Monday, November 10, 2008 - 07:35 am: Edit Post

I have a suggestion - although I'm not too sure about the feasibility of implementation. MPI inherently numbers processes between 0 and (n-1); where n is the number of subdomains. I'm guessing that every processor patch can identify which MPI process / sub-domain it talks to. So, for every processor patch, identify if the neighbouring sub-domain is numbered higher than the current one. If yes, then add the number of faces for the processor patch to the current sub-domain's face-count, and discard if otherwise. Finally, you can do a conventional reduce with sumOp, to sum up the face count for all sub-domains.

By Henrik Rusche on Sunday, November 30, 2008 - 06:11 am: Edit Post

OpenFOAM uses domain decomposition. Therefore each processor only knows about the cells and faces residing on it.

Your problem is difficult because you want to identify cells and then do calculations with their neighbours. This stretches the "domain decomposition" paradigmn because the neighbour might reside on a different processor.

My recipie for you:

1) Create an indicator field locally
2) interpolate the result to the faces (This should update the processor-processor boundaries)
3) Identify the neighbour cells: They have faces with non-zero values in the interpolated indicator field
4) Do the calc involving neighbours

There is probably a much more elegant solution, but I would have to dig a lot.

Henrik

By Ivan Flaminio Cozza on Sunday, November 30, 2008 - 07:58 am: Edit Post

@Jaswinder
I'm intrested to in this kind of problem as I will create a solver that needs to know for each cell the neighbours, and I want to make it work in parallel.
My question is: does exist any structure in Pstream that store the local cell location?
So, something that tells us if a cell in the x domain is on the boundary with the y domain.
Thanks!

By Nat Trask on Tuesday, December 02, 2008 - 02:29 pm: Edit Post

processorPolyPatch contains a list of faces on the patch between different processors and functions for getting the processor number of each patch.

I believe what you want to do is loop over the faces of your cell, check to see if the faces are on a processorPolyPatch, and then use the functions in processorPolyPatch to return the two processor IDs.

-Nat

By Heng Xiao on Saturday, December 13, 2008 - 10:09 am: Edit Post

Anyone has a follow-up on this issue? Or are the suggestions (recipie) of Nat and Henrik are the optimal ones already?

Suggestions are appreciated!

Heng

[xiao]

I changed to a more illustrative name hoping someone with this experience would say it and share their experience, but apparently it was not very successful.

Or maybe this is indeed "stretching" OpenFoam design principles, as Henrik Rusche pointed out.