[psosnows]

Hello FOAMers!

I have a quite specific question to ask-
did any of you run into problems with running multi-region parallel simulations using MPI?

The solver:
I am working on quite extended version of conjugated heat transfer solver.
I added some additional scalar fields, and for each field I created a new boundary condition.

The case:
Initially: box of fluid, simple rectangular solid inside. All at the same initial temperature.
Process: the walls of the box are being cooled.

I decompose both fluid and solid between 8 processors. Unfortunately I arrive to situation, when the "coupling" boundaries on the fluid and solid part are on different processors. (The same problem appears after decomposing to 2 processors)

What happens:
I run the case. For several hundred iterations there is no problem. Then suddenly I get a message from MPI that MPI_Waitall message failed.
After I restart the simulation from the nearest time step, the error occurs at exact same iteration.

The error occurs during creation of fvScalarMatrix, after the boundaries have been updated, but the matrix does not initialize and is not solved.

What can be the problem in my case?
Is it because the coupled patches are not on the same processor? (I thought that OF handled that well)
Or can it be another thing?

Thank you for any suggestions.
With best regards,
Pawel S