[jploz]

Hi Foamers,

I've created a fvDynamicMesh class for moving the mesh based on flow forces in 1.5-dev. When running the case on a single processor everything works as expected. However, if I run the case in parallel it does not work.
For testing purposes the points are set to the (fixed) original coordinates - but i cannot move the points in parallel.

Does not work means:

if I use tetFaceDecomposition the simulation runs (i.e. it does not crash) but never finishes the fvMesh::movePoints step-the processors run with 100% but nothing happens for a really long time (strange)

if I use fv based motion the first mesh update is done but the simulation crashes when the fvMesh::movePoints step is reached for the second time.

Here is the traceback:

Getting new points
PCG:  Solving for cellMotionUx, Initial residual = 0, Final residual = 0, No Iterations 0
PCG:  Solving for cellMotionUy, Initial residual = 0, Final residual = 0, No Iterations 0
PCG:  Solving for cellMotionUz, Initial residual = 0.00115045894, Final residual = 9.72441498e-06, No Iterations 26
Getting new points (fix)
Calling for move points
[host:08332] *** Process received signal ***
[host:08333] *** Process received signal ***
[host:08334] *** Process received signal ***
[host:08334] Signal: Segmentation fault (11)
[host:08334] Signal code:  (-6)
[host:08334] Failing at address: 0x1f80000208e
[host:08333] Signal: Segmentation fault (11)
[host:08333] Signal code:  (-6)
[host:08333] Failing at address: 0x1f80000208d
[host:08332] Signal: Segmentation fault (11)
[host:08332] Signal code:  (-6)
[host:08332] Failing at address: 0x1f80000208c
[host:08334] [ 0] /lib64/libc.so.6 [0x7f6d6a0c16e0]
[host:08334] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f6d6a0c1645]
[host:08334] [ 2] /lib64/libc.so.6 [0x7f6d6a0c16e0]
[host:08334] [ 3] /lib64/libc.so.6 [0x7f6d6a105bcb]
[host:08334] [ 4] /lib64/libc.so.6(__libc_malloc+0xa1) [0x7f6d6a107891]
[host:08334] [ 5] /usr/lib64/libstdc++.so.6(_Znwm+0x1d) [0x7f6d6a91cf4d]
[host:08334] [ 6] /usr/lib64/libstdc++.so.6(_Znam+0x9) [0x7f6d6a91d089]
[host:08334] [ 7] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam4ListIcE7setSizeEi+0x33) [0x7f6d6af27593]
[host:08332] [ 0] /lib64/libc.so.6 [0x7f2bffc546e0]
[host:08332] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f2bffc54645]
[host:08332] [ 2] /lib64/libc.so.6 [0x7f2bffc546e0]
[host:08332] [ 3] /lib64/libc.so.6 [0x7f2bffc98a9a]
[host:08332] [ 4] /lib64/libc.so.6(__libc_malloc+0xa1) [0x7f2bffc9a891]
[host:08332] [ 5] /usr/lib64/libstdc++.so.6(_Znwm+0x1d) [0x7f2c004aff4d]
[host:08332] [ 6] /usr/lib64/libstdc++.so.6(_Znam+0x9) [0x7f2c004b0089]
[host:08332] [ 7] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libinterfaceProperties.so(_ZN4Foam5FieldINS_6VectorIdEEEC1Ei+0x67) [0x7f2c036ee667]
[host:08332] [ 8] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam13primitiveMesh22calcCellCentresAndVolsEv+0xad) [0x7f2c00c0077d]
[host:08332] [ 9] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam13primitiveMesh11cellCentresEv+0x1d) [0x7f2c00c009bd]
[host:08332] [10] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam9polyPatch15faceCellCentresEv+0x58) [0x7f2c00b6a5a8]
[host:08332] [11] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam18processorPolyPatch12initGeometryEv+0xca) [0x7f2c00b8952a]
[host:08332] [12] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam16polyBoundaryMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0xd1) [0x7f2c00bb53e1]
[host:08332] [13] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8polyMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x262) [0x7f2c00be3c12]
[host:08332] [14] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam6fvMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x119) [0x7f2c0255da89]
[host:08332] [15] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libmyMotionFvMesh.so(_ZN4Foam17translation3DOFfv6updateEv+0x693) [0x7f2c01000fb3]
[host:08332] [16] myInterDyMFoam-1.5-dev [0x4226d9]
[host:08332] [17] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f2bffc40586]
[host:08332] [18] myInterDyMFoam-1.5-dev [0x41fba9]
[host:08332] *** End of error message ***
[host:08334] [ 8] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8OPstreamC1ENS_7Pstream10commsTypesEiiNS_8IOstream12streamFormatENS3_13versionNumberE+0xcc) [0x7f6d6af2689c]
[host:08334] [ 9] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam18processorPolyPatch12initGeometryEv+0xbf) [0x7f6d6aff651f]
[host:08334] [10] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam16polyBoundaryMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0xd1) [0x7f6d6b0223e1]
[host:08334] [11] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8polyMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x262) [0x7f6d6b050c12]
[host:08334] [12] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam6fvMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x119) [0x7f6d6c9caa89]
[host:08334] [13] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libmyMotionFvMesh.so(_ZN4Foam17translation3DOFfv6updateEv+0x693) [0x7f6d6b46dfb3]
[host:08334] [14] myInterDyMFoam-1.5-dev [0x4226d9]
[host:08334] [15] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f6d6a0ad586]
[host:08334] [16] myInterDyMFoam-1.5-dev [0x41fba9]
[host:08334] *** End of error message ***
[host:08333] [ 0] /lib64/libc.so.6 [0x7f6cda8d66e0]
[host:08333] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f6cda8d6645]
[host:08333] [ 2] /lib64/libc.so.6 [0x7f6cda8d66e0]
[host:08333] [ 3] /lib64/libc.so.6 [0x7f6cda91aa9a]
[host:08333] [ 4] /lib64/libc.so.6(__libc_malloc+0xa1) [0x7f6cda91c891]
[host:08333] [ 5] /usr/lib64/libstdc++.so.6(_Znwm+0x1d) [0x7f6cdb131f4d]
[host:08333] [ 6] /usr/lib64/libstdc++.so.6(_Znam+0x9) [0x7f6cdb132089]
[host:08333] [ 7]
 /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libinterfaceProperties.so(_ZN4Foam5FieldINS_6VectorIdEEEC1Ei+0x67) [0x7f6cde370667]
[host:08333] [ 8] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam9polyPatch15faceCellCentresEv+0x2a) [0x7f6cdb7ec57a]
[host:08333] [ 9] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam18processorPolyPatch12initGeometryEv+0xca) [0x7f6cdb80b52a]
[host:08333] [10] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam16polyBoundaryMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0xd1) [0x7f6cdb8373e1]
[host:08333] [11] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8polyMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x262) [0x7f6cdb865c12]
[host:08333] [12] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam6fvMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x119) [0x7f6cdd1dfa89]
[host:08333] [13] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libmyMotionFvMesh.so(_ZN4Foam17translation3DOFfv6updateEv+0x693) [0x7f6cdbc82fb3]
[host:08333] [14] myInterDyMFoam-1.5-dev [0x4226d9]
[host:08333] [15] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f6cda8c2586]
[host:08333] [16] myInterDyMFoam-1.5-dev [0x41fba9]
[host:08333] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 8332 on node host exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Do I need special decomposition in order to run dynamic meshes in parallel or did I miss something else? I suppose dynamic meshes work in parallel, don't they?

Any help or suggestions are welcome!

Last but not least here is a code snippet from my dynamic mesh class using the fvMotion solver. Can I move the mesh in this way?

    pointVectorField& pointMotionU =
        const_cast<pointVectorField&>
        (
            this->objectRegistry::lookupObject<pointVectorField>("pointMotionU")
        );

    pointMotionU.boundaryField()[bodyPatchID_] == ode_.U().value();

    Info<< "Translation: " << ode_.X().value().x() << tab
                            << ode_.X().value().y() << tab
                            << ode_.X().value().z() << endl;

    Info << "Getting new points" << endl;
//    pointField newPoints = motionPtr_->newPoints();
// this works

    Info << "Getting new points (fix)" << endl;
    pointField newPoints = stationaryPoints_ * 1.0;

    Info << "Calling for move points" << endl;
    movePoints(newPoints);          // <-- this fails
    Info << "Move points done." << endl;

    volVectorField& U = 
        const_cast<volVectorField&>(lookupObject<volVectorField>("U"));
    U.correctBoundaryConditions();

Thank you so much in advance.

Best regards.
Jean-Peer

[hjasak]

You need to run blocking comms: everyting will be fine with that.

I'll have a closer look in 1.6-dev merge...

Hrv

[jploz]

Ah...ok, it works flawless now.
Thank you.

Jean-Peer

[rlc138]

Hi.

I am having trouble moving my dynamic mesh in parallel. It seems that vertices shared by more than two processors are not moved correctly. It works fine in serial mode. I have the commsType set to blocking and I am using the laplaceFaceDecomposition solver.

I have a simple test case with a fin in a rectangular flow field. I specify a velocity of the fin's patch only in the y-direction and solve using moveDynamicMesh. The figure below shows vertices on two processors in blue and vertices shared by three processors in red. The vertices in red are not being moved correctly. Can anyone help???

Rob

[markc]

Hello All,

Just to share my experience with you: I experienced similar problems with crashes when running moving-mesh cases in parallel. I suspect that the original mesh needs to have an even number of cells. So far I have seen now everything runs fine in that case.

Brgds,

Mark

[rlc138]

Mark:

Thanks for the input.

I checked my case and it has 6000 cells (all hexahedral cells with 7005 points).

The decomposed mesh, however, does not have an even number for all processors. Do you think this is a problem?

Rob

Processor 0
Number of cells = 738
Processor 1
Number of cells = 772
Processor 2
Number of cells = 738
Processor 3
Number of cells = 769
Processor 4
Number of cells = 760
Processor 5
Number of cells = 728
Processor 6
Number of cells = 760
Processor 7
Number of cells = 735

[markc]

Hello Rob,

I have no idea whether this is a problem. I only noted for a few cases I have done
so far that this may be a clue.

Brgds,

Mark

[markc]

Btw, Hrv mentiones to use "blocking comms". Uh, what's that?

Brgds,

Mark

[rlc138]

Mark:

I already have the comms set to blocking. This controls how the interprocessor messages are sent/received and is set in foam/etc/controlDict and the options are: blocking, scheduled, or nonblocking. BTW, I have posted this issue as an OpenFOAM bug, to which Hrv has responded: http://www.cfd-online.com/Forums/ope...r-1-5-dev.html .

Rob