[paul b]

Dear forum,

trying to run moving mesh cases in parallel using the laplaceFaceDecomposition motion solver I encounter two problems:

1) In 1.5-dev there has been a bug related to the points on processor boundaries not being moved correctly during parallel runs. See this post for reference:

http://www.cfd-online.com/Forums/ope...r-1-5-dev.html

This is supposed to be corrected in the new 1.6-ext version, however I still encounter that error. Comms are set to blocking, relaxing matching tolerances just delays the problem. To illustrate this I set up a 2D case using dynamicBodyFvMesh (colours show processors):



2) Another 3d case, again using dynamicBodyFvMesh runs fine first, but then suddenly the motion solver diverges (1000 iterations) and after several timesteps crashes. The resulting mesh is completely distorted.

Before (colored patches = processor boundaries):


After:



I tried this using the moveDynamicMesh utility and ICCG as well as PCG solvers for motionU (AMG solver does not seem to work any more for solving the motionU field).

What am I doing wrong or whats still wrong with the code?

If anybody had the time to take a look at my cases I'd be really glad. I'm trying to du parallel runs with that motion solver for about half a year now and thought the issue would be solved with the new ext release.

Hoping for your help!

Sincerely, Paul

ps. I can email the cases, also some videos, the board won't let me upload files of that size...

[deepsterblue]

Hrv suggested using GMRES instead, but I still had problems similar to what you're seeing. You could try this (might need to tweak it a bit):

    
    motionU
    {
        solver          GMRES;
        nDirections     8;
        preconditioner  Cholesky;
        tolerance       1e-09;
        relTol          0.0001;
    }

Otherwise, if you don't mind restricting yourself to tri-prism / tet meshes, you can look at mesquiteMotionSolver instead, which I've managed to hack to work in parallel (see attached file)

These options tend to work okay for me (might need some tweaking too):

mesquiteOptions
{
    // Optimization metric
    optMetric               AspectRatioGamma;

    // Objective function
    objFunction             LPtoP;

    // Optimization algorithm
    optAlgorithm            FeasibleNewton;

    // Termination criteria sub-dictionary (takes default values if not specified)
    // Specifying an empty sub-dictionary terminates with available options
    tcInner
    {
       relGradL2            1e-2;
       cpuTime              0.5;
    }
    // tcOuter
    // {}

    // For composite functions, two objectives need to be specified
    firstFunction           LPtoP;
    secondFunction          LInf;

    // For scaled functions, scale and objective needs to be specified
    // scaleFunction        PMeanP;
    // scale                1.5;

    // Power value for the LPtoP objective function
    pValue                  2;
    power                   2;

    // Specify a tolerance for the CG solver
    tolerance               1e-4;

    // Specify number of CG sweeps
    nSweeps                 1;

    // Specify slip patches for the motionSolver
    slipPatches
    {
        sideWall;
        topWall;
    }

    /*
    cylindricalConstraints
    {
        // Specify options per slip patch
        sideWall
        {
            axisPoint      (0.0 0.0 0.0);
            axisVector     (0.0 0.0 1.0);
            radius          1.0;
        }
    }
    */

    // Specify fixedValue patches for the motionSolver
    fixedValuePatches
    {
        topWall
        {
            //type          angularOscillatingDisplacement;
            //amplitude     -0.0125;
            type          oscillatingDisplacement;
            amplitude     (0 0 -0.01);
            axis          (1 0 0);
            origin        (0 0 3);
            angle0        0.0;
            omega         0.15;
            value         uniform (0 0 0);
        }
    }

    // Specify interval for surface smoothing
    surfInterval            1;
}

[paul b]

Hey Sandeep,

thanks a lot for the advice!
I will try this when I'm back at work on monday and report how far I get.

Regards,
Paul

[paul b]

Hi Sandeep,

first of all: this is exactly what I was looking for! I'm working on simulating animal locomotion, flying birds and so on, and we often have to deal with large mesh deformation. At the moment I'm using a mesh class which I build using a axisymmetric mesh which encloses the object (bird wing etc.) and which can rotate and translate in the surrounding mesh. The deformation of the latter is solved by a motion solver and coupling between meshes is done by ggi.
However, now I'd like to use the mesquite motion solver for the surrounding mesh and I'm a little stuck at this point.
1) the solver you posted compiled fine.
2) In my dynamicFvMesh class I used the tetDecompositionMotionSolver. The only thing I needed to do in order to define the motion of my moving patches was get the pointer to the motion solver object:

tetDecompositionMotionSolver& mSolver =
                dynamic_cast<tetDecompositionMotionSolver&>
                (
                motionPtr_()
                );

Then I could access the mesh motion velocity field motionU:

tetPointVectorField& motionU = mSolver.motionU();

and use this pointer to set the desired mesh motion on my moving patch.

But now with the mesquite solver by looking at the code there only seems to be a way of specifying the motion of the moving patches via the method:

applyFixedValuePatches()

which reads from the dynamicMeshDict.

To make a long story short: is there a way to specify the motion of my moving patch from within my dynamicFvMesh class or do I have to implement such a feature?

[paul b]

Ok, I had a further look at that applyFixedValuePatches() method of the mesquiteMotionSolver. Seems like the movement of the fixedValuePatches is specified via moving boundary conditions which are simply read from the dynamicMeshDict. Thus the part

pField().updateCoeffs();

where the motion for the point field pField happens has to be replaced by my own motion specification which I might pass to the mesquiteMotionSolver as an argument of the applyFixedValuePatches() method.

Please correct me if this is nonsense!

I'll try...

[deepsterblue]

I would advise against source-code modification. It's possible to look-up the "refPoints" pointField from the objectRegistry and assign boundary displacement values.

        
        // Boundary motion specified for the motionSolver
        pointField& refPoints = const_cast<pointField&>
        (
            mesh.lookupObject<pointField>("refPoints")
        );

        // Assign boundary conditions to the motion solver at patch 'pI'
        const labelList& meshPts = mesh.boundaryMesh()[pI].meshPoints();

        // Apply displacement
        forAll(meshPts,pointI)
        {
            refPoints[meshPts[pointI]] += dispField[pointI];
        }

[paul b]

Sandeep,

thanks a lot for the advice, this seems to be a more elegant way!

I'll post again if I succeed...

[hjasak]

Hi Paul,

Can I please have this case to look at - this MUST WORK and if it does not I need to know what happened.

We are running lots of parallel mesh motion and there are no open issues.

As for parallel AMG for FEM, you will appreciate they agglomerative AMG like the one used in the FVM cannot parallelise, so we need Selective AMG coarsening. This currently works in serial and I know who will parallelise this

Please let me know - I guess you can easily find my E-mail.

Hrv

[kalle]

Hi Sandeep,

cool to see you did a parallel hack! I compiled it and tried it on the simpleEngine tutorial you made last month. Just running a decomposPar with method simple and splitting in two domains gave me the same error as before: (at the first time step)

[1] Cannot find point in pts1 matching point 2 coord:(-2.11212 -0.726195 0.247738) in pts0 when using tolerance 4.38945e-05
[1] Searching started from:0 in pts1
[0] Cannot find point in pts1 matching point 4 coord:(-7.88788 -2.27381 -0.247738) in pts0 when using tolerance 4.38945e-05
[1] Compared coord:(-2.11212 -0.726195 -0.247738) with difference to point 0.495476
[0] Searching started from:2 in pts1
[0] Compared coord:(-2.21917 -0.326676 0.247738) with difference to point 6.01424
[0] Compared coord:(-7.88788 -2.27381 0.247738) with difference to point 0.495476
[0] 
[0] 
[0] --> FOAM FATAL ERROR: 
[0]  Failed point match: 
 My proc: 0
 neiProcNo: 1
 size(bufPoints): 20
 size(recvField): 20
...

Would there be some additional settings to get the case running in parallel?

Regards,
Kalle

[deepsterblue]

Kalle,
Could you post the case so that I can take a look at it?

[kalle]

Sure, of course. I've emailed it to you... (why having a limit of 97kB year 2011??) The log file is found below.

It is almost identical to the case you posted here. I only added a decomposParDict and decomposed the case. I've also uploaded the log file.

Regards,
Kalle

[deepsterblue]

Yup, there are still bugs that need to be fixed. The trouble with this particular case is that the 'valveSide' patch has points on the parallel patch on edge-boundaries as well. I suppose you could use the modified configuration that is attached.

Meanwhile, I'll look into a fix. Also, please be aware that the dynamicTopoFvMesh class is not fully parallelized for topology changes (yet), so you'll be stuck with mesh-motion only at the moment.

[kalle]

Thanks! Now it runs (as long as I don't decompose in the axial direction). Speed up seems to be good too!

Cool work this library!


Regards,
Kalle

[paul b]

Dear all,

the problem that I described in the first post is solved. Hrv was so kind as to have a look at my cases and he found that due to the metis decomposition which I used some strange edge decomposition occured which was not accounted for in the tetDecompMotionSolver (maybe Hrv could elaborate a bit further...).

He provided a fix for the problem, you can find it in the attachement.
Extract the stuff into OpenFOAM-1.6-ext/scr/tetDecompositionFiniteElement, rebuild and all will be well.


Thanks to Hrv for fixing the issue, and also thanks to Sandeep for pointing me towards the mesquite motion solver!

Regards, Paul

[flowris]

Dear all,


I have also been trying to run dynamicFvMesh in parallel, as well in 1.5-dev as 1.6-ext. i try to make a cylinder move inside a mesh. In serial, all goes well, but in parallel, some cells deform (see screenshot) until they become negative and move uncontrolled. These cells are on and near the processor boundaries.

I installed the moitionSolverFix, but this did not fix the problem. I also tried several solver for motionU. If anybody is willing to check my case, I will send the necessary files to her/him.


Kind regards,


Joris

[hjasak]

In the test you talk about we were doing simple mesh motion and this works without a problem.

In the test you have set up, there is a topological change, which is different. You need updates in the dynamic mesh support + a change in the dynamic mesh class + updated domain decomposition and reconstruction tool. I would recommend getting some support if you need this asap, or wait for the updated examples.

The 1.5-dev and 1.6-ext available to you has got a part of the features, but the new stuff makes it easier, faster and more robust.

Hrv

[flowris]

Thank you, though this is quite discouraging. While waiting for updates (how long will this take?), I was wondering if I can do this trick:
- first move the mesh in serial and save it at each time step, which I think is possible
- use it for a parallel simulation, with fixed time step

[hjasak]

I am sorry, I do not understand: what do you mean "though this is quite discouraging"? Could you please explain.

Hrv

[flowris]

Hello,

Never mind about that, I was just hoping to get my cases running in a couple of days, but it will take some more effort. Do you think the trick I propose is possible?

[deepblue17]

The laplaceFaceDecomposition and displacementLaplacian motion solver will only work in parallel if commsType is set scheduled or blocking. If commsType is set to nonBlocking they will crash.