[tikulju]

Hi!
I implemnetd mesh refinement from interDyMFoam to multiphaseEulerFoam and it seems to work well. But after a while I'm stuck with load balancing issues. Does anybody know, is there a way to redistribute the mesh during the simulation? Or if I have to code it, does anybody have idea where to start?

Thanks,
Timo

[kmooney]

Originally Posted by tikulju

Hi!
I implemnetd mesh refinement from interDyMFoam to multiphaseEulerFoam and it seems to work well. But after a while I'm stuck with load balancing issues. Does anybody know, is there a way to redistribute the mesh during the simulation? Or if I have to code it, does anybody have idea where to start?

Thanks,
Timo

I believe that the fvMeshDistribute library might do it for you but its implementation might not be so straightforward. The source code is in the dynamicMesh sub directories.

[tikulju]

Thanks Kyle, I'll try to start from that one.

- Timo

[tikulju]

Originally Posted by kmooney

I believe that the fvMeshDistribute library might do it for you but its implementation might not be so straightforward. The source code is in the dynamicMesh sub directories.

I managed to do it somehow by applying redistributePar code. But the mesh is not the same as is the used in a simulation (it is read from the directory). Any idea how to operate with the mesh, which is used in the main code (i.e. dynamicFvMesh and apparently stored in memory)?

- Timo

[tikulju]

Originally Posted by tikulju

I managed to do it somehow by applying redistributePar code. But the mesh is not the same as is the used in a simulation (it is read from the directory). Any idea how to operate with the mesh, which is used in the main code (i.e. dynamicFvMesh and apparently stored in memory)?

- Timo

Ok, I ended up using the redistributePar-code by modifying it a bit. Now my implementation involves writing and reading of mesh and fields to and from HD, but I couldn't make up any wiser way to do it. But now it works anyway

- Timo

[kmooney]

Timo,

I am currently working in coupling interDyMFoam to fvMeshDistribute. I can properly load balance the mesh, however, the refinement engine seems to get confused about the old-to-new cell refinement levels after the balance.

Were you able to fix this?

[tikulju]

Kyle,

I didn't get it actually to work. It hanged after repartition... With manual repartition the problem was the same. If I remember correctly, it had something to do with refinementHistory, which probably needs also to be updated. If you have any ideas how it could be fixed, I would be curious to know.

- Timo

[kmooney]

I have not figured it out quite yet. The refinementHistory class as a ::distribute(mapDisributePolyMesh) member function, as does the hexRef8 class which actually calls refinementHistory::distribute(...).

I added some extra pointers to dyanmicFvMesh and hacked the redistributePar source to load the dynamic meshes. This way, after the load balance I can send the processor-processor map to the mesh cutter and related structures as such:

in redistributePar.C ~line 1000

    // Do actual sending/receiving of mesh
    autoPtr<mapDistributePolyMesh> map = distributor.distribute(finalDecomp);

    mesh.meshCutter().distribute(map());

I get a seg fault at line 282 in mapDistributeTemplates.C at

                    forAll(map, i)
                    {
                        subField[i] = field[map[i]];
                    }

The segfault occurs when its preparing to send the cellLevel integer lists to processor neighbors. This is getting a little deep into mpi comms for my programming skills so it will take me some time to proceed at this point.

I've attempted to use the cellLevel volScalarField that is dumped out for post processing but without the pointLevel field I don't think the hexRef8 engine knows what to do with it. I attempted to map a pointLevel pointScalarField but unfortunately I couldn't get it to map during redistribution.

[wyldckat]

Greetings to all!

This bug report might interest you guys: http://www.openfoam.org/mantisbt/view.php?id=638
Try using the latest 2.1.x and see if you still have the same problem!

Best regards,
Bruno

[kmooney]

Originally Posted by wyldckat

Greetings to all!

This bug report might interest you guys: http://www.openfoam.org/mantisbt/view.php?id=638
Try using the latest 2.1.x and see if you still have the same problem!

Best regards,
Bruno

Hi Bruno,

I am indeed using the latest and greatest 2.1.x repo version. This bug report contains some useful discussion on mapping, however, my problem surrounds load balancing which appears to demand quite a lot of subtle processor communication. I will try and gain some insight from it regardless. I appreciate your help!

Cheers,

Kyle

[Xinze]

I am not sure whether you have got the solution or any update for this problem. Recently I also encountered this neck of performance, especially using ~DyMFoam. So I want to active this thread again.

I find this codes but it seems exist some questions.
https://github.com/tgvoskuilen/meshBalancing
https://bugs.openfoam.org/view.php?id=2461

https://bugs.openfoam.org/view.php?id=638

If possible, anyone may update the latest progress on this problem.

XZ

[kmooney]

Originally Posted by Xinze

I am not sure whether you have got the solution or any update for this problem. Recently I also encountered this neck of performance, especially using ~DyMFoam. So I want to active this thread again.

I find this codes but it seems exist some questions.
https://github.com/tgvoskuilen/meshBalancing
https://bugs.openfoam.org/view.php?id=2461

https://bugs.openfoam.org/view.php?id=638

If possible, anyone may update the latest progress on this problem.

XZ

The ptscotch dgraph error comes up quite a bit. If you switch to hierarchical its not the perfect decomp but its far more robust.

[Xinze]

@kmooney
Thank you for you constructive information. I will try to adapt this trick.

Addition, there two clues deposit here and is possible useful for improving the feature:
(1)
https://bugs.openfoam.org/view.php?id=2461
PIMPLE: iteration 1
Selected 119 cells for refinement out of 115612.
Refined from 115612 to 116445 cells.
Selected 217 split points out of a possible 10399.
Unrefined from 116445 to 114926 cells.
Maximum imbalance = 27.2854 %
Re-balancing dynamically refined mesh
Selecting decompositionMethod ptscotch
(1): ERROR: dgraphFold2: out of memory (2)
(2): ERROR: dgraphFold2: out of memory (2)
(4): ERROR: dgraphFold2: out of memory (2)

This bug can be tracked on here：
https://github.com/poulson/scotch/bl.../dgraph_fold.c

fldvelolocnbr = (orggrafptr->veloloctax != NULL) ? fldvertlocnbr : 0;

if (memAllocGroup ((void **) (void *) /* Allocate distributed graph public data */
&fldgrafptr->vertloctax, (size_t) ((fldvertlocnbr + 1) * sizeof (Gnum)),
&fldgrafptr->vnumloctax, (size_t) ( fldvertlocnbr * sizeof (Gnum)),
&fldgrafptr->veloloctax, (size_t) ( fldvelolocnbr * sizeof (Gnum)), NULL) == NULL) {
errorPrint ("dgraphFold2: out of memory (2)");
cheklocval = 1;
}

(2) Alternatively, it maybe solve these problems if update to a new parallel partitioning libraray, such as Zoltan.

http://www.cs.sandia.gov/~kddevin/pa...dagstuhl09.pdf
http://www.cs.sandia.gov/zoltan/Zoltan.html

XZ