[tgvosk]

To anyone interested,

I have put together an add-on for OpenFoam to allow run-time parallel load balancing. This is particularly useful for parallel VOF simulations with dynamic meshing enabled. The code is at https://github.com/tgvoskuilen/meshBalancing, and some of the issues it first had have been fixed in bug report 1203 (http://www.openfoam.org/mantisbt/view.php?id=1203)

I haven't tested it in a few months, so no guarantees.

-Tyler

[guin]

Just in case anyone is interested I ported a minimal version of this code (just the load-balancing stuff) to OF-4.1. So far it seems to work "as every other prototype". I created a bug-report because there are some crashes that seem to come from ptscotch library (https://bugs.openfoam.org/view.php?id=2461). In the meanwhile, if anyone feels confident enough to deal with the error issue I am sure that the whole user community would be very grateful.

[brugiere_olivier]

Dear Rodrigo,

I've try your library but I've some problems.
As in your tutorial, if the simulation is a laminar case there is no problems. But if you are using RAS or LES models, it's doesn't work. Have you same problems ? And have you try to work on in or not ?

Thanks for your answer

Regards

Olivier

[guin]

Hi Brugiere,
Sorry, I didn't test it with turbulence models up to now. This might be related with the viscosities issue mentioned by Tyler in https://github.com/tgvoskuilen/meshBalancing (last point when you scroll down to the bottom). But that's something I would like to research further. I have to prepare some material some material for a conference next week. I will try to work on it once I am back. Indeed I still owe a related bug report to the Scotch development team!

[brugiere_olivier]

Dear Rodrigo,

Thanks for your answer. I think it's not a problem due to viscosity model. I have this warning :

--> FOAM Warning :
    From function Foam::fixedValueFvPatchField<Type>::fixedValueFvPatchField(const Foam::fixedValueFvPatchField<Type>&, const Foam::fvPatch&, const Foam::DimensionedField<Type, Foam::volMesh>&, const Foam::fvPatchFieldMapper&) [with Type = double]
    in file /Softs/OpenFOAM/OpenFOAM-4.1/src/finiteVolume/lnInclude/fixedValueFvPatchField.C at line 78
    On field subsetnut patch walls patchField fixedValue : mapper does not map all values.
    To avoid this warning fully specify the mapping in derived patch fields.
--> FOAM Warning :
    From function Foam::fixedValueFvPatchField<Type>::fixedValueFvPatchField(const Foam::fixedValueFvPatchField<Type>&, const Foam::fvPatch&, const Foam::DimensionedField<Type, Foam::volMesh>&, const Foam::fvPatchFieldMapper&) [with Type = double]
    in file /Softs/OpenFOAM/OpenFOAM-4.1/src/finiteVolume/lnInclude/fixedValueFvPatchField.C at line 78
    On field subsetomega patch walls patchField fixedValue : mapper does not map all values.
    To avoid this warning fully specify the mapping in derived patch fields.

After that, the simulation crashes because solver don't find omega and nu_t.

If you find a solution, I would be happy to receive any ideas

Regards

Olivier

[iLyrics]

The dynamic load balancing with adaptive mesh refinement tools don't seem work properly in OpenFoam-3.0.1. I doubt there is something wrong in the redistribute step, which locate in the fvMeshDistribute file.

[dzordz]

Has there been any development on this front? Maybe compatibility with OFv1812?

[guin]

Originally Posted by dzordz

Has there been any development on this front? Maybe compatibility with OFv1812?

hi!
... not from my side by now. It is starting to become the water mark from my to-do list, but unfortunately it remains outside the range of workload I am able to manage in the moment :-S

I would't expect any big issue in case anyone wants to port it to OFv1812... at the very end something like 99% of the code remains unchanged from version to version.

[dzordz]

Hi,
EDIT FOR CLARIFICATION: this is set up in such a way that a OF41 files are copied to v1812 and modified accordingly. This effectively means non of the ofv1812 code is used. So basically I am trying to say, this is not ported correctly.

If anybody is interested I made the code compatible with ofv1812. The change is that I removed enabledBalancing, which is now always turned on when calling the library.

The code works normally for the added tutorial. I only have issues with my mesh, where the redistribution crashes when it reaches allowedImbalance:

Maximum imbalance = 7.0903263 %                 
Re-balancing dynamically refined mesh           
[1]                                             
[1]                                             
[1] --> FOAM FATAL ERROR:                       
[1] Cell 208 is not visible                     
[1]                                             
[1]     From function Foam::label Foam::refinementHistory::parentIndex(Foam::label) const
[1]     in file ../OpenFoam/OpenFOAM-v1812/src/dynamicMesh/lnInclude/refinementHistory.H at line 306.
[1]   
...

If there are 4 processors, 4 cells will not be visible. 6 processors, 6. And so on.

Any idea why this is happening? It is probably due to refinementHistory writting of adapted mesh, but I have no idea how to fix this.


Cheers!

[abe]

Hi,


Did you get the chance to fix the issue??


Best,
/Abe

[guin]

@Abe So far, we have something "kind-of-working" for OpenFOAM-6 (I didn't upload the version for OpenFOAM-7, because I still didn't get it working properly). Feel free to test it:
https://github.com/dolphguin/dynamic...vMesh/tree/of6
I recommend you to disable the option for refining basing on regions.... I am not totally comfortable with it and will probably replace it soon by something else that offers similar functionality and hopefully does not enter into conflict with our previous multi-field approach.


Alternatively, you may be also interested on testing the implementation from the TU-Darmstadt team, who seem to have worked on their own extension of Tyler's code too
https://github.com/ElsevierSoftwareX/SOFTX_2018_143
It looks promising, because they also merged stuff for 2D-refinement alongside. Unfortunately, this repository seems a frozen snapshot of the time when they published the explanatory paper https://doi.org/10.1016/j.softx.2019.100317
In case you need something more recent, the best I can offer is a quick-&-dirty porting to OpenFOAM-6
https://github.com/dolphguin/SOFTX_2018_143/tree/of6

Nevertheless, I must warn you that I've experienced several hanging issues that prevented us from using it beyond trial simulations. Anyway, feel free to test and report ;-)

[guin]

Quick update: actually, just compiling above version in OpenFOAM-7 seems to work (...unless for the incompatibility between the region-based refinement and the load balancing)
https://github.com/dolphguin/dynamic...vMesh/tree/of7

[abe]

Dear Rodrigo,

I really appreciate your fast reply & comments. I will use your code right away and can come back with some results. I need the AMR+Balancing in a solver which a volScalarField defines the refinement criterion. So, I will not use the multi-field approach at the moment.

I had tested the code that is kindly provided by TU-Darmstadt team. Everything works fine with the compiling, load balancing and mapping. The code works very well in fully hex cells. But when there are different cell shapes, I run into a problem in the first refinement occasion after the load balancing (cell redistribution). It seems something has been missed during the redistribution (maybe protectedCells list). Then, I get the following error:

************
Selected 283 cells for refinement out of 44508017.
hexRef : Dumping cell as obj to ".../processor622/cell_44331.obj"
hexRef : Dumping cell as obj to ".../processor590/cell_6
--> FOAM FATAL ERROR:
cell 44331 of level 0 uses more than 8 points of equal or lower level
Points so far:8(11798 11799 11800 11801 12386 12387 12388 12389)3080.ob

From function Foam::List<Foam::List<int>> Foam::hexRef8::setRefinement(const Foam::List<int> &, Foam:olyTopoChange &)
in file polyTopoChange/polyTopoChange/hexRef/hexRef8/hexRef8.C at line 1142.
FOAM parallel run aborting
***********

I have contacted one of the guys from TU-Darmstadt team for help. I will update the forum if I get any update on that.


Best,

[abe]

Hi again,


I used the code in OF6, and got into the same problem, i.e. crashing at the first refinement occasion after redistribution. The problem is not related to protectedCells. It comes from cells with level 0 next to the cells with level 1. These cells have more than 8 faces (since at least one of their neighbors are split) but probably has been stored as hex cells.



Any idea how to solve it?


All the best,
/Abe

[Henning86]

Hi,


I ported the library released with the paper to v1812:

Rettenmaier, Daniel, et al. "Load balanced 2D and 3D adaptive mesh refinement in OpenFOAM." SoftwareX 10 (2019): 100317.

link:
https://www.sciencedirect.com/scienc...52711018301699


source code:
https://github.com/ElsevierSoftwareX/SOFTX_2018_143

git clone https://github.com/HenningScheufler/multiDimAMR.git


Best,


Henning

[HPE]

That's bloody cool - Great many thanks!

EDIT: Will/would this implementation be in the next release?

[S_teph_2000]

Thank you for your Contribution!!
I tried to compiled it on OpenFoam 1912, and the following line in "dynamicMultiDimRefineFvMesh.C" prevented from compiling "libdynamicLoadBalanceFvMesh.so":

const label oldPointi = iter.object();

it has to be:

const label oldPointi = iter();

I still have to check that everything works correctly but: is there a particular reason to add the ".object()"?

Kazunobu

[S_teph_2000]

Originally Posted by abe

Dear Rodrigo,

I really appreciate your fast reply & comments. I will use your code right away and can come back with some results. I need the AMR+Balancing in a solver which a volScalarField defines the refinement criterion. So, I will not use the multi-field approach at the moment.

I had tested the code that is kindly provided by TU-Darmstadt team. Everything works fine with the compiling, load balancing and mapping. The code works very well in fully hex cells. But when there are different cell shapes, I run into a problem in the first refinement occasion after the load balancing (cell redistribution). It seems something has been missed during the redistribution (maybe protectedCells list). Then, I get the following error:

************
Selected 283 cells for refinement out of 44508017.
hexRef : Dumping cell as obj to ".../processor622/cell_44331.obj"
hexRef : Dumping cell as obj to ".../processor590/cell_6
--> FOAM FATAL ERROR:
cell 44331 of level 0 uses more than 8 points of equal or lower level
Points so far:8(11798 11799 11800 11801 12386 12387 12388 12389)3080.ob

From function Foam::List<Foam::List<int>> Foam::hexRef8::setRefinement(const Foam::List<int> &, Foam:olyTopoChange &)
in file polyTopoChange/polyTopoChange/hexRef/hexRef8/hexRef8.C at line 1142.
FOAM parallel run aborting
***********

I have contacted one of the guys from TU-Darmstadt team for help. I will update the forum if I get any update on that.


Best,

Hello Abe,
Did you find a way to solve this problem?
regards,
Kazunobu

[S_teph_2000]

Originally Posted by Henning86

Hi,


I ported the library released with the paper to v1812:

Rettenmaier, Daniel, et al. "Load balanced 2D and 3D adaptive mesh refinement in OpenFOAM." SoftwareX 10 (2019): 100317.

link:
https://www.sciencedirect.com/scienc...52711018301699


source code:
https://github.com/ElsevierSoftwareX/SOFTX_2018_143

git clone https://github.com/HenningScheufler/multiDimAMR.git


Best,


Henning

Hello,
I would like to know more about your version of dynamicMeshDict:
1) The solver complains about "operation" missing
2) What is the difference between "maxCellLevel" and "maxRefinement" ?

Kazunobu

[cyw]

Originally Posted by S_teph_2000

Hello,
I would like to know more about your version of dynamicMeshDict:
1) The solver complains about "operation" missing
2) What is the difference between "maxCellLevel" and "maxRefinement" ?

Kazunobu

Have you found the reason? I also met this error! Thanks