[snappyHexMesh] sHM in parallel, different decompose methods -> different meshes and results?

Babelsty · December 6, 2021, 07:29

While experimenting which method for snappyHexMesh in parallel is fastest for my case, I discovered that the different decompose methods lead to different meshes and thus significantly different results.

My actual goal is to determine the forces on a wing in two-phase space (water and air) with interFoam. For the following tetscases, I always resolved wings and edges with refinement level 3 (resulting in ~356k cells) and ran the simulation 1800 iterations. I'm using OpenFOAM v2012.

To systematically investigate this problem, I have a reference test case that computes snappyHexMesh as well as interFoam without parallel run on a single CPU.
I compared the lifting force of the last iteration of this reference test case with the other last lifting forces of the other test cases, where I only changed the decompose method and CPU count to estimate the deviations from the reference force:

-hierarchical (2 CPUs), order (1 2 1): ~7%.
-hierarchical (2 CPUs), order (1 1 2): ~13,8%
-hierarchical (4 CPUs), order (1 2 2): ~25.8%
-scotch (4 CPUs): ~16.3

When I previously ran sHM without parallel run, interFoam in parallel came up with approximately the same results:

-hierarchical (4 CPUs), order (1 2 2): ~1.937e-4 %.
-scotch (2 CPUs): 5.107e-4 %
-scotch (4 CPUs): 6.6916e-4 %

so it can be said that only sHM in parallel leads to different meshes and thus different results depending on the method. Of course, checkMesh found all meshes to be "ok".
Synatx for my commands to sHM in parallel run:

blockMesh
surfaceFeatureExtract
decomposePar
mpirun -np 4 snappyHexMesh -overwrite -parallel >log.sHM
reconstructParMesh -mergeTol 1e-06 -latestTime -constant

Questions:
Is this normal? Or are there any settings or flags to avoid this?
If not, how to handle it?

Any help is gratefully accepted!
Best regards,
Babelsty

AtoHM · December 7, 2021, 06:49

Not fully sure that I can follow everything, but your analysis seems to be thorough.

But, if you suspect the meshes to be the cause, why don't you compare the meshes more thoroughly? "ok" by checkMesh does not mean the meshes are the same. E.g. in Paraview you can easily create histograms for the mesh quality parameters like nonOrtho, skewness, ... . I would go and compare these numbers to proof the mesh is causing this.

Babelsty · December 8, 2021, 08:42

I think that just the fact that different cell counts (+/- 300 cells) come out between the different decompose methods makes it clear that different decompose methods interpret slightly different meshes from the same snappyHexMeshDict.

This observation can be made for everyone, for example, on the
motorBike tutorial from incompressible/simpleFoam.
I modified the snappyHexMeshDict (refinement level 3), then decomposeParDict (4 subdomains) and ran it once with hierarchical, and once with scotch. All other settings remained untouched, run with ./Allrun.
Result:
157956 cells running on hierarchical.
157998 cells running on scotch.
The difference of 42 cells already leads to noticeable deviations on all other mesh parameters, probably then also on the final results.

Only on my computer or can you also reproduce this?

December 6, 2021, 07:29	sHM in parallel, different decompose methods -> different meshes and results?	#1
Babelsty New Member Marcel Join Date: Dec 2021 Posts: 2 Rep Power: 0	While experimenting which method for snappyHexMesh in parallel is fastest for my case, I discovered that the different decompose methods lead to different meshes and thus significantly different results. My actual goal is to determine the forces on a wing in two-phase space (water and air) with interFoam. For the following tetscases, I always resolved wings and edges with refinement level 3 (resulting in ~356k cells) and ran the simulation 1800 iterations. I'm using OpenFOAM v2012. To systematically investigate this problem, I have a reference test case that computes snappyHexMesh as well as interFoam without parallel run on a single CPU. I compared the lifting force of the last iteration of this reference test case with the other last lifting forces of the other test cases, where I only changed the decompose method and CPU count to estimate the deviations from the reference force: -hierarchical (2 CPUs), order (1 2 1): ~7%. -hierarchical (2 CPUs), order (1 1 2): ~13,8% -hierarchical (4 CPUs), order (1 2 2): ~25.8% -scotch (4 CPUs): ~16.3 When I previously ran sHM without parallel run, interFoam in parallel came up with approximately the same results: -hierarchical (4 CPUs), order (1 2 2): ~1.937e-4 %. -scotch (2 CPUs): 5.107e-4 % -scotch (4 CPUs): 6.6916e-4 % so it can be said that only sHM in parallel leads to different meshes and thus different results depending on the method. Of course, checkMesh found all meshes to be "ok". Synatx for my commands to sHM in parallel run: blockMesh surfaceFeatureExtract decomposePar mpirun -np 4 snappyHexMesh -overwrite -parallel >log.sHM reconstructParMesh -mergeTol 1e-06 -latestTime -constant Questions: Is this normal? Or are there any settings or flags to avoid this? If not, how to handle it? Any help is gratefully accepted! Best regards, Babelsty

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Coarser meshes can produce better results?	asda3D	Main CFD Forum	6	April 21, 2019 23:00
interFOAM: different results for similar meshes	bramDeJaegher	OpenFOAM Running, Solving & CFD	0	January 31, 2017 19:03
[snappyHexMesh] different sHM results on same geometry when everythin is in one or several stl-files.	Laika	OpenFOAM Meshing & Mesh Conversion	1	September 8, 2016 05:09
Weird results on Cavity and unstructured meshes	x86_64leon	OpenFOAM Running, Solving & CFD	12	March 1, 2016 20:44
Different results from similar quality cfx and ICEM meshes	Nick R	CFX	3	January 17, 2011 08:48

December 7, 2021, 06:49		#2
AtoHM Senior Member M Join Date: Dec 2017 Posts: 703 Rep Power: 13	Not fully sure that I can follow everything, but your analysis seems to be thorough. But, if you suspect the meshes to be the cause, why don't you compare the meshes more thoroughly? "ok" by checkMesh does not mean the meshes are the same. E.g. in Paraview you can easily create histograms for the mesh quality parameters like nonOrtho, skewness, ... . I would go and compare these numbers to proof the mesh is causing this.

December 8, 2021, 08:42		#3
Babelsty New Member Marcel Join Date: Dec 2021 Posts: 2 Rep Power: 0	I think that just the fact that different cell counts (+/- 300 cells) come out between the different decompose methods makes it clear that different decompose methods interpret slightly different meshes from the same snappyHexMeshDict. This observation can be made for everyone, for example, on the motorBike tutorial from incompressible/simpleFoam. I modified the snappyHexMeshDict (refinement level 3), then decomposeParDict (4 subdomains) and ran it once with hierarchical, and once with scotch. All other settings remained untouched, run with ./Allrun. Result: 157956 cells running on hierarchical. 157998 cells running on scotch. The difference of 42 cells already leads to noticeable deviations on all other mesh parameters, probably then also on the final results. Only on my computer or can you also reproduce this?