[bartek_g]

Hello all,


I'm trying to decompose my stl file into few parts to run snappyHexMesh in parallel.

I couldn't find any detailed documentation for surfaceRedistributePar OpenFOAM utility but maybe someone here will help me solve this problem.


I'm running surfaceRedistributePar on decomposed case with:

mpirun --allow-run-as-root -np 4 surfaceRedistributePar -parallel sand.stl follow

sand.stl is in constant/triSurface folder.

The geometry is described in snappyHexMeshDict:

geometry
{
    sand.stl 
    {        
    regions
        {
              ascii             // Named region in the STL file
              {
                  name sand_boundary; // User-defined patch name
              }                       // otherwise given sphere.stl_secondSolid
        } 
    type distributedTriSurfaceMesh;
    distributionType follow;
    }
    main_cylinder
    {
        type            searchableCylinder;
        point1          (2.10099737 2.10099737 -0.3);
        point2          (2.10099737 2.10099737 3.5);
        radius          2.05298263;
        name boundaryWalls; 
    }
};

I tried to run surfaceRedistributePar in OpenFOAM 2.4 but

*** An error occurred in MPI_Bsend
*** reported by process [140337071128577,140333761429507]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_BUFFER: invalid buffer pointer
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)

I tried also to run it in OpenFOAM 5.0 - in this version it couldn't find the stl file.



Line

Loading undecomposed surface "s3down10_diststl/constant/triSurface/sand.stl"

in OpenFOAM 2.4 looks like

Loading undecomposed surface ""

in 5.0 and then fatal error occures.


I checked differences between OpenFOAM 2.4 and 5.0 and couldn't find significant ones (beyond blockMeshDict file location).

[bartek_g]

I will add that the error i OpenFOAM 5.0 was:

Cannot find triSurfaceMesh starting from "/export/cluster/meshimpact/s3down10_diststl5/processor1/constant/triSurface/sand.stl"

It was possible to get it around by copying manually stl file to processor*/constant/triSurface folders.


However, the same error as in OF 2.4 is present and it seems to be related to MPI.


I use SYSTEMOPENMPI implementation in OpenFOAM 2.4.0 and OPENMPI in OF 5.0

[wildfire230]

I'm currently having this exact same problem. Please let me know if you have solved this issue.

[wildfire230]

Can someone please take a look at this? There is very limited information about how to actually use surfaceRedistributePar available. I have a large geometry file that I am trying to use with snappyHexMesh (~3+ GB). The default behavior of snappyHexMesh is to load the geometry file on every processor, which is obviously consuming a tremendous amount of memory and limiting the number of processors that I can use. Apparently surfaceRedistributePar can be used to decompose the geometry file onto the processors so that only the relevant part of the geometry is loaded. However, this does not appear to work as expected.


The normal case setup for snappyHexMesh is to store the .stl or .obj files in constant/triSurface. The workflow to use snappyHexMesh and surfaceRedistributePar together is apparently along these lines: blockMesh -> decomposePar -> surfaceRedistributePar -> snappyHexMesh. However, running decomposePar obviously does not decompose the .stl or .obj geometry, and surfaceRedistributePar will immediately complain with a fatal error that it cannot find the triSurfaceMesh in the processor*/constant/triSurface directories. Does this mean that you have to first copy the (large) full geometry file onto each of the processor directories? That seems incorrect. However, even if you do copy the file into each processor directory, surfaceRedistributePar will still fail with other errors, including the fileModificationChecking setting which can be solved by changing the setting to "timeStamp". However, then surfaceRedistributePar complains about MPI errors, such as


Selecting decompositionMethod simple
[Laptop:19880] *** An error occurred in MPI_Bsend
[Laptop:19880] *** reported by process [3787915265,0]
[Laptop:19880] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0
[Laptop:19880] *** MPI_ERR_BUFFER: invalid buffer pointer
[Laptop:19880] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[Laptop:19880] *** and potentially your MPI job)
[Laptop:19878] 3 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[Laptop:19878] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages


Does anyone actually know the correct way to use this utility?

[bartek_g]

Originally Posted by wildfire230

Does this mean that you have to first copy the (large) full geometry file onto each of the processor directories? That seems incorrect.

I coped with this by copying the link to the geometry file instead of the original file but I agree with you it is not correct. In OF 2.4 this problem is not present and it looks like a bug that appeared in the next versions.

Originally Posted by wildfire230

However, then surfaceRedistributePar complains about MPI errors, such as

Selecting decompositionMethod simple
[Laptop:19880] *** An error occurred in MPI_Bsend
[Laptop:19880] *** reported by process [3787915265,0]
[Laptop:19880] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0
[Laptop:19880] *** MPI_ERR_BUFFER: invalid buffer pointer
[Laptop:19880] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[Laptop:19880] *** and potentially your MPI job)
[Laptop:19878] 3 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[Laptop:19878] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

It seems the problem is related to MPI buffer... I had not time to read more about that but maybe MPI_BUFFER_SIZE is key to that (look at this thread)? It would be great if it solved the problem.


Which version of OpenFOAM do you have? And which implementation of MPI do you use? (you can check in etc/bashrc file in your OpenFOAM installation folder)

I will report my progress when I come back to this problem.

[wildfire230]

I think you are right that this is a bug introduced in the later versions of OpenFOAM. I am using version 6. I think I am using cray-mpich/7.6.3. I may give it a try with the other branch of OpenFOAM (v1802 I think?), but I'm not looking forward to going through compilation yet again.


If someone else sees this, please enlighten us!

[otaolafr]

hello ,
i have just discover this possibility to use in large STLs, have anyone succed in the workflow of this application? and if the bug have been solved with the newer vr 7?
thanks