[Kalabagh]

Hi Foamers,

My simulation is about an industrial stirred mixer. The shaft is almost 10 m long with 5 turbines of 900 mm diam. That means the final snappy mesh has to be very, very large.

As my 4 core computer was not able to perform an accurate mesh when running in serial, I tried in parallel with 3 cores. No way, it's not powerful enough.

The problem I want to solve, before going ahead with 10000000000 cores' cloud servers, is that decomposePar literally IGNORES some patches of the boundary dicts, it only takes the ones that come from blockMesh and does not create all "processorBoundaryXtoX" for each file.


Here is the decomposeParDict, all OK:

/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 5.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 5.x-197d9d3bf20a
Exec : decomposePar
Date : Jan 15 2020
Time : 19:16:50
Host : "ordenador-Precision-T1600"
PID : 6092
I/O : uncollated
Case : /home/ordenador/OpenFOAM/ordenador-5.0/run/OXIRIS
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Decomposing mesh region0

Create mesh

Calculating distribution of cells
Selecting decompositionMethod scotch

Finished decomposition in 0.17 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Distributing points to processors

Constructing processor meshes

Processor 0
Number of cells = 20073
Number of faces shared with processor 2 = 1275
Number of processor patches = 1
Number of processor faces = 1275
Number of boundary faces = 2543

Processor 1
Number of cells = 19900
Number of faces shared with processor 2 = 1325
Number of processor patches = 1
Number of processor faces = 1325
Number of boundary faces = 2523

Processor 2
Number of cells = 20027
Number of faces shared with processor 0 = 1275
Number of faces shared with processor 1 = 1325
Number of processor patches = 2
Number of processor faces = 2600
Number of boundary faces = 1334

Number of processor faces = 2600
Max number of cells = 20073 (0.365% above average 20000)
Max number of processor patches = 2 (50% above average 1.33333)
Max number of faces between processors = 2600 (50% above average 1733.33)

Time = 0

Processor 0: field transfer
Processor 1: field transfer
Processor 2: field transfer

End

Here is an example of the original boundary dict U:

 
 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  5.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       volVectorField;
    location    "0";
    object      U;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

dimensions      [0 1 -1 0 0 0 0];

internalField   uniform (0 0 0);

boundaryField
{
    walls
    {
        type            noSlip;
    }
    inlet
    {
        type            noSlip;
    }
    outlet
    {
        type            noSlip;
    }
    TURBINA
    {
    type        noSlip;
    }
    REACTOR
    {
    type        noSlip;
    }
    ROTOR
    {
    type        fixedValue;
    value        uniform (0 0 0);
    }
    ROTOR_slave
    {
    type        fixedValue;
    value        uniform (0 0 0);
    }
}

// ************************************************************************* //

And the U dictionary inside the folder "processor0":

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  5.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       volVectorField;
    location    "0";
    object      U;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

dimensions      [0 1 -1 0 0 0 0];

internalField   uniform (0 0 0);

boundaryField
{
    walls
    {
        type            noSlip;
    }
    inlet
    {
        type            noSlip;
    }
    outlet
    {
        type            noSlip;
    }
    procBoundary0to2
    {
        type            processor;
        value           uniform (0 0 0);
    }
}


// ************************************************************************* //

When it's about 3 cores it's easy to edit all the dictionaries in the processor* folders, but I want to try on an 8 core server...

Here are blockMeshDict and snappyHexMeshDict for more info:

blockMeshDict has inlet and outlet (I don't really need an inlet and outlet) because I'm using an improved cylinder-square background mesh and I don't want to modify the code.

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     |                                                 |
|   \\  /    A nd           | Copyright (C) 2016 Ehsan Madadi-Kandjani        |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      blockMeshDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
// General macros to create cylinder mesh

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

convertToMeters 1;

// Inner square side half

// Inner square side curvature

// cylinder radius

// Height of cylinder

// Base z

// Outlet z

// Number of cells at inner square

// Number of cells between inner square and circle

// Number of cells in the cylinder height

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

// 45 degree points angle

// Half of 45 degree points angle

// Inner square x and y position

// x

// y

// Circle x and y positions

// x

// y

// Inner square x and y position middle curvatures

// x

// y

// Circle x and y positions middle curvatures

// x

// y

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

vertices
(
    (0.6 -0.6 -1.0) // Vertex s0b = 0 
    (-0.6 -0.6 -1.0) // Vertex s1b = 1 
    (-0.6 0.6 -1.0) // Vertex s2b = 2 
    (0.6 0.6 -1.0) // Vertex s3b = 3 
    
    (0.777817459305203 -0.777817459305202 -1.0) // Vertex r0b = 4 
    (-0.777817459305202 -0.777817459305203 -1.0) // Vertex r1b = 5 
    (-0.777817459305202 0.777817459305203 -1.0) // Vertex r2b = 6 
    (0.777817459305203 0.777817459305202 -1.0) // Vertex r3b = 7 
    
    (0.6 -0.6 4.3) // Vertex s0t = 8 
    (-0.6 -0.6 4.3) // Vertex s1t = 9 
    (-0.6 0.6 4.3) // Vertex s2t = 10 
    (0.6 0.6 4.3) // Vertex s3t = 11 
    
    (0.777817459305203 -0.777817459305202 4.3) // Vertex r0t = 12 
    (-0.777817459305202 -0.777817459305203 4.3) // Vertex r1t = 13 
    (-0.777817459305202 0.777817459305203 4.3) // Vertex r2t = 14 
    (0.777817459305203 0.777817459305202 4.3) // Vertex r3t = 15 
);

blocks
(
    //block0
    hex (1 0 3 2 9 8 11 10)
    square
    (20 20 50)
    simpleGrading (1 1 1)
    
    //block1
    hex (0 4 7 3 8 12 15 11)
    innerCircle
    (10 20 50)
    simpleGrading (1 1 1)
    
    //block2
    hex (3 7 6 2 11 15 14 10)
    innerCircle
    (10 20 50)
    simpleGrading (1 1 1)
    
    //block3
    hex (2 6 5 1 10 14 13 9)
    innerCircle
    (10 20 50)
    simpleGrading (1 1 1)
    
    //block4
    hex (1 5 4 0 9 13 12 8)
    innerCircle
    (10 20 50)
    simpleGrading (1 1 1)
);

edges
(
    //Circle edges
    arc 7 4 (1.1 0 -1.0)
    arc 4 5 (6.73555739531045e-17 -1.1 -1.0)
    arc 5 6 (-1.1 1.34711147906209e-16 -1.0)
    arc 6 7 (6.73555739531045e-17 1.1 -1.0)
    
    //Circle edges
    arc 15 12 (1.1 0 4.3)
    arc 12 13 (6.73555739531045e-17 -1.1 4.3)
    arc 13 14 (-1.1 1.34711147906209e-16 4.3)
    arc 14 15 (6.73555739531045e-17 1.1 4.3)
    
    arc 3 0 (0.7 0 -1.0)
    arc 0 1 (0 -0.7 -1.0)
    arc 1 2 (-0.7 0 -1.0)
    arc 2 3 (0 0.7 -1.0)
    
    arc 11 8 (0.7 0 4.3)
    arc 8 9 (0 -0.7 4.3)
    arc 9 10 (-0.7 0 4.3)
    arc 10 11 (0 0.7 4.3)
    
);

patches
(
    wall walls
    (
        (4 7 15 12)
        (5 4 12 13)
        (6 5 13 14)
        (7 6 14 15)
    )
    
    patch inlet
    (
        (3 0 1 2)
        (3 7 4 0)
        (2 6 7 3)
        (1 5 6 2)
        (0 4 5 1)
    )
    
    patch outlet
    (
        (11 10 9 8)
        (11 8 12 15)
        (10 11 15 14)
        (9 10 14 13)
        (8 9 13 12)
    )
);

mergePatchPairs
(
);

/*--------------------------------*- C++ -*----------------------------------*\
 | =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  5                                     |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      snappyHexMeshDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

// Which of the steps to run
castellatedMesh true;
snap            true;
addLayers       true;

geometry
{
    TURBINA
    {
        type triSurfaceMesh;
        file "TURBINA_SCALED.stl";
    }

    REACTOR
    {
    type triSurfaceMesh;
    file "REACTOR_SCALED.stl";
    }

    ROTOR
    {
    type triSurfaceMesh;
    file "ROTOR_SCALED.stl";
    }

};



// Settings for the castellatedMesh generation.
castellatedMeshControls
{

    // Refinement parameters
    // ~~~~~~~~~~~~~~~~~~~~~

    // If local number of cells is >= maxLocalCells on any processor
    // switches from from refinement followed by balancing
    // (current method) to (weighted) balancing before refinement.
    maxLocalCells 1000000;

    // Overall cell limit (approximately). Refinement will stop immediately
    // upon reaching this number so a refinement level might not complete.
    // Note that this is the number of cells before removing the part which
    // is not 'visible' from the keepPoint. The final number of cells might
    // actually be a lot less.
    maxGlobalCells 4000000;

    // The surface refinement loop might spend lots of iterations refining just a
    // few cells. This setting will cause refinement to stop if <= minimumRefine
    // are selected for refinement. Note: it will at least do one iteration
    // (unless the number of cells to refine is 0)
    minRefinementCells 10;

    // Allow a certain level of imbalance during refining
    // (since balancing is quite expensive)
    // Expressed as fraction of perfect balance (= overall number of cells /
    // nProcs). 0=balance always.
    maxLoadUnbalance 0.10;


    // Number of buffer layers between different levels.
    // 1 means normal 2:1 refinement restriction, larger means slower
    // refinement.
    nCellsBetweenLevels 3;



    // Explicit feature edge refinement
    // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    // Specifies a level for any cell intersected by its edges.
    // This is a featureEdgeMesh, read from constant/triSurface for now.
    features
    (
        {
            file "TURBINA_SCALED.eMesh";
            level 6;
        }
    {
        file "REACTOR_SCALED.eMesh";
        level 5;
    }
    {
        file "ROTOR_SCALED.eMesh";
        level 0;
    }
    );



    // Surface based refinement
    // ~~~~~~~~~~~~~~~~~~~~~~~~

    // Specifies two levels for every surface. The first is the minimum level,
    // every cell intersecting a surface gets refined up to the minimum level.
    // The second level is the maximum level. Cells that 'see' multiple
    // intersections where the intersections make an
    // angle > resolveFeatureAngle get refined up to the maximum level.

    refinementSurfaces
    {
    TURBINA
        {
            // Surface-wise min and max refinement level
            level (3 4);

            // Optional specification of patch type (default is wall). No
            // constraint types (cyclic, symmetry) etc. are allowed.
            patchInfo
            {
               type wall;
            }
        }
    
    REACTOR
    {
        level (3 4);

        patchInfo
        {
           type wall;
        }
    }
    
    ROTOR
    {
         level (0 0);//1 2);
         cellZone ROTOR;
         faceZone ROTOR;
         cellZoneInside inside;
    }
    }

    // Resolve sharp angles
    resolveFeatureAngle 1;


    // Region-wise refinement
    // ~~~~~~~~~~~~~~~~~~~~~~

    // Specifies refinement level for cells in relation to a surface. One of
    // three modes
    // - distance. 'levels' specifies per distance to the surface the
    //   wanted refinement level. The distances need to be specified in
    //   descending order.
    // - inside. 'levels' is only one entry and only the level is used. All
    //   cells inside the surface get refined up to the level. The surface
    //   needs to be closed for this to be possible.
    // - outside. Same but cells outside.

    refinementRegions
    {
        REACTOR
        {
            mode inside;
        levels ((0 0));//1));
    }

    ROTOR
    {
        mode inside;
        levels ((1 1));//1 1));
        cellZone ROTOR;
        faceZone ROTOR;
        cellZoneInside inside;
    }
    }

    // Mesh selection
    // ~~~~~~~~~~~~~~

    // After refinement patches get added for all refinementSurfaces and
    // all cells intersecting the surfaces get put into these patches. The
    // section reachable from the locationInMesh is kept.
    // NOTE: This point should never be on a face, always inside a cell, even
    // after refinement.
    locationInMesh (0.05000000 0.0000000 0.0000000);


    // Whether any faceZones (as specified in the refinementSurfaces)
    // are only on the boundary of corresponding cellZones or also allow
    // free-standing zone faces. Not used if there are no faceZones.
    allowFreeStandingZoneFaces true;
}



// Settings for the snapping.
snapControls
{
    //- Number of patch smoothing iterations before finding correspondence
    //  to surface
    nSmoothPatch 4;

    //- Relative distance for points to be attracted by surface feature point
    //  or edge. True distance is this factor times local
    //  maximum edge length.
    tolerance 2.0;

    //- Number of mesh displacement relaxation iterations.
    nSolveIter 30;

    //- Maximum number of snapping relaxation iterations. Should stop
    //  before upon reaching a correct mesh.
    nRelaxIter 5;

    // Feature snapping

        //- Number of feature edge snapping iterations.
        //  Leave out altogether to disable.
        nFeatureSnapIter 10;

        //- Detect (geometric only) features by sampling the surface
        //  (default=false).
        implicitFeatureSnap false;

        //- Use castellatedMeshControls::features (default = true)
        explicitFeatureSnap true;

        //- Detect points on multiple surfaces (only for explicitFeatureSnap)
        multiRegionFeatureSnap false;
}



// Settings for the layer addition.
addLayersControls
{
    // Are the thickness parameters below relative to the undistorted
    // size of the refined cell outside layer (true) or absolute sizes (false).
    relativeSizes true;

    // Per final patch (so not geometry!) the layer information
    layers
    {
        TURBINA
        {
            nSurfaceLayers 3;
        }

    REACTOR
    {
        nSurfaceLayers 3;
    }
    }

    // Expansion factor for layer mesh
    expansionRatio 1.5;

    // Wanted thickness of final added cell layer. If multiple layers
    // is the thickness of the layer furthest away from the wall.
    // Relative to undistorted size of cell outside layer.
    // See relativeSizes parameter.
    finalLayerThickness 0.5;

    // Minimum thickness of cell layer. If for any reason layer
    // cannot be above minThickness do not add layer.
    // Relative to undistorted size of cell outside layer.
    minThickness 0.1;

    // If points get not extruded do nGrow layers of connected faces that are
    // also not grown. This helps convergence of the layer addition process
    // close to features.
    // Note: changed(corrected) w.r.t 17x! (didn't do anything in 17x)
    nGrow 0;

    // Advanced settings

    // When not to extrude surface. 0 is flat surface, 90 is when two faces
    // are perpendicular
    featureAngle 100;

    // At non-patched sides allow mesh to slip if extrusion direction makes
    // angle larger than slipFeatureAngle.
    slipFeatureAngle 30;

    // Maximum number of snapping relaxation iterations. Should stop
    // before upon reaching a correct mesh.
    nRelaxIter 3;

    // Number of smoothing iterations of surface normals
    nSmoothSurfaceNormals 1;

    // Number of smoothing iterations of interior mesh movement direction
    nSmoothNormals 3;

    // Smooth layer thickness over surface patches
    nSmoothThickness 10;

    maxFaceThicknessRatio 0.5;

    maxThicknessToMedialRatio 0.3;

    minMedianAxisAngle 90;

    nBufferCellsNoExtrude 0;

    nLayerIter 50;
}

meshQualityControls
{
    #include "meshQualityDict"
    nSmoothScale 4;
    errorReduction 0.75;
}

writeFlags
(
    scalarLevels
    layerSets
    layerFields     // write volScalarField for layer coverage
);

// ************************************************************************* //

Thanks.

[tomf]

Hi,

The decomposition is fine, if you look at the bold lines below, you will see that processor 0 only connects to processor2, not to processor1. That is why there is no procBoundary0to1.

Processor 0
Number of cells = 20073
Number of faces shared with processor 2 = 1275
Number of processor patches = 1
Number of processor faces = 1275
Number of boundary faces = 2543

Processor 1
Number of cells = 19900
Number of faces shared with processor 2 = 1325
Number of processor patches = 1
Number of processor faces = 1325
Number of boundary faces = 2523

Processor 2
Number of cells = 20027
Number of faces shared with processor 0 = 1275
Number of faces shared with processor 1 = 1325
Number of processor patches = 2
Number of processor faces = 2600
Number of boundary faces = 1334

So I think you can continue just fine after the decomposition to run in parallel. You might run out of memory however?

Regards,
Tom

[Kalabagh]

Well, seems that the problem has changed.

Going ahead with snappy, it does not find a trisurfaceMesh file.

It is searching the file inside processor*/trisurface while this folder has not been created.

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 4.x-56a4152a9309
Exec   : snappyHexMesh -parallel
Date   : Jan 17 2020
Time   : 18:08:56
Host   : "ip-172-31-5-149.us-west-2.compute.internal"
PID    : 6392
Case   : /home/cfd/OpenFOAM/cfd-4.x/OXIRIS
nProcs : 8
Slaves : 
7
(
"ip-172-31-5-149.us-west-2.compute.internal.6393"
"ip-172-31-5-149.us-west-2.compute.internal.6394"
"ip-172-31-5-149.us-west-2.compute.internal.6395"
"ip-172-31-5-149.us-west-2.compute.internal.6397"
"ip-172-31-5-149.us-west-2.compute.internal.6398"
"ip-172-31-5-149.us-west-2.compute.internal.6399"
"ip-172-31-5-149.us-west-2.compute.internal.6400"
)

Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Read mesh in = 0.09 s

Overall mesh bounding box  : (-1.1 -1.1 -1) (1.1 1.1 4.3)
Relative tolerance         : 1e-06
Absolute matching distance : 6.14573e-06

[0] 
[0] 
[0] --> FOAM FATAL ERROR: 
[0] Cannot find triSurfaceMesh starting from "/home/cfd/OpenFOAM/cfd-4.x/OXIRIS/processor0/constant/triSurface/TURBINA"
[0] 
[0]     From function static const Foam::fileName& Foam::triSurfaceMesh::checkFile(const Foam::fileName&, const Foam::fileName&)
[0]     in file searchableSurface/triSurfaceMesh.C at line 109.
[0] [7] 
[7] 
[7] --> FOAM FATAL ERROR: 
[7] Cannot find triSurfaceMesh starting from "/home/cfd/OpenFOAM/cfd-4.x/OXIRIS/processor7/constant/triSurface/TURBINA"
[7] 
[7]     From function static const Foam::fileName& Foam::triSurfaceMesh::checkFile(const Foam::fileName&, const Foam::fileName&)
[7]     in file searchableSurface/triSurfaceMesh.C at line 109.
[7] 
FOAM parallel run exiting
[7] 

FOAM parallel run exiting
[0] 
[1] 
[1] 
[1] --> FOAM FATAL ERROR: 
[1] Cannot find triSurfaceMesh starting from "/home/cfd/OpenFOAM/cfd-4.x/OXIRIS/processor1/constant/triSurface/TURBINA"
[1] 
[1]     From function static const Foam::fileName& Foam::triSurfaceMesh::checkFile(const Foam::fileName&, const Foam::fileName&)
[1]     in file searchableSurface/triSurfaceMesh.C at line 109.
[1] 
FOAM parallel run exiting

[Kalabagh]

Please, I need help for this...

[tomf]

Well it is hard to help with the information you provided. Please make sure you have your files and names of files and input in dictionaries similar to tutorials of the version of OpenFOAM that you use.

[Kalabagh]

In fact, this happens to me only when trying with a AWS server. The 3D files are never in processor*/constant/triSurface so no idea why it is searching them in that direction.

Also, the AMI I'm using is a public one with OF4 installed. My local version is OF5. I have remeshed since blockMesh in the server but maybe there is another type version incompatibility...

Many thanks for keeping in touch

[Kalabagh]

Hi, I have found the solution.


stl entry for snappyHexMesh is not the same with OF4 than for OF5.


In OF5 is like this:


NAMEOFPATCH (whatever you want)


type triSurfaceMesh;
file "REALNAMEOFFILE.stl";


And in OF4 is like this:


REALNAMEOFFILE.stl


type triSurfaceMesh;




Many thanks.