[andrewpomeroy]

I have successfully run blockMesh, snappyHexMesh and this model as a whole in serial, however when I moved to try and compile the model in parallel I'm getting an error that I have not been able to understand. I successfully generate a blockMesh and decompose the mesh into a number of subdomains (I get processor folders). However when I move onto the snappyHexMesh stage, I receive the error below. Any insight on what might be the cause of this error, or steps that I could undertake to investigate further, would be much appreciated.

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v1712                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : v1712
Arch   : "LSB;label=32;scalar=64"
Exec   : /pawsey/cle60up05/apps/PrgEnv-intel/6.0.4/intel/17.0.4.196/haswell/openfoam+/v1712/OpenFOAM-v1712/platforms/linux64CrayDPInt32Opt/bin/snappyHexMesh -parallel -overwrite
Date   : Apr 01 2018
Time   : 15:17:57
Host   : "nid00012"
PID    : 1805
I/O    : collated (maxThreadFileBufferSize 1e+09)
         Threading activated since maxThreadFileBufferSize > 0.
         Requires thread support enabled in MPI, otherwise the simulation
         may "hang".  If thread support cannot be enabled, deactivate threading
         by setting maxThreadFileBufferSize to 0 in the OpenFOAM etc/controlDict
Case   : /scratch/pawsey0106/apomeroy/newCoral
nProcs : 72
Hosts  :
(
    (nid00012 24)
    (nid00013 24)
    (nid00014 24)
)
Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
trapFpe: Floating point exception trapping enabled (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStamp (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Read mesh in = 0.34 s
[0] 
[0] 
[0] --> FOAM FATAL ERROR: 
[0] bad size -1
[0] 
[0]     From function Foam::List<T>::List(int) [with T = char]
[0]     in file lnInclude/List.C at line 47.
[0] 
FOAM parallel run aborting
[0] 
[0] #0  Foam::error::printStack(Foam::Ostream&) addr2line failed
[0] #1  Foam::error::abort() addr2line failed
[0] #2  Foam::fileOperations::masterUncollatedFileOperation::readAndSend(Foam::fileName const&, Foam::IOstream::compressionType, Foam::UList<int> const&, Foam::PstreamBuffers&) addr2line failed
[0] #3  Foam::fileOperations::masterUncollatedFileOperation::NewIFstream(Foam::fileName const&) const addr2line failed
[0] #4  Foam::functionEntries::includeEntry::execute(Foam::dictionary&, Foam::Istream&) addr2line failed
[0] #5  Foam::functionEntry::execute(Foam::word const&, Foam::dictionary&, Foam::Istream&) addr2line failed
[0] #6  Foam::entry::New(Foam::dictionary&, Foam::Istream&, Foam::entry::inputMode) addr2line failed
[0] #7  Foam::dictionary::read(Foam::Istream&, bool) addr2line failed
[0] #8  Foam::dictionary::dictionary(Foam::fileName const&, Foam::dictionary const&, Foam::Istream&) addr2line failed
[0] #9  Foam::dictionaryEntry::dictionaryEntry(Foam::keyType const&, Foam::dictionary const&, Foam::Istream&) addr2line failed
[0] #10  Foam::entry::New(Foam::dictionary&, Foam::Istream&, Foam::entry::inputMode) addr2line failed
[0] #11  Foam::dictionary::read(Foam::Istream&, bool) addr2line failed
[0] #12  Foam::operator>>(Foam::Istream&, Foam::dictionary&) addr2line failed
[0] #13  Foam::baseIOdictionary::readData(Foam::Istream&) addr2line failed
[0] #14  Foam::fileOperations::masterUncollatedFileOperation::read(Foam::regIOobject&, bool, Foam::IOstream::streamFormat, Foam::word const&) const addr2line failed
[0] #15  Foam::regIOobject::readHeaderOk(Foam::IOstream::streamFormat, Foam::word const&) addr2line failed
[0] #16  Foam::IOdictionary::IOdictionary(Foam::IOobject const&) addr2line failed
[0] #17  ? addr2line failed
[0] #18  __libc_start_main addr2line failed
[0] #19  ? addr2line failed
Rank 0 [Sun Apr  1 15:18:00 2018] [c0-0c0s3n0] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
srun: error: nid00012: task 0: Aborted
srun: Terminating job step 2749891.0
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 2749891.0 ON nid00012 CANCELLED AT 2018-04-01T15:18:01 ***
srun: error: nid00014: tasks 48-50,52-53,56-59,61-64,66,68-69,71: Killed
srun: error: nid00012: tasks 1,3-5,7-23: Killed
srun: error: nid00013: tasks 24-40,42-47: Killed
srun: error: nid00013: task 41: Killed
srun: error: nid00014: tasks 51,54-55,60,65,67,70: Killed
srun: error: nid00012: tasks 2,6: Killed

The contents of snappyHexMesh:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  5                                     |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      snappyHexMeshDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

// Which of the steps to run
castellatedMesh true;
snap            true;
addLayers       true;


// Geometry. Definition of all surfaces. All surfaces are of class
// searchableSurface.
// Surfaces are used
// - to specify refinement for any mesh cell intersecting it
// - to specify refinement for any mesh cell inside/outside/near
// - to 'snap' the mesh boundary to the surface
geometry
{
    tableCoral
    {
        type triSurfaceMesh;
        file "table_coral.stl";
    }
};

// There is no refinement box specified. This could be added to improve resolution. 

// Settings for the castellatedMesh generation.
castellatedMeshControls
{

    // Refinement parameters
    // ~~~~~~~~~~~~~~~~~~~~~
    // If local number of cells is >= maxLocalCells on any processor
    // switches from from refinement followed by balancing
    // (current method) to (weighted) balancing before refinement.
    maxLocalCells 1000000;

    // Overall cell limit (approximately). Refinement will stop immediately
    // upon reaching this number so a refinement level might not complete.
    // Note that this is the number of cells before removing the part which
    // is not 'visible' from the keepPoint. The final number of cells might
    // actually be a lot less.
    maxGlobalCells 10000000;

    // The surface refinement loop might spend lots of iterations refining just a
    // few cells. This setting will cause refinement to stop if <= minimumRefine
    // are selected for refinement. Note: it will at least do one iteration
    // (unless the number of cells to refine is 0)
    minRefinementCells 3;

    // Number of buffer layers between different levels.
    // 1 means normal 2:1 refinement restriction, larger means slower
    // refinement.
    nCellsBetweenLevels 10;

    // Explicit feature edge refinement
    // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    // Specifies a level for any cell intersected by its edges.
    // This is a featureEdgeMesh, read from constant/triSurface for now.
    features

    (
        {
            file "Table_Edit.extendedFeatureEdgeMesh";
            level 2;
        }
    );


    // Surface based refinement
    // ~~~~~~~~~~~~~~~~~~~~~~~~

    // Specifies two levels for every surface. The first is the minimum level,
    // every cell intersecting a surface gets refined up to the minimum level.
    // The second level is the maximum level. Cells that 'see' multiple
    // intersections where the intersections make an
    // angle > resolveFeatureAngle get refined up to the maximum level.

    refinementSurfaces
    {
        tableCoral
        {
            // Surface-wise min and max refinement level
            level (2 2);
        }
    }

    resolveFeatureAngle 30;


    // Region-wise refinement
    // ~~~~~~~~~~~~~~~~~~~~~~

    // Specifies refinement level for cells in relation to a surface. One of
    // three modes
    // - distance. 'levels' specifies per distance to the surface the
    //   wanted refinement level. The distances need to be specified in
    //   descending order.
    // - inside. 'levels' is only one entry and only the level is used. All
    //   cells inside the surface get refined up to the level. The surface
    //   needs to be closed for this to be possible.
    // - outside. Same but cells outside.

    refinementRegions
{
}

    // Mesh selection
    // ~~~~~~~~~~~~~~

    // After refinement patches get added for all refinementSurfaces and
    // all cells intersecting the surfaces get put into these patches. The
    // section reachable from the locationInMesh is kept.
    // NOTE: This point should never be on a face, always inside a cell, even
    // after refinement.
    // This is an outside point locationInMesh (-0.033 -0.033 0.0033);
    locationInMesh (0.49 0.49 0.49); // Inside point

    // Whether any faceZones (as specified in the refinementSurfaces)
    // are only on the boundary of corresponding cellZones or also allow
    // free-standing zone faces. Not used if there are no faceZones.
    allowFreeStandingZoneFaces true;
}



// Settings for the snapping.
snapControls
{
    //- Number of patch smoothing iterations before finding correspondence
    //  to surface
    nSmoothPatch 3;

    //- Relative distance for points to be attracted by surface feature point
    //  or edge. True distance is this factor times local
    //  maximum edge length.
    tolerance 2.0;

    //- Number of mesh displacement relaxation iterations.
    nSolveIter 30;

    //- Maximum number of snapping relaxation iterations. Should stop
    //  before upon reaching a correct mesh.
    nRelaxIter 5;

    // Feature snapping

        //- Number of feature edge snapping iterations.
        //  Leave out altogether to disable.
        nFeatureSnapIter 10;

        //- Detect (geometric) features by sampling the surface
        implicitFeatureSnap false;

        //- Use castellatedMeshControls::features
        explicitFeatureSnap true;

        //- Detect features between multiple surfaces
        //  (only for explicitFeatureSnap, default = false)
        multiRegionFeatureSnap true;
}



// Settings for the layer addition.
addLayersControls
{
    // Are the thickness parameters below relative to the undistorted
    // size of the refined cell outside layer (true) or absolute sizes (false).
    relativeSizes true;

    // Per final patch (so not geometry!) the layer information
    layers
    {
        tableCoral
        {
            nSurfaceLayers 3;
        }
    }

    // Expansion factor for layer mesh
    expansionRatio 1.2;


    // Wanted thickness of final added cell layer. If multiple layers
    // is the thickness of the layer furthest away from the wall.
    // Relative to undistorted size of cell outside layer.
    // See relativeSizes parameter.
    finalLayerThickness 0.9;

    // Minimum thickness of cell layer. If for any reason layer
    // cannot be above minThickness do not add layer.
    // See relativeSizes parameter.
    minThickness 0.1;

    // If points get not extruded do nGrow layers of connected faces that are
    // also not grown. This helps convergence of the layer addition process
    // close to features.
    nGrow 1;


    // Advanced settings

    // When not to extrude surface. 0 is flat surface, 90 is when two faces
    // are perpendicular
    featureAngle 30;

    // Maximum number of snapping relaxation iterations. Should stop
    // before upon reaching a correct mesh.
    nRelaxIter 5;

    // Number of smoothing iterations of surface normals
    nSmoothSurfaceNormals 1;

    // Number of smoothing iterations of interior mesh movement direction
    nSmoothNormals 3;

    // Smooth layer thickness over surface patches
    nSmoothThickness 10;

    // Stop layer growth on highly warped cells
    maxFaceThicknessRatio 0.5;

    // Reduce layer growth where ratio thickness to medial
    // distance is large
    maxThicknessToMedialRatio 0.3;

    // Angle used to pick up medial axis points
    minMedianAxisAngle 90;

    // Create buffer region for new layer terminations
    nBufferCellsNoExtrude 0;


    // Overall max number of layer addition iterations. The mesher will exit
    // if it reaches this number of iterations; possibly with an illegal
    // mesh.
    nLayerIter 50;

    // Max number of iterations after which relaxed meshQuality controls
    // get used. Up to nRelaxIter it uses the settings in meshQualityControls,
    // after nRelaxIter it uses the values in meshQualityControls::relaxed.
    nRelaxedIter 20;
}



// Generic mesh quality settings. At any undoable phase these determine
// where to undo.
meshQualityControls
{
    #include "meshQualityDict"

    // Optional : some meshing phases allow usage of relaxed rules.
    // See e.g. addLayersControls::nRelaxedIter.
    relaxed
    {
        //- Maximum non-orthogonality allowed. Set to 180 to disable.
        maxNonOrtho 75;
    }

    // Advanced

    //- Number of error distribution iterations
    nSmoothScale 4;
    //- Amount to scale back displacement at error points
    errorReduction 0.75;
}


// Advanced

// Write flags
writeFlags
(
    scalarLevels    // write volScalarField with cellLevel for postprocessing
    layerSets       // write cellSets, faceSets of faces in layer
    layerFields     // write volScalarField for layer coverage
);


// Merge tolerance. Is fraction of overall bounding box of initial mesh.
// Note: the write tolerance needs to be higher than this.
mergeTolerance 1E-6;


// ************************************************************************* //

[Antimony]

Hi,

When you try the parallel case, do you remove the previously generated mesh and then start from scratch? This may not be the solution, but is worth giving a shot...

Hope this helps.

Cheers,
Antimony

[andrewpomeroy]

Hi Antimony,
Thanks for your reply.

I ended up trying to assemble my model again using a single processor (not parallel) and it then I found an error - it seems when I copied over the model I missed a file. So I ended up getting the single processor model working. So that seems to be what the error was referring to.

I have since gone back to parallel and am faced with new errors. I've deduced that this is due to the fact I have cyclic boundary conditions. Once I removed those and made them patches, everything worked out. I'm still investigating why the cyclic boundaries are causing problems. I've got to do some digging in these forums...

Thanks again for taking the time to put forward a suggestion - it prompted me to restart the process and that was really helpful.