[Jeanp]

Hi everyone,

I have problems using decomposePar after I do sHM. Everything its ok after snappy but when I try to decompose the mesh problems occur. I prepare my cases in my laptop (4 processors, 8GB) before sending them to a cluster (48 processors). The cases I prepare in my laptop are the same with the exception of the refinement levels in sHM and the decomposeParDict (nb of processors). I do this to have a "lighter" case to be sure I haven’t done any silly error before sending the case to the cluster. First, I had some problems decomposing in my computer, the processors(n) files contained only the constant file, there was no field transfer. I fixed that using "decomposePar -latestTime".
What I don’t understand is that in the cluster the processors(n) files don’t have the 0 file (I overwrite when I sHM) even if I use “"decomposePar -latestTime".

Here my decomposePar log:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.2.1                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 
Exec   : decomposePar -latestTime
Date   : 
Time   : 
Host   : 
PID    : 
Case   : 
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Disallowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time



Decomposing mesh region0

Create mesh

Calculating distribution of cells
Selecting decompositionMethod scotch

Finished decomposition in 61.54 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Distributing points to processors

Constructing processor meshes

Processor 0
    Number of cells = 156676
    Number of faces shared with processor 1 = 1463
    Number of faces shared with processor 3 = 2235
    Number of faces shared with processor 4 = 99
    Number of faces shared with processor 5 = 1753
    Number of faces shared with processor 7 = 2890
    Number of faces shared with processor 12 = 1392
    Number of faces shared with processor 13 = 2912
    Number of faces shared with processor 15 = 28
    Number of faces shared with processor 29 = 1119
    Number of faces shared with processor 30 = 43
    Number of faces shared with processor 33 = 752
    Number of faces shared with processor 43 = 1663
    Number of faces shared with processor 44 = 277
    Number of processor patches = 13
    Number of processor faces = 16626
    Number of boundary faces = 3973

Processor 1
    Number of cells = 155343
    Number of faces shared with processor 0 = 1463
    Number of faces shared with processor 2 = 4228
    Number of faces shared with processor 3 = 684
    Number of faces shared with processor 4 = 492
    Number of faces shared with processor 5 = 61
    Number of faces shared with processor 12 = 3211
    Number of faces shared with processor 15 = 732
    Number of faces shared with processor 17 = 113
    Number of processor patches = 8
    Number of processor faces = 10984
    Number of boundary faces = 5711

% [..] The same for every processor
Processor 47
    Number of cells = 154530
    Number of faces shared with processor 43 = 89
    Number of faces shared with processor 44 = 114
    Number of faces shared with processor 46 = 1413
    Number of processor patches = 3
    Number of processor faces = 1616
    Number of boundary faces = 62009

Number of processor faces = 250893
Max number of cells = 156913 (1.15258% above average 155125)
Max number of processor patches = 17 (102.985% above average 8.375)
Max number of faces between processors = 16626 (59.0415% above average 10453.9)

Time = 0

%it stops here...

Here my decomposePar dict :

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 48;//4

method          scotch;//simple


simpleCoeffs
{
    n               (2 2 1);
    delta           0.001;
}

hierarchicalCoeffs
{
    n               (1 1 1);
    delta           0.001;
    order           xyz;
}

manualCoeffs
{
    dataFile        "cellDecomposition";
}

metisCoeffs
{
    //n                   (5 1 1);
    //cellWeightsFile     "constant/cellWeightsFile";
}


// ************************************************************************* //

And here the output from the cluster :

### auto-loading modules openmpi/1.6.3--gnu--4.7.2
### auto-loading modules gnu/4.7.2


--> FOAM FATAL IO ERROR: 
Cannot find patchField entry for train_CATIASTL

file: myCase/0/p.boundaryField from line 26 to line 64.

    From function GeometricField<Type, PatchField, GeoMesh>::GeometricBoundaryField::readField(const DimensionedField<Type, GeoMesh>&, const dictionary&)
    in file/prod/build/applications/openfoam/2.2.1-gnu-4.7.2/openmpi--1.6.3--gnu--4.7.2/BA_WORK/OpenFOAM-2.2.1/src/OpenFOAM/lnInclude/GeometricBoundaryField.C at line 198.

FOAM exiting

--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:              node119
  Registerable memory:     32768 MiB
  Total memory:            48434 MiB

Your MPI job will continue, but may be behave poorly and/or hang.
--------------------------------------------------------------------------
[0] 
[0] 
[0] --> FOAM FATAL IO ERROR: 
[0] cannot find file
[0] 
[0] file: myCase/processor0/0/p at line 0.
[0] 
[0]     From function regIOobject::readStream()
[0]     in file db/regIOobject/regIOobjectRead.C at line 73.
[0] 
FOAM parallel run exiting
[0] 

%[...]
% The same for every processor
%[...]

[9] 
[9] 
[9] --> FOAM FATAL IO ERROR: 
[9] cannot find file
[9] 
[9] file: myCase/processor9/0/p at line 0.
[9] 
[9]     From function regIOobject::readStream()
[9]     in file db/regIOobject/regIOobjectRead.C at line 73.
[9] 
FOAM parallel run exiting
[9] 
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 11 in communicator MPI_COMM_WORLD 
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 24 with PID 5991 on
node node128ib0 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[node119:09393] 47 more processes have sent help message help-mpi-btl-openib.txt / reg mem limit low
[node119:09393] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[node119:09393] 47 more processes have sent help message help-mpi-api.txt / mpi-abort


--> FOAM FATAL ERROR: 
No times selected

    From function reconstructPar
    in file reconstructPar.C at line 178.

FOAM exiting

If you need something else please just ask me
Thanks!

[ryanc6]

Hello Jeanp,


I am experiencing the same problem. My decomposePar does not write the field data to the processor* folders. DecomposePar runs until it reaches the "Time = 0" line. (same as yours) Then I get the exact same errors.



Did you ever find a solution to this problem?



Thanks,
ryanc6

[ryanc6]

This morning I was able to run decomposePar correctly. I made no changes to the decomposeParDict or other system files. I did however rerun my mesh generation. I used blockMesh along with refineMesh and snappyHexMesh. I did not change any of the mesh generation dicts; I simply cleaned it all and ran again. CheckMesh passed and everything looked good on both mesh generation accounts so I'm not sure what the problem was. I hope this helps anyone else with this problem.

[Hughtong]

I searched this problem during the decompose process was stucked in the "Time =0 " in the cluster. But after about 30 minutes, it starts to "Processor *: field transfer" XD