[Orgogozo]

Dear foamers,

I am trying to build up a mesh of about 600 millions of cells (3396*3396*50) and when I run blockMesh I get the following error :


"
Creating cells
Creating points with scale 1

--> FOAM FATAL ERROR:
bad set size -835122496

From function List<T>::setSize(const label)
in file /usr/local/OpenFoam/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/List.C at line 322.

FOAM aborting
"

The procedure I use work for the building of a four time smaller mesh, of about 150 millions of cells (1698*1698*50), but give the error above for the larger case.

Does anyone have an idea of what should be done to solve this problem ? Thanks by advance for any help.

Laurent

[jans]

Hi Laurent,
I just encountered the same error. Were you able to overcome this problem?

My guess is that this is a case of overflow in a variable of type int. For example, for large meshes, the variables holding the number of points, faces, cells etc are susceptible to overflow. The solution may be to switch to "long int" when this happens.

@ OpenFOAM developers: Should this be submitted as a bug report?

Thanks
Anirban

[Abracurcix]

Hello Laurent,
This questions of yours appears to have not gotten a response from any of the other fellow FOAMers. Did you manage to solve this problem? The problem seems to be a bit more involved than merely promoting "int" to "long int" in the label.H header file in primitives/ints/label (by defining FOAM_LABEL64 or setting FOAM_LABEL_MAX to a value greater than 2,000,000,000).

Cheers,
Albert

Originally Posted by Orgogozo

Dear foamers,

I am trying to build up a mesh of about 600 millions of cells (3396*3396*50) and when I run blockMesh I get the following error :


"
Creating cells
Creating points with scale 1

--> FOAM FATAL ERROR:
bad set size -835122496

From function List<T>::setSize(const label)
in file /usr/local/OpenFoam/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/List.C at line 322.

FOAM aborting
"

The procedure I use work for the building of a four time smaller mesh, of about 150 millions of cells (1698*1698*50), but give the error above for the larger case.

Does anyone have an idea of what should be done to solve this problem ? Thanks by advance for any help.

Laurent

[Orgogozo]

Dear Jans, Dear Abracurcix,

I didn't manage to slve this problem yet ; instead I dealt with smaller meshes. However I will try again soon so thank you for your threads I'll look cqrefully about this issue of long int. I'll post a threads as soon as (hopefully) i succed in working with lqrger meshes.

Best regards,

Laurent

[wyldckat]

Greetings to all!

If my memory doesn't fail me, the estimate is that 3396*3396*50 cells will require around 575 GB of RAM! A machine to test this is rather difficult to come by for testing such a big case. Let alone the time it takes to run in a single core for generating it...

Nonetheless, if your machines do have that amount of RAM or more, I do strongly suggest that you report this on the bug tracker: http://www.openfoam.org/bugs/

Best regards,
Bruno

[Orgogozo]

Dear wyldckat,

I do use a fat node with 2TB RAM. But i encountered this bug with an old version of OF (2.0.1 I think), so I'll try again with 2.2 in the following days and then do a bug reports if I still get the same error.

Best regards,

Laurent

[wyldckat]

Greetings to all!

I've given this a try and I found out that there are several pieces of source code in OpenFOAM that are not prepared for "FOAM_LABEL64"

The steps I've taken were:

1. Edit the file "wmake/rules/linux64Gcc/c++" and added:

   -DFOAM_LABEL64

   to the end of the line "ptFLAGS".
2. Run:

   foam
   wcleanAll

3. Then run (for using 6 cores):

   export WM_NCOMPPROCS=6
   ./Allwmake > make.log 2>&1

And the library "surfMesh" was only the tip of the iceberg Then came "meshTools", "scotchDecomp", "ptscotchDecomp" and "autoMesh", if I'm not mistaken.

So, once again, since you guys have the machines with tons of RAM and are interested in the solution, please report this on the bug tracker: http://www.openfoam.org/bugs/

Best regards,
Bruno

[Orgogozo]

Dear all,

The bug is reported (ID 0000903 on the mantis).

Thank you all for your answers

Cheers

Laurent

[wyldckat]

Greetings to all!

I haven't had the time and opportunity to look into this myself, but in the following bug report there are some instructions that might help: http://www.openfoam.org/mantisbt/view.php?id=1033

By the way, it's possible that OpenCFD won't fix the problem unless someone sponsors the fix and subsequent testing.

Best regards,
Bruno

[mchurchf]

Laurent,

Were you ever able to build very large meshes, or successfully compile with -DFOAM_LABEL64?

[Orgogozo]

Matthew,

I checked again with the version 2.0.x of OpenFOAM : it still does not work. With parallel meshing however one can build up relatively large meshes, with 100 millions number of mesh cells or more. See for example the following paper :
http://authors.elsevier.com/a/1PsPx2OInFfaV

Best regards,

Laurent

[andrea.pasquali]

Dear all,

I found the same problem with the integer size in OpenFOAM.
I recompiled OpenFOAM with the extension "-DFOAM_LABEL64" checking and correcting the pieces of code where the compilation gives me error.
Now it works but I noticed that OpenFOAM in 64 requires 30% memory more than in 32 (running the same application with the same grid size).
Should I check something else during the compilation?
Or are there any other suggestions?

Best regards
Andrea

[wyldckat]

Greetings Andrea,

Originally Posted by andrea.pasquali

Now it works but I noticed that OpenFOAM in 64 requires 30% memory more than in 32 (running the same application with the same grid size).

I believe you've tripped over the reason why OpenFOAM isn't built by default with "FOAM_LABEL64" turned on and why it's not being very well maintained.
The problem of using 64-bit "labels" is that this means that in fact it uses 64-bit integers, instead of 32-bit integers. This means that all integers used in the code will use up 8 bytes per value, instead of 4 bytes. Therefore, consider yourself lucky it's only consuming more 30% RAM and not 50%

One solution for this would require diagnosing all parts of the code and to isolate the situations where "2^31 = 2147483648" items (both negative and positive) is more than enough, i.e. where more than 2 billion cells/faces/points/hash lists are necessary. Also keep in mind that this can scale up to needing matrices for the equations with roughly as many entries over at least 2 dimensions...
Keep in mind that OpenFOAM relies heavily on indexing points, cells and faces, hence using all labels at the same bit-length.

Another solution would be to rely on hybrid mathematical libraries, such as GMP: https://gmplib.org/ - in theory, it could be able to handle 48-bit mathematics, which would equate to "2^47 = 1.407374884×10¹⁴" items. problem is that storage and access to 48-bit data isn't very "standard", which means that it would possibly require some strong efforts in having an intermediate interpretation layer between OpenFOAM's way of doing things and GMP's way of doing things. There was some discussion about this sometime ago... let me check... here you go: http://www.cfd-online.com/Forums/ope...precision.html - although that was more focused on the floating precision and not integer precision...

The bottom line is: if you need to go into the 64-bit realm, be sure to have enough RAM to pay the price
The other solution would be to handle your mesh in parallel, so that each partition doesn't go over the 32-bit limit.

Best regards,
Bruno

[wyldckat]

Greetings to all and a Happy New Year!

For those who aren't keeping track of the bug report, it has been fixed yesterday!

• Bug report: http://www.openfoam.org/mantisbt/view.php?id=1033
• The specific commit that fixed it: https://github.com/OpenFOAM/OpenFOAM...5452f909e23ffa

This change will unlikely make it's way to OpenFOAM 2.3, since this is a pretty major new feature... and fix... completion of a feature

Using this repository should roughly be used similarly to the git repository 2.3.x.

Best regards,
Bruno

[Orgogozo]

Dear all,

I have just tested the building of a mesh of ~600 millions cells with the Dev version of OpenFOAM on the fat node of CALMIP* (UVPROD, 2TB RAM) and it works fine. I will make a more detail return on this issue later on this year.

Greetings,

*http://www.calmip.univ-toulouse.fr

[Orgogozo]

Dear all,

With the development version of OpenFoam*, we have managed to make a scaling curve from 400 to 3200 cores on a case with a 1.2 billion cells mesh. We experienced super-linearity until 3200 cores.

The runs have been done on EOS**, the new cluster of CALMIP. The building and the decomposition of the mesh were performed on an UVPROD cluster, with ~3TB RAM avalaible (the requests were using maximum 1.5 TB requests, i.e. half of the machine).

This result has been presented in the beginning of may 2015 at the Maison de la Simulation in Saclay, France. You may find at the url below the pdf of this seminar (billion mesh cells scaling results : slides 20-22).

http://www.maisondelasimulation.fr/en/semmod.php?a
(scroll to may 2015 to find the presentation)

Cheers,

Laurent


*https://github.com/OpenFOAM/OpenFOAM-dev

**http://www.calmip.univ-toulouse.fr/s...php?article388

[Saeb-sahand]

To resolve the problem, simply you could set geometry of domain to involve negative values further positive ones. For instance, set the mesh box to start from (-10,-10,1-0) to (100,100,100).

***This may deals with problem***

/*--------------------------------*- C++ -*----------------------------------*\
========= |
\\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\\ / O peration | Website: https://openfoam.org
\\ / A nd | Version: 10
\\/ M anipulation |
\*---------------------------------------------------------------------------*/
FoamFile
{
format ascii;
class dictionary;
object blockMeshDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

convertToMeters 1;


x1 20;
x2 150;
x3 22;
x4 20;
x5 60;
x6 50;

h1 100; // elevation of the upper patch
h2 3;
h3 25;
h4 5;

d 100; // d is the depth in y direction normol to the plan


vertices
(

name v0 ( 0 0 0 ) // 0 node [QUOTE][/this locate box first vertices 0n (0,0,0) leads the problem ]
name v1 ( $x1 0 0) // 1 node
name v2 ( $x1 $h1 0) // 2 node
name v3 ( 0 $h1 0) // 3 node
name v4 ( #calc"$x1+$x2" $h2 0) // 4 node
name v5 ( #calc"$x1+$x2" $h1 0) // 5 node


name v6 ( #calc"$x1+$x2+$x3" $h3 0) // 6 node
name v7 ( #calc"$x1+$x2+$x3" $h1 0) // 7 node

name v8 ( #calc"$x1+$x2+$x3+$x4" $h3 0) // 8 node
name v9 ( #calc"$x1+$x2+$x3+$x4" $h1 0) // 9 node

name v10 ( #calc"$x1+$x2+$x3+$x4+$x5" $h4 0) // 10 node
name v11 ( #calc"$x1+$x2+$x3+$x4+$x5" $h1 0) // 11 node

name v12 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h4 0) // 12 node
name v13 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h1 0) // 13 node



name v14 ( 0 0 $d) // 14 node
name v15 ( $x1 0 $d) // 15 node
name v16 ( $x1 $h1 $d) // 16 node
name v17 ( 0 $h1 $d) // 17 node
name v18 ( #calc"$x1+$x2" $h2 $d) // 18 node
name v19 ( #calc"$x1+$x2" $h1 $d) // 19 node


name v20 ( #calc"$x1+$x2+$x3" $h3 $d) // 20 node
name v21 ( #calc"$x1+$x2+$x3" $h1 $d) // 21 node

name v22 ( #calc"$x1+$x2+$x3+$x4" $h3 $d) // 22 node
name v23 ( #calc"$x1+$x2+$x3+$x4" $h1 $d) // 23 node

name v24 ( #calc"$x1+$x2+$x3+$x4+$x5" $h4 $d) // 24 node
name v25 ( #calc"$x1+$x2+$x3+$x4+$x5" $h1 $d) // 25 node

name v26 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h4 $d) // 26 node
name v27 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h1 $d) // 27 node


);

blocks
(

hex (v0 v1 v2 v3 v14 v15 v16 v17) (5 20 20) simpleGrading (1 1 1) //block1
hex (v1 v4 v5 v2 v15 v18 v19 v16) (20 20 20) simpleGrading (1 1 1) //block2
hex (v4 v6 v7 v5 v18 v20 v21 v19) (5 20 20) simpleGrading (1 1 1) //block3
hex (v6 v8 v9 v7 v20 v22 v23 v21) (5 20 20) simpleGrading (1 1 1) //block4
hex (v8 v10 v11 v9 v22 v24 v25 v23) (10 20 20) simpleGrading (1 1 1) //block3
hex (v10 v12 v13 v11 v24 v26 v27 v25) (10 20 20) simpleGrading (1 1 1) //block4

);

boundary
(
front
{
type patch;
faces
(
(v0 v1 v2 v3)
(v1 v4 v5 v2)
(v4 v6 v7 v5)
(v6 v8 v9 v7)
(v8 v10 v11 v9)
(v10 v12 v13 v11)
);
}


Back
{
type patch;
faces
(
(v14 v15 v16 v17)
(v15 v18 v19 v16)
(v18 v20 v21 v19)
(v20 v22 v23 v21)
(v22 v24 v25 v23)
(v24 v26 v27 v25)
);
}

inlet
{
type patch;
faces
(
(v0 v3 v17 v14)
);
}
outlet
{
type patch;
faces
(
(v13 v12 v26 v27)
);
}

ground
{
type wall;
faces
(
(v0 v1 v15 v14)
(v1 v4 v18 v15)
(v4 v6 v20 v18)
(v6 v8 v22 v20)
(v8 v10 v24 v22)
(v10 v12 v26 v24)
);
}
upperPatch
{
type patch;
faces
(
(v2 v16 v17 v3)
(v2 v5 v19 v16)
(v5 v7 v21 v19)
(v7 v9 v23 v21)
(v9 v11 v25 v23)
(v11 v13 v27 v25)

);
}
);

// ************************************************** *********************** //


but

*** This may solve the problem ***

/*--------------------------------*- C++ -*----------------------------------*\
========= |
\\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\\ / O peration | Website: https://openfoam.org
\\ / A nd | Version: 10
\\/ M anipulation |
\*---------------------------------------------------------------------------*/
FoamFile
{
format ascii;
class dictionary;
object blockMeshDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

convertToMeters 1;

x_0 -10;
y_0 -10;

x1 #calc"20+$x_0"; //20;
x2 150;
x3 22;
x4 20;
x5 60;
x6 50;

h1 #calc"100+$y_0"; //100; // elevation of the upper patch
h2 #calc"3+$y_0"; //3;
h3 #calc"25+$y_0"; //25;
h4 #calc"5+$y_0"; //5;

d 50; //100; // d is the depth in y direction normol to the plan


vertices
(

name v0 ($x_0 $y_0 0 ) // 0 node this solve the problem x_0 -10; y_0 -10;
name v1 ( $x1 $y_0 0) // 1 node
name v2 ( $x1 $h1 0) // 2 node
name v3 ( $x_0 $h1 0) // 3 node
name v4 ( #calc"$x1+$x2" $h2 0) // 4 node
name v5 ( #calc"$x1+$x2" $h1 0) // 5 node


name v6 ( #calc"$x1+$x2+$x3" $h3 0) // 6 node
name v7 ( #calc"$x1+$x2+$x3" $h1 0) // 7 node

name v8 ( #calc"$x1+$x2+$x3+$x4" $h3 0) // 8 node
name v9 ( #calc"$x1+$x2+$x3+$x4" $h1 0) // 9 node

name v10 ( #calc"$x1+$x2+$x3+$x4+$x5" $h4 0) // 10 node
name v11 ( #calc"$x1+$x2+$x3+$x4+$x5" $h1 0) // 11 node

name v12 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h4 0) // 12 node
name v13 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h1 0) // 13 node



name v14 ( $x_0 $y_0 $d) // 14 node
name v15 ( $x1 $y_0 $d) // 15 node
name v16 ( $x1 $h1 $d) // 16 node
name v17 ( $x_0 $h1 $d) // 17 node
name v18 ( #calc"$x1+$x2" $h2 $d) // 18 node
name v19 ( #calc"$x1+$x2" $h1 $d) // 19 node


name v20 ( #calc"$x1+$x2+$x3" $h3 $d) // 20 node
name v21 ( #calc"$x1+$x2+$x3" $h1 $d) // 21 node

name v22 ( #calc"$x1+$x2+$x3+$x4" $h3 $d) // 22 node
name v23 ( #calc"$x1+$x2+$x3+$x4" $h1 $d) // 23 node

name v24 ( #calc"$x1+$x2+$x3+$x4+$x5" $h4 $d) // 24 node
name v25 ( #calc"$x1+$x2+$x3+$x4+$x5" $h1 $d) // 25 node

name v26 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h4 $d) // 26 node
name v27 ( #calc"$x1+$x2+$x3+$x4+$x5+$x6" $h1 $d) // 27 node


);

blocks
(

hex (v0 v1 v2 v3 v14 v15 v16 v17) (5 20 20) simpleGrading (1 1 1) //block1
hex (v1 v4 v5 v2 v15 v18 v19 v16) (20 20 20) simpleGrading (1 1 1) //block2
hex (v4 v6 v7 v5 v18 v20 v21 v19) (5 20 20) simpleGrading (1 1 1) //block3
hex (v6 v8 v9 v7 v20 v22 v23 v21) (5 20 20) simpleGrading (1 1 1) //block4
hex (v8 v10 v11 v9 v22 v24 v25 v23) (10 20 20) simpleGrading (1 1 1) //block3
hex (v10 v12 v13 v11 v24 v26 v27 v25) (10 20 20) simpleGrading (1 1 1) //block4

);

boundary
(
front
{
type patch;
faces
(
(v0 v1 v2 v3)
(v1 v4 v5 v2)
(v4 v6 v7 v5)
(v6 v8 v9 v7)
(v8 v10 v11 v9)
(v10 v12 v13 v11)
);
}


Back
{
type patch;
faces
(
(v14 v15 v16 v17)
(v15 v18 v19 v16)
(v18 v20 v21 v19)
(v20 v22 v23 v21)
(v22 v24 v25 v23)
(v24 v26 v27 v25)
);
}

inlet
{
type patch;
faces
(
(v0 v3 v17 v14)
);
}
outlet
{
type patch;
faces
(
(v13 v12 v26 v27)
);
}

ground
{
type wall;
faces
(
(v0 v1 v15 v14)
(v1 v4 v18 v15)
(v4 v6 v20 v18)
(v6 v8 v22 v20)
(v8 v10 v24 v22)
(v10 v12 v26 v24)
);
}
upperPatch
{
type patch;
faces
(
(v2 v16 v17 v3)
(v2 v5 v19 v16)
(v5 v7 v21 v19)
(v7 v9 v23 v21)
(v9 v11 v25 v23)
(v11 v13 v27 v25)

);
}
);

// ************************************************** *********************** //

in the second blockMeshDict file

x_0 -10;
y_0 -10;


first vertices start name v14 ( $x_0 $y_0 $d)