[marco.müller]

Hi,

I'm using SHM for OF 2.1.1 out of box on a VirtualBox machine on Windows 7 with 48 GB RAM and 8 physical cores. 40 GB and all cores are enabled for VirtualBox.

My model runs fine with 2 or 4 cores. When I switch to 8 cores SHM aborts (while not even 10% RAM are used) randomly like this:

Shell refinement iteration 1
----------------------------

Marked for refinement due to refinement shells    : 10644 cells.
Determined cells to refine in = 2.02 s
Selected for internal refinement : 39957 cells (out of 1056197)
(5): ERROR: dgraphFold2: out of memory (2)
(4): ERROR: dgraphFold2: out of memory (2)
--------------------------------------------------------------------------
mpirun has exited due to process rank 5 with PID 3976 on

I cannot even find this error in the forum. Any hints?

Thanks
Marco

[wyldckat]

Hi Marco,

A few questions:

1. Which decomposition method are you using while snappyHexMesh is running?
2. Which OpenFOAM version are you using?
3. What Linux distribution and architecture are you using?

Best regards,
Bruno

[marco.müller]

Hi Bruno,

thanks for your answer.

1. I tried ptscotch. with simple there is no such error. So finally this is an acceptable solution.
2. 2.1.1.
3. the ubuntu version this OF version was built on officially.

Marco

[wyldckat]

Hi Marco,

Which architecture of Ubuntu? i386/i686 or x86_64/amd64? You can check by running:

uname -m

If you are using i386/i686, it's only natural that it crashes when it reaches 2GiB of RAM per process.

Best regards,
Bruno

[caduqued]

Well, now I have encountered the same problem. Trying to get the meshing with snappyHexMesh, I got:

[QUOTE]...
Feature refinement iteration 24
------------------------------

Marked for refinement due to explicit features : 92 cells.
Determined cells to refine in = 13.28 s
Selected for feature refinement : 136 cells (out of 10647165)
Edge intersection testing:
Number of edges : 36562736
Number of edges to retest : 8226
Number of intersected edges : 4230508
Refined mesh in = 2.76 s
After refinement feature refinement iteration 24 : cells:10648117 faces:36562736 points:15512183
Cells per refinement level:
0 1149402
1 228530
2 967512
3 2859801
4 5442872
(14): ERROR: dgraphFold2: out of memory (2)
(13): (16): ERROR: (9): (15): ERROR: dgraphFold2: out of memory (3)
dgraphFold2: out of memory (2)ERROR: [17]
dgraphFold2: out of memory (2)ERROR:
dgraphFold2: out of memory (2)#[11] #
0 0 Foam::error: printStack(Foam::Ostream&)Foam::error: printStack(Foam::Ostream&)--------------------------------------------------------------------------
mpirun has exited due to process rank 9 with PID 23598 on
...
[/QUOTE]

The only available information seems to be related with the scotch library, but not much info really.

In my case, I am running snappyHexMesh in a cluster using 64bit (x86_64), 36 cores, and requesting 4GB per core (way too decent for this type of mesh)... and still... ohh surprise, out of memory problem. Does anyone know how to solve this?

Regards,

[wyldckat]

Greetings caduqued,

If you ran in parallel, then the library "ptscotch" does have some limitations.

By the way, which OpenFOAM version are you using? Because versions older than OpenFOAM 2.2.0 are using Scotch 5.1.1; while as of OpenFOAM 2.2.0, it uses Scotch 6.0.0, which is allegedly (missing reference ) a lot better!

Best regards,
Bruno

[caduqued]

Hi Bruno,

Thanks for your quick response. Yes, I am now using OpenFOAM 2.2.0, with (in fact) scotch 2.0.0. The parallelization method I am using is scotch (not ptscotch, as it used to be before), but I assume that is irrelevant, as it is just the name of the method (library being always scotch). With this hindsight, I can now (sadly !) confirm that it seems this new scotch version also exhibits some problems, although I must confess didn't know about them. So, in the meantime, waiting for a fix, I assume it will be safer to just stick to one of the traditional methods, right?

Regards,

Carlos

[wyldckat]

Hi Carlos,

I forgot that they did rename "ptscotch" to "scotch" when used in parallel, so that the same dictionary could be used for the decomposition.

The only other thing I can remember about is the "multilevel" decomposition strategy... I mentioned it some time ago here: http://www.cfd-online.com/Forums/ope...tml#post367979 - post #8. It's not the silver bullet, but if used correctly, it might help to get the best of both worlds...

Best regards,
Bruno

[caduqued]

Originally Posted by wyldckat

Hi Carlos,

I forgot that they did rename "ptscotch" to "scotch" when used in parallel, so that the same dictionary could be used for the decomposition.

The only other thing I can remember about is the "multilevel" decomposition strategy... I mentioned it some time ago here: http://www.cfd-online.com/Forums/ope...tml#post367979 - post #8. It's not the silver bullet, but if used correctly, it might help to get the best of both worlds...

Best regards,
Bruno

Hi Bruno,

Thanks again. I did not know about the multilevel strategy, thanks a lot for bring my attention to it!!! (with openFOAM every day is a learning day!!!)

I will try it...

Regards,

Carlos

[Slanth]

Hello, caduqued, about the dgraphFold2: out of memory error, have you solved it? I met this problem and couldn't find a solution.

[wyldckat]

Greetings Slanth,

You'll have to provide more specific details about the case you're meshing, because the error message:

dgraphFold2: out of memory error

can occur for a lot of reasons.
It is even possible that the problem is:

1. With your base mesh.
2. With the decomposition method being used.
3. With how many cores being used.
4. With which OpenFOAM version you're using.

Best regards,
Bruno

[capucsc]

I likewise encounter this error frequently. I'm meshing on a 32 core Opteron machine. I used 'scotch' for decomposition and I work with OF2.3.0. I get messages like:

(16): ERROR: dgraphFold2: out of memory (2)
(17): ERROR: dgraphFold2: out of memory (2)
(28): (29): ERROR: dgraphFold2: out of memory (2)
(31): ERROR: dgraphFold2: out of memory (2)
ERROR: dgraphFold2: out of memory (2)

about 4 out of 10 runs. I've watched this happen with top running, and as other commented have noted, there isn't any issue with available RAM. It would be great to work out a solution, as this is the most unreliable aspect of using OF.

Other details:
uname -a:

Linux sim2 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

ldd /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/bin/snappyHexMesh:

linux-vdso.so.1 => (0x00007fff095b1000)
libfiniteVolume.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libfiniteVolume.so (0x00007fefdfe65000)
libdecompositionMethods.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libdecompositionMethods.so (0x00007fefdfc10000)
libptscotchDecomp.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5/libptscotchDecomp.so (0x00007fefdfa01000)
libmeshTools.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libmeshTools.so (0x00007fefdf461000)
libsurfMesh.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libsurfMesh.so (0x00007fefdf168000)
libfileFormats.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libfileFormats.so (0x00007fefdeef0000)
libdynamicMesh.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libdynamicMesh.so (0x00007fefde962000)
libautoMesh.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libautoMesh.so (0x00007fefde559000)
libOpenFOAM.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so (0x00007fefddc35000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fefdda16000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fefdd712000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fefdd40c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fefdd1f5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fefdce2f000)
libPstream.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5/libPstream.so (0x00007fefdcc21000)
libtriSurface.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libtriSurface.so (0x00007fefdc98e000)
libptscotch.so => /home/someuser/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5/libptscotch.so (0x00007fefdc742000)
libptscotcherrexit.so => /home/someuser/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5/libptscotcherrexit.so (0x00007fefdc53e000)
libscotch.so => /home/someuser/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5/libscotch.so (0x00007fefdc2b5000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fefdc0ad000)
libextrudeModel.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libextrudeModel.so (0x00007fefdbe96000)
liblagrangian.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/liblagrangian.so (0x00007fefdbc77000)
libedgeMesh.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libedgeMesh.so (0x00007fefdba0e000)
libdistributed.so => /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/libdistributed.so (0x00007fefdb7ba000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fefdb5a0000)
/lib64/ld-linux-x86-64.so.2 (0x00007fefe128c000)
libmpi.so.1 => /home/someuser/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/openmpi-1.6.5/lib64/libmpi.so.1 (0x00007fefdb1f8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fefdafd9000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fefdadd6000)

[wyldckat]

Greetings capucsc,

Originally Posted by capucsc

I likewise encounter this error frequently. I'm meshing on a 32 core Opteron machine. I used 'scotch' for decomposition and I work with OF2.3.0. I get messages like:
[...]
about 4 out of 10 runs.

You're able to reproduce this error a lot more than I was expecting
Any chance you have one of those examples that have this issue in a small file/case size and be allowed to share it with us, either publicly or privately? This way it would make it a lot easier to try and diagnose the source of this problem!

Best regards,
Bruno

[capucsc]

Hi Bruno,

The cases can be found here:

https://www.dropbox.com/s/4u4jk2ps6o...es.tar.gz?dl=0

Thanks for taking a look!
-Chris

[GRAUPS]

Gentlemen,

I'm also seeing this same issue on a meshing case every once in a while. I'm on an Intel 16 core machine using OF 2.3.x and the scotch decomposition. Unfortunately, I don't have a case I can share at the moment, but please do let me know if a public bug report gets created for this so that I can keep an eye on it. If I come across a case I can share, I will do so!

Thanks!

Brock

[capucsc]

I believe, although I have little in way of evidence, that the occurrence of this error is tied to the number of cores used. I never encounter problems when running with 6 cores, but about 70% of runs fail when using 32 cores. If we were to make a bug report, where would it go? OpenFOAM or scotch or somewhere else?

[GRAUPS]

The main place to report bugs for openfoam is here...

http://www.openfoam.org/mantisbt/

But if you report something, make sure you give the developer everything he needs to fix or at least diagnose the problem. It's best if you can isolate the problem down to a small model and then give them that model to debug with. I've had some good luck in reporting, one of the bugs I reported was fixed within 24 hrs!

Brock

[simpomann]

Hey,
I encountered the same problem today while also using scotch decomposition with 32 processors on a 64bit cluster headnode. Before, I did a similiar case with the same geometry, but I split up the .stl-files and now I get this error every time.

Case is running if i switch to 27 cores though.

(16): ERROR: dgraphFold2: out of memory (2)
(17): ERROR: dgraphFold2: out of memory (2)
(28): (29): ERROR: dgraphFold2: out of memory (2)
(31): ERROR: dgraphFold2: out of memory (2)
ERROR: dgraphFold2: out of memory (2)

Is there a solution somewhere right now? I looked through the bug tracker but couldnt find a ticket.

[capucsc]

I have yet to pinpoint the exact reason this happens, so I haven't submitted a bug report. Debugging this turns out to be rather tricky. I just use fewer processors to mesh.

[Slanth]

I have tried many times and finally walk around. At first, you should use surfaceCheck utility to make sure your stl file is closed and does not have any unconneceted parts. Even the total stl file has been decomposed into several parts, it should be closed when combining all of them.