[Ship Designer]

I let a simulation run overnight and the next day I saw that it was aborted for no apparent reason other than this message some time after it started:

[…]

GAMG:  Solving for p_rgh, Initial residual = 0.114596, Final residual = 0.000297992, No Iterations 2
GAMG:  Solving for p_rgh, Initial residual = 0.000313041, Final residual = 2.35967e-06, No Iterations 7
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node 701f164b7b63 exited on signal 9 (Killed).
--------------------------------------------------------------------------

I use OpenFOAM v8 on a Mac with Docker and at the time, Docker was set to 8 GB of memory for containers. When I checked memory usage in macOS' Activity Monitor, it reported approx 600 MB for the com.docker.hyperkit process. This is the value I usually see when running similar computations and I wondered more than once, how OpenFOAM can use so little memory for cases with > 1M cells…

I restarted the computation from the last saved time step and it was killed again. Since this mesh had more cells then I usually deal with (~2.7M) I increased the setting from 8 GB to 12 GB in Docker. It took longer this time but it was killed again eventually. Confused by the much lower used memory reported by Activity Monitor, I decided that I needed a way to monitor the memory usage from within the container while the computation si running. I added this function object to controlDict

FreeMemorySystemCall {
             type  systemCall;
             libs  ("libutilityFunctionObjects.so");

     executeCalls  ();
       writeCalls  ("free");
         endCalls  ();
     writeControl  timeStep;
    writeInterval  10;
 }

which would output something like this every 10 time steps (values are in kB):

              total        used        free      shared  buff/cache   available
Mem:       12268828     7852356     3653360         768      763112     4214616
Swap:       2097148           0     2097148

I didn't analize the complete logs yet, but so far I have seen increases in used memory of anything between 0 and up to 240 MB every 10 time steps. The used memory keeps increasing until ~1 GB of memory is left available, at which point the used memory in Activity Monitor starts to increase. After a while though, interFoam gets killed again.

Obviously interFoam has to store in memory all the fields and other data at least until it writes them to permanent storage. The steady increase in memory usage however suggests that data is stored much longer than that or that this is some sort of memory leakage. I would assume that since the mesh is static and the simulation of a pseudo steady state type, the total amount of average data across time steps shouldn't increase. I can't think of any runtime-generated data — in such quantities at least — that would be needed to be accumulated in memory for the entire duration of the computation, or is there?

Is this behaviour intentional or is this an indication of memory-related bug?

If interFoam is stopped and restarted again (without restarting Docker), memory usage is reset (aaumulation disappears). Memory usage right after starting the OpenFOAM container is ~350 MB. After starting interFoam, it jumps to several GB which seems reasonable (depending on mesh size) but then keeps steadily increasing.

Some background info about the case: it is a ship hull resistance calculation whit static mesh, ~2.7M cells, solved by interFoam in parallel mode and local time stepping. RAS is used with turbulence switched off. Simulation and field values progress as expected with clear tendency to converge. Function objects used are forces, fieldMinMax and yPlus.

For those not familiar with macOS, Activity Monitor is the built in utility that reports the resources usage of the system (CPU, Memory, Energy etc.). I'm sure that the heavily underreported memory used by the container has its reasons is not an OpenFOAM issue. It rather misled me to believe that running OpenFOAM is not memory intense, also because I never encountered this problem before (killed siumlation due to low memory).

I would be grateful for any hints, thanks! Have a nice weekend,

Claudio

[HPE]

Hi,

- is it possible for you to run `Massif` on your application and share the output here (if you know how to use it)?
- or a tutorial case with the same settings that reproduces the issue?
- can you comment out the function objects and rerun the simulation? (please comment them out instead of using `enabled no;` keyword pair).

[Ship Designer]

Thank you HPE for your quick reply. I have to look into "Massif" as I'm not familiar with it. If I manage to get it to work I'll be happy to share any results. I'm currently doing computations for a project and will be able to make some test calculations in a couple of days time. As you suggested I commented out most function objects except forces and fieldMinMax which are necessary and the problem persists. I'll get back to you as soon as I can run some test computations.

Cheers, Claudio

[Ship Designer]

Just a quick update, I was able to narrow down the problem to the div(phi,alpha) Gauss MPLIC scheme that I used as the cause of the increasing memory usage over the course of the computation. Since both MPLIC and PLIC were inroduced with OpenFOAM v8, I did a run with PLIC as well. Below is the memory usage of the v8 DTC Hull tutorial case found in /tutorials/multiphase/interFoam/RAS/DTCHull running in parallel on 4 cores:

           Scheme  div(phi,alpha) Gauss interfaceCompression vanLeer 1 (original of tutorial)
Memory usage [kB]  
      at Time   0  1651880 
      at Time  10  2077792
      at Time 500  2043216


           Scheme  div(phi,alpha) Gauss PLIC interfaceCompression vanLeer 1
Memory usage [kB]  
      at Time   0  1677056 
      at Time  10  2577572
      at Time 500  4120516


           Scheme  div(phi,alpha) Gauss MPLIC
Memory usage [kB]  
      at Time   0  1677184 
      at Time  10  2601868
      at Time 500  4121904

I let the computation run for 500 time steps which is sufficient to expose the increase of memory usage over time. It can be seen that the memory usage with the first scheme remains roughly constant. I can look a bit more into it but if it is too time consuming, I won't be able to perform useful diagnostics. I understand that using massif can slow down program execution up to 20 times, which is prohibitive for me as I have other work to do with OpenFOAM. I'll keep you posted.

Cheers, Claudio

[Ship Designer]

I apologize for my late follow-up on this and it appears that the problem was already solved. Since making any modifications to a Docker container is not very straightforward, I tried to do the profiling with Valgrind on an OpenFOAM v8 installation I have on Ubuntu 18 in VirtualBox. Surprise surprise, when I ran the case, memory usage didn't increase over time…

After a bit of research I then remembered that I had installed the OpenFOAM v8 on that virtual machine sometime after the v8 Docker initial release. It then occurred to me, that it was the patched version that I installed an the virtual machine. I then downloaded and installed the latest (patched) v8 Docker image — which I internally call OpenFOAM v8.1 — and the problem was gone. MPLIC now works like a charm, without increasing memory usage over time. The Docker images available for download are all called v8, even the patched one, and it's difficult to tell them apart. I would welcome the Docker image designations to be more specific.

I can't explain why the error disappeared, since except 1-2 source files that seem not directly related, the source files were last modified at the time of the v8 initial release and not later. I also couldn't find any release notes that suggest that such a bug was fixed and thus it never occurred to me to update my initial v8 Docker container. Go figure…

Below is again the memory usage of the v8 DTC Hull tutorial case found in /tutorials/multiphase/interFoam/RAS/DTCHull running in parallel on 4 cores as above, except this time running on the patched Docker OpenFOAM v8:

           Scheme  div(phi,alpha) Gauss interfaceCompression vanLeer 1 (original)
Memory usage [kB]  
      at Time   0  1651880 
      at Time  10  2077792
      at Time 500  2043216

           Scheme  div(phi,alpha) Gauss PLIC interfaceCompression vanLeer 1
Memory usage [kB]  
      at Time   0  1605536 
      at Time  10  2493776
      at Time 500  2532268 (okay)

           Scheme  div(phi,alpha) Gauss MPLIC
Memory usage [kB]  
      at Time   0  1612616 
      at Time  10  2477772
      at Time 500  2525464 (okay)

Cheers, Claudio

[babakflame]

Hi Claudio

Is it possible to elaborate how did you get the memory usage of each scheme??

Thank you

[erice]

Hello Claudio

I have the same problem in OpenFOAM10 in interFoam, I use Ubuntu 20.04 for wsl and the tutorial is fine. However for my own case the Memory Accumulates until killed. Do you have some suggestion?

Thank you

[Pappelau]

IMHO this is not really a Problem as long as u actually use the PLIC/MPLIC-Solver. They allocate more memory for each iteration as long as your Watersurface *complexity* increases, the memory allocation should increase (this is true even more for MPLIC than PLIC). But if it rises while simulating a "flat" water surface something is wrong .. greetings