[kingdomkarl]

Hello everyone,

I'm experiencing weird behavior with OF when it comes to very large IO dictionaries. Basically, I am interpolating a lot of my fluid variables from a large multidimensional table. Due to the number of dimensions these tables tend to get quite large (easily 10GB+).

When running my case in serial everything is fine and the initial results look good. When running in parallel with smaller tables (< 2-3 GB) also everything is fine and the results are in agreement with experimental data. Now when I increase my table size above 2-3 GB and run the case in parallel the solver "hangs" when reading in the dictionary and does not advance nor give any kind of error message. Again on single core the large tables work fine, so the table file shouldn't be compromised.

RAM requirement should also be fine on the machine I'm using, although I'm guessing that what I'm doing is terribly inefficient since I'm utilizing only MPI and no shared memory message passing (eg. openMP).

The IO read in for the table as an IODictionary looks like the following:

IOdictionary control
(
IOobject
(
fileName_,
instance,
obr,
IOobject::MUST_READ,
IOobject::NO_WRITE
)
);
control.readEntry("fields", entries_);

So basically a direct copy of the readIn method in "interpolationLookUpTable".

For any hints on how to solve this issue I'd be really thankful

[klausb]

Maybe this helps:

There's a section called "OptimisationSwitches" in /home/.../OpenFOAM/OpenFOAM-8/etc/controlDict which deals with Parallel IO handling. It could help to set it to "collated" and set a buffer size...

Here you can find a comprehensive explanation what it is about: https://github.com/OpenFOAM/OpenFOAM...ae435d4ab42e6f

Basically, you'll run the case in parallel but now, IO is only done and managed through the master process. If you write your own IO code for your purposes, it might help to leverage this functionality.

[HPE]

There are also ADIOS and ADIOS2 in the OpenFOAM modules.

[kingdomkarl]

Hey,

they both seems like viable options, thank you very much for the quick replies I'll dig through it next week and will post my updates accordingly.

[Nealcaffrey]

Hi kingdomkarl,

Any updates on the collated cluster run?

Did it work on the cluster?

Please post your script for decompose, runnning and reconstructing.

I am trying the same on cluster to save some file limit. Would be nice if you could share your updates and how you overcome the issue

Regards