[Nealcaffrey]

Hi guys,



I am running CHTMultiregionFoam on hpc cluster with 512 cores.

The problem i face is the number of files that can be saved on the cluster.

Lets say I would like to save a field ( T & P_rgh) in one of the regions during the transient run of 2000s saved at every second.

The total number of files created would be = 512 proc x 2000 folders x 2 fields

That would be around 2.048 Million files together.



Is there any way to control the writing output, to save the file of each timestep reconstructed to move to postprocessing folder automatically?

So eventually there will be 4000 files saved instead of saving inside the processer folders.

Since it is transient simulation, I would like to know how to save the space on HPC cluster/file management.

Help would be appreciated

[Pappelau]

Hey there,
i never ran into this problem but a possible way to avoid having to much files inside the folder:
write a script that run alongside your calculation that reconstruct every now and then. After successful reconstruct delete all reconstructed processor time folders ?

[Nealcaffrey]

Hi Pappelau,

We are uploading the files to Telekom HPCs.

Is there any script that can be included inside the functions?
Or do you mean Shell scripting?

Help would be appreciated.

Like you said, reconstructing on the fly is one way to solve the file size limit issue.
Is there any other way to save the files on the fly?

Regards

[tobiasS]

Try to use the collated file handler. If I get your problem correctly, this should help you, that there will be only one folder (processors512) instead of 512 folders(processor0..processor511)

[Pappelau]

Yes i mean shell/bach scripting, but tobiass way seems more appropriate.

[tobiasS]

Two additional procedures to save space:
Turn on the compression flag in controlDict. And if you know already what do want to postprocess in the time steps, then use the openfoam functioobjects ( slices, probes, avg values) instead of saving the whole domain more often.

[Nealcaffrey]

Hi Toby,

Vielen Dank, was wir genau wollten.

Grüß aus Köln
Balaji

[Nealcaffrey]

Hi Guys,

I tried the collated option on cluster, still I get the binary reading block error with 512 cores. I tried the same on my local workstation with multiple hosts of 60 cores. Here the collated function worked and created only one folder named processor60 where the whole simulation ran.

Could anyone point out what is the issue on HPC with 512 cores?

I used in my controDict write as binary without any compression.

Is is something to do with maxThreadFileBufferSize 0; and maxMasterFileBufferSize 2e9; ?
Or may be there is aa version/format issue on the cluster?

I am using openfoam 2012 on the cluster

OptimisationSwitches
{
   

    //- Parallel IO file handler
    // uncollated (default), collated or masterUncollated
    fileHandler collated;

    //- collated: thread buffer size for queued file writes.
    // If set to 0 or not sufficient for the file size threading is not used.
    // Default: 2e9
    maxThreadFileBufferSize 0;

    //- masterUncollated: non-blocking buffer size.
    // If the file exceeds this buffer size scheduled transfer is used.
    // Default: 2e9
    maxMasterFileBufferSize 2e9;
}

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : _7bdb509494-20201222 OPENFOAM=2012
Arch   : "LSB;label=64;scalar=64"
Exec   : chtMultiRegionFoam -parallel
Date   : Sep 24 2021
Time   : 17:52:15
Host   : r36c2t8n4
PID    : 3570139
I/O    : uncollated
Case   : /lustre/hpe/ws10/ws10.1/ws/sfrp0004-b03b9b34e10ac82eb000/b03b9b34e10ac82eb000/output
nProcs : 512
Hosts  :
(
    
    (r42c4t5n4 128)
    (r42c4t6n3 128)
    (r42c4t6n2 128)
    (r42c4t6n4 128)

)
Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
trapFpe: Floating point exception trapping enabled (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 5, maxFileModificationPolls 20)
allowSystemOperations : Allowing user-supplied system call operations
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time
Overriding OptimisationSwitches according to controlDict
    fileHandler (unregistered)
    maxThreadFileBufferSize 0;
    maxMasterFileBufferSize 2e+09;
Overriding fileHandler to collated
I/O    : collated (maxThreadFileBufferSize 0)
         Threading not activated since maxThreadFileBufferSize = 0.
         Writing may run slowly for large file sizes.
Create fluid mesh for region cabin for time = 0
[0] 
[0] 
[0] --> FOAM FATAL IO ERROR: (openfoam-2012)
[0] Expected a ')' while reading binaryBlock, found on line 41: label 17969938 at stream position 72434812
[0] 
[0] 
[0] file: /lustre/hpe/ws10/ws10.1/ws/sfrp0004-b03b9b34e10ac82eb000/b03b9b34e10ac82eb000/output/processors64/constant/cabin/polyMesh/points at line 41.
[0] 
[0]     From bool Foam::Istream::readEnd(const char*)
[0]     in file db/IOstreams/IOstreams/Istream.C at line 134.
[0] 
FOAM parallel run exiting

[tobiasS]

Hi Balaji,
I usually don't change the optimisation switches. I simply use the argument "-fileHandler collated" with every openfoam tool. For example:
decomposePar -allRegions -fileHandler collated -dict system/decomposeParDict.
The same argument can be used with solvers and other utilities. I can't help you with your version, as I use basically the foundation versions i.e 7, 8,9. But maybe others can suggest something, if it really has something to do with the openfoam version.
Regards
Hussam