|
[Sponsors] |
openFoam HPC 512 cores file limit issue 16000 |
![]() |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
![]() |
![]() |
#1 |
New Member
Balaji
Join Date: May 2013
Posts: 21
Rep Power: 13 ![]() |
Hi guys,
I am running CHTMultiregionFoam on hpc cluster with 512 cores. The problem i face is the number of files that can be saved on the cluster. Lets say I would like to save a field ( T & P_rgh) in one of the regions during the transient run of 2000s saved at every second. The total number of files created would be = 512 proc x 2000 folders x 2 fields That would be around 2.048 Million files together. Is there any way to control the writing output, to save the file of each timestep reconstructed to move to postprocessing folder automatically? So eventually there will be 4000 files saved instead of saving inside the processer folders. Since it is transient simulation, I would like to know how to save the space on HPC cluster/file management. Help would be appreciated |
|
![]() |
![]() |
![]() |
![]() |
#2 |
Member
Lukas
Join Date: Sep 2021
Posts: 36
Rep Power: 4 ![]() |
Hey there,
i never ran into this problem but a possible way to avoid having to much files inside the folder: write a script that run alongside your calculation that reconstruct every now and then. After successful reconstruct delete all reconstructed processor time folders ? |
|
![]() |
![]() |
![]() |
![]() |
#3 |
New Member
Balaji
Join Date: May 2013
Posts: 21
Rep Power: 13 ![]() |
Hi Pappelau,
We are uploading the files to Telekom HPCs. Is there any script that can be included inside the functions? Or do you mean Shell scripting? Help would be appreciated. Like you said, reconstructing on the fly is one way to solve the file size limit issue. Is there any other way to save the files on the fly? Regards |
|
![]() |
![]() |
![]() |
![]() |
#4 |
Member
Hussam
Join Date: Aug 2017
Location: Germany
Posts: 32
Rep Power: 9 ![]() |
Try to use the collated file handler. If I get your problem correctly, this should help you, that there will be only one folder (processors512) instead of 512 folders(processor0..processor511)
|
|
![]() |
![]() |
![]() |
![]() |
#5 |
Member
Lukas
Join Date: Sep 2021
Posts: 36
Rep Power: 4 ![]() |
Yes i mean shell/bach scripting, but tobiass way seems more appropriate.
|
|
![]() |
![]() |
![]() |
![]() |
#6 |
Member
Hussam
Join Date: Aug 2017
Location: Germany
Posts: 32
Rep Power: 9 ![]() |
Two additional procedures to save space:
Turn on the compression flag in controlDict. And if you know already what do want to postprocess in the time steps, then use the openfoam functioobjects ( slices, probes, avg values) instead of saving the whole domain more often. |
|
![]() |
![]() |
![]() |
![]() |
#7 |
New Member
Balaji
Join Date: May 2013
Posts: 21
Rep Power: 13 ![]() |
Hi Toby,
Vielen Dank, was wir genau wollten. Grüß aus Köln Balaji |
|
![]() |
![]() |
![]() |
![]() |
#8 |
New Member
Balaji
Join Date: May 2013
Posts: 21
Rep Power: 13 ![]() |
Hi Guys,
I tried the collated option on cluster, still I get the binary reading block error with 512 cores. I tried the same on my local workstation with multiple hosts of 60 cores. Here the collated function worked and created only one folder named processor60 where the whole simulation ran. Could anyone point out what is the issue on HPC with 512 cores? I used in my controDict write as binary without any compression. Is is something to do with maxThreadFileBufferSize 0; and maxMasterFileBufferSize 2e9; ? Or may be there is aa version/format issue on the cluster? I am using openfoam 2012 on the cluster Code:
OptimisationSwitches { //- Parallel IO file handler // uncollated (default), collated or masterUncollated fileHandler collated; //- collated: thread buffer size for queued file writes. // If set to 0 or not sufficient for the file size threading is not used. // Default: 2e9 maxThreadFileBufferSize 0; //- masterUncollated: non-blocking buffer size. // If the file exceeds this buffer size scheduled transfer is used. // Default: 2e9 maxMasterFileBufferSize 2e9; } Code:
/*---------------------------------------------------------------------------*\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: v2012 | | \\ / A nd | Website: www.openfoam.com | | \\/ M anipulation | | \*---------------------------------------------------------------------------*/ Build : _7bdb509494-20201222 OPENFOAM=2012 Arch : "LSB;label=64;scalar=64" Exec : chtMultiRegionFoam -parallel Date : Sep 24 2021 Time : 17:52:15 Host : r36c2t8n4 PID : 3570139 I/O : uncollated Case : /lustre/hpe/ws10/ws10.1/ws/sfrp0004-b03b9b34e10ac82eb000/b03b9b34e10ac82eb000/output nProcs : 512 Hosts : ( (r42c4t5n4 128) (r42c4t6n3 128) (r42c4t6n2 128) (r42c4t6n4 128) ) Pstream initialized with: floatTransfer : 0 nProcsSimpleSum : 0 commsType : nonBlocking polling iterations : 0 trapFpe: Floating point exception trapping enabled (FOAM_SIGFPE). fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 5, maxFileModificationPolls 20) allowSystemOperations : Allowing user-supplied system call operations // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // Create time Overriding OptimisationSwitches according to controlDict fileHandler (unregistered) maxThreadFileBufferSize 0; maxMasterFileBufferSize 2e+09; Overriding fileHandler to collated I/O : collated (maxThreadFileBufferSize 0) Threading not activated since maxThreadFileBufferSize = 0. Writing may run slowly for large file sizes. Create fluid mesh for region cabin for time = 0 [0] [0] [0] --> FOAM FATAL IO ERROR: (openfoam-2012) [0] Expected a ')' while reading binaryBlock, found on line 41: label 17969938 at stream position 72434812 [0] [0] [0] file: /lustre/hpe/ws10/ws10.1/ws/sfrp0004-b03b9b34e10ac82eb000/b03b9b34e10ac82eb000/output/processors64/constant/cabin/polyMesh/points at line 41. [0] [0] From bool Foam::Istream::readEnd(const char*) [0] in file db/IOstreams/IOstreams/Istream.C at line 134. [0] FOAM parallel run exiting |
|
![]() |
![]() |
![]() |
![]() |
#9 |
Member
Hussam
Join Date: Aug 2017
Location: Germany
Posts: 32
Rep Power: 9 ![]() |
Hi Balaji,
I usually don't change the optimisation switches. I simply use the argument "-fileHandler collated" with every openfoam tool. For example: decomposePar -allRegions -fileHandler collated -dict system/decomposeParDict. The same argument can be used with solvers and other utilities. I can't help you with your version, as I use basically the foundation versions i.e 7, 8,9. But maybe others can suggest something, if it really has something to do with the openfoam version. Regards Hussam |
|
![]() |
![]() |
![]() |
Tags |
chtmultiregionfoam, filelimit, hpc, multiple cores, writecontrol |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
what is swap4foam ?? | AB08 | OpenFOAM | 28 | February 2, 2016 01:22 |
[Other] Adding solvers from DensityBasedTurbo to foam-extend 3.0 | Seroga | OpenFOAM Community Contributions | 9 | June 12, 2015 17:18 |
[OpenFOAM] Annoying issue of automatic "Rescale to Data Range " with paraFoam/paraview 3.12 | keepfit | ParaView | 60 | September 18, 2013 03:23 |
OpenFOAM on MinGW crosscompiler hosted on Linux | allenzhao | OpenFOAM Installation | 127 | January 30, 2009 19:08 |
DxFoam reader update | hjasak | OpenFOAM Post-Processing | 69 | April 24, 2008 01:24 |