[Rickard.Solsjo]

Hi, does anyone know what this means.
'
[0] [2] [3]
[0]
[0] --> FOAM FATAL IO ERROR:
[0] error in IOstream "/disk/global/home/rsolsjo/OpenFOAM/rsolsjo-1.6.x/run/RealCase3/processor0/0.00032/T" for operation Ostream& operator<<(Ostream&, const char)
[0]
[0] file: /disk/global/home/rsolsjo/OpenFOAM/rsolsjo-1.6.x/run/RealCase3/processor0/0.00032/T at line 101267.
[0]
[0] From function IOstream::check(const char*) const
[0] in file db/IOstreams/IOstreams/IOstream.C at line 91.
[0]
FOAM parallel run exiting
[0]

[2]
[2] --> FOAM FATAL IO ERROR:
[2] error in IOstream "/disk/global/home/rsolsjo/OpenFOAM/rsolsjo-1.6.x/run/RealCase3/processor2/0.00032/N-DODECANE" for operation Ostream& operator<<(Ostream&, const Scalar&)
[2]


and

[2] #0 Foam::error:rintStack(Foam::Ostream&) in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libOpenFOAM.so"
[2] #1 Foam::sigFpe::sigFpeHandler(int) in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libOpenFOAM.so"
[2] #2 __restore_rt at sigaction.c:0
[2] #3 pow.L in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/applications/bin/linux64IccDPOpt/dieselFoam"
[2] #4 Foam::RanzMarshall::Nu(double, double) const in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libdieselSpray.so"
[2] #5 Foam::RanzMarshall::relaxationTime(double, double, double, double, double, double) const in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libdieselSpray.so"
[2] #6 Foam:arcel::setRelaxationTimes(int, double&, Foam::Field<double>&, double&, Foam::Field<double>&, Foam::spray const&, double, Foam::Vector<double> const&, double, double, Foam::Field<double> const&, Foam::Field<double> const&, double) in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libdieselSpray.so"
[2] #7 Foam:arcel::move(Foam::spray&) in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libdieselSpray.so"
[2] #8 void Foam::Cloud<Foam:arcel>::move<Foam::spray>(Foam: :spray&) in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libdieselSpray.so"
[2] #9 Foam::spray::move() in "/disk/global/szrobi/OpenFOAM/OpenFOAM-1.6.x/lib/linux64IccDPOpt/libdieselSpray.so"

Thanks!!!

[wyldckat]

Greetings Rickard,

It seems that you are running in parallel, and that the path you are sharing between the various instances of the solver is read-only. I.e. probably "/disk/global/" is shared with read-only options, or it failed to comply with the remote write request due to file size or broken network link.
If it's a broken network link, it could be due to some glitch in the version of remote file sharing technology you are using (NFS, samba, sshfs, etc...)

Best regards,
Bruno

[NorbertB]

I had the same error (first one) and it seems your run has been killed by the system, perhaps due to an overflow or a quota limit.

No idea for the second one.

[wyldckat]

It looks to me that the two errors are related, since both refer to "Foam::Ostream&"!

[wiedangel]

Originally Posted by NorbertB

I had the same error (first one) and it seems your run has been killed by the system, perhaps due to an overflow or a quota limit.

No idea for the second one.

I think you are right about the quota limit ... I get the same error when the disk is full ...

[arunsmec]

I got the same error while calculating average (i'm doing LES). But the error comes after a number of time steps (~3500). MY disk is not full. It has more than 25 GB free space. What could be the reason?

[wyldckat]

Hi Arun,

Originally Posted by arunsmec

I got the same error while calculating average (i'm doing LES). But the error comes after a number of time steps (~3500). MY disk is not full. It has more than 25 GB free space. What could be the reason?

There are two types of quota limit:

• Space limit, where one might only have a limit of 10GB on the shared filesystem.
• File count limit, where the limit is actually on the number of files associated to your account.

I remember there was a thread that detailed a bit how one can offload data from the cluster during runtime, to avoid reaching the file count quota... here you go: http://www.cfd-online.com/Forums/ope...ne-folder.html

Best regards,
Bruno

[arunsmec]

Thank you wyldckat for the quick reply!
I'll try this after my current run. I have solved the issue by creating more space in my disk.

But I would also like to know if it is possible to write outputs from different processors into a single file for a given time ?

[wyldckat]

Originally Posted by arunsmec

But I would also like to know if it is possible to write outputs from different processors into a single file for a given time ?

Do you mean that you want to write the values during the parallel run? If so, that would require the master process to be doing the writing, which would slow down the run, since the master process would be delayed because it had to do more output than the other slave processes.

[arunsmec]

The problem still remains unsolved. The data files are eating the space on my disk. The details are as follows

LES of round jet
no of cells ~ 6 million
no of processors ~ 8
delta T = 5 e-5
writeinterval = 200
fields being written : U, U_0, Umean, Uprime2mean, P, P_0, Pmean, Pprime2mean, phi, phi_0, nuSgs
size of files/processor/writeinterval:
P files ~ 8 MB
U files ~ 26 MB
phi ~ 25 MB
Uprime2mean ~ 55 MB

I am half way through my simulation. The data files altogether has a size of about 90 GB. I had to kill the process. If I want to proceed, I have to free some space and also make sure unwanted details are not being written. As I understand U_0 and phi_0 are fields of previous timestep required for restart. Is it necessary to save these files? Can i remove these files from the directories corresponding to earlier time steps?

[Bernhard]

First, what does writeFormat and writeCompression read in your controlDict? Save your data at least in binary form, and maybe compressed. You can change the format of existing files using the utility foamFormatConvert.

You can also consider using purgeWrite, to cyclic overwrite of old files.

To answer your question, as long as you don't want to restart from a certain time-step, it is safe to remove any of the files in older time-steps. I think there is even a functionObject that controls the fields that are written during outputtimes.

[arunsmec]

Thank you Bernhard
I am writing in ascii format without any compression. How much space I can save by compression?
I have not tried compression before. As compression stores files in zip format, should I decompress the files before executing the reconstructPar?

If I have to restart from time step t1, can I remove the *_0 fields from time steps older than t1?

[Bernhard]

Also try to store in binary, it can save you a lot of disk space, but that depends on your case. In my experience it can save up to a factor of 6 in disk-space.

All OpenFOAM utilities can handle either binary and/or compressed data (as they all use the same readers). Some third-party tools like PyFoam cannot handle reading the binary files to my knowledge.

If you don't need the _0 files for anything, it is safe to remove them, but I would do that only when you are certain that you don't want to restart from an earlier time.

[arunsmec]

Is paraview/paraFoam able to handle binary/compressed data

[Bernhard]

Yes.
Tecplot can also handle it, as far as I know, except for binary meshed. In case of problem you can always return to ascii/uncompressed with foamFormatConvert -time.

[f0208secretx]

I've run into similar problem when running openfoam jobs. I'm sure I still have plenty of disk space and there shouldn't be for the file count.

The error is not completely the same as my Openfoam was manually installed in a folder I have read/write access, and I've also taken care in env $TMPDIR to make sure I have read/write access to this too. However, the simulation will spit out IO error as attached once in a while and I have to restart the simulation from that time step, its very annoying..

system info:
OpenFOAM 2.4.0
CentOS 6.6
64 cores run (4 nodes with 16 cores each) using OpenMPI
pimpleFoam with SADDES turbulence model

I've enabled openmpi debugging. error section in my log:

[41]
[41]
[41] --> FOAM FATAL IO ERROR:
[41] error in IOstream "/scratch/users/jw969/OpenFOAM-run/SADDES_head_v10mph/processor41/1.4068/Ug" for operation Ostream& operator<<(Ostream&, const UList&)
[41]
[41] file: /scratch/users/jw969/OpenFOAM-run/SADDES_head_v10mph/processor41/1.4068/Ug at line 22.
[41]
[41]     From function IOstream::check(const char*) const
[41]     in file db/IOstreams/IOstreams/IOstream.C at line 99.                                                                           [41]                                                                                                                                     FOAM parallel run exiting
[41]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 41 in communicator MPI_COMM_WORLD                                                                          with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.                                                                      You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

[wyldckat]

Greetings f0208secretx,

If the error occurs sporadically, than my first guess would be that there is a time out problem or write error with the file/folder sharing system you are using in your cluster. You have 64 processes, 16 per machine, all writing at the same time to a remotely shared folder "/scratch".

My other guess would be that this occurs only when a bad value is being converted to be written to file. A few questions:

1. Are you using "ascii" or "binary" mode defined in "system/controlDict"?
2. Are you using "compressed" or "uncompressed" in "system/controlDict"?

Best regards,
Bruno

[f0208secretx]

I now believe its a file count problem. I will report if this continue to happen. Thanks.