I am developing a 3D time dependent CFD code by FORTRAN. The present way to save the results is like this: after finishing the calculation of each time step, the program will open an output text file, read line by line to the end, write results of finished time step into the file, close the file, and start the computation of new time step.

On this way, after hundreds of time steps, the output file is of a large size like 500M even 1G. It will take much time to read the file line by line to the end.

Could you please give me any suggestion to change this situation?

Thanks in advance.

don't write results every timestep????

Save every tiem step in different file, so you do not need to read the file before writing. You can always create a script later to put them together. Then, are you sure you need every time step ?. If you are doing DNS it is okey (prepare to buy huge hard drives). If not maybe saving statistics will be enough and writing them at the end. Of course you will ned to save now and then to avoid panic attacks when someone switch off your computer after 5 days simulation.

In fortran, you can open the file with the "position-'append'" clause. Then your write statement will add the new time step results to the end of the file.

Below, the open statement opens unit 33 for 'tecfile.out', it's an 'old' file and your write statements will append the information at the end of the file.

open (unit=33,form='formatted',file='tecfile.out',

x position='append',status='old')

Once the file is open, it's write (33,???) ... .

I lifted the open statement from one of my working codes.

Good luck!

To echo the other posters, do you really need to save every time step?

Use binary format, it's better then txt format. Later you can easily convert it

Thanks for suggestions from you guys.

Yes, I prefer to save every time step. My case involves mocrogravity and g-jitter, which means the gravity term will change with time.

Could you please give my some instructions on how to use binary format? Where can I find the detail on this?

Files File organization refers to the way records are physically arranged on a storage device.

Record type refers to whether records in a file are all the same length, are of varying length, or use other conventions to define where one record ends and another begins.

Record access refers to the method used to read records from or write records to a file, regardless of its organization. The way a file is organized does not necessarily imply the way in which the records within that file will be accessed.

Fortran supports two kinds of file organizations: sequential and relative. The organization of a file is specified by means of the ORGANIZATION keyword in the OPEN statement. Relative files must be stored on disk. However, sequential files can be stored on either magnetic tape or disk. Other peripheral devices, such as terminals, pipes, card readers, and line printers, are treated as sequential files.

A sequentially organized file consists of records arranged in the sequence in which they are written to the file (the first record written is the first record in the file, the second record written is the second record in the file, and so on). As a result, records can be added only at the end of the file. Attempting to add records at someplace other than the end of the file will result in the file begin truncated at the end of the record just written.

Within a relative file are numbered positions, called cells. These cells are of fixed equal length and are consecutively numbered from 1 to n, where 1 is the first cell, and n is the last available cell in the file. Each cell either contains a single record or is empty. Records in a relative file are accessed according to cell number. A cell number is a record's relative record number; its location relative to the beginning of the file. By specifying relative record numbers, you can directly retrieve, add, or delete records regardless of their locations. (Detecting deleted records is only available if you specified the /vms option when the program was compiled. For information, see the /vms option.)

Fortran supports two methods of file access (sequential and direct) and three kinds of file structure (formatted, unformatted, and binary). Sequential-access and direct-access files can have any of the three file structures. The following kinds of files are possible:

Formatted Sequential

Formatted Direct

Unformatted Sequential

Unformatted Direct

Binary Sequential

Binary Direct

Each kind of file has advantages and the best choice depends on the application you are developing:

Formatted Files

You create a formatted file by opening it with the FORM='FORMATTED' option, or by omitting the FORM parameter when creating a sequential file. The records of a formatted file are stored as ASCII characters; numbers that would otherwise be stored in binary form are converted to ASCII format. Each record ends with the ASCII carriage return (CR) and line feed (LF) characters.

If you need to view a data file's contents, use a formatted file. You can load a formatted file into a text editor and read its contents directly, that is, the numbers would look like numbers and the strings like character strings, whereas an unformatted or binary file looks like a set of hexadecimal characters.

Unformatted Files

You create an unformatted file by opening it with the FORM='UNFORMATTED' option, or by omitting the FORM parameter when creating a direct-access file. An unformatted file is a series of records composed of physical blocks. Each record contains a sequence of values stored in a representation that is close to that used in program memory. Little conversion is required during input/output.

The lack of formatting makes these files quicker to access and more compact than files that store the same information in a formatted form. However, if the files contain numbers, you will not be able to read them with a text editor.

Binary Files

You create a binary file by specifying FORM='BINARY'. Binary files are the most compact, and good for storing large amounts of data.

Sequential-Access Files

Data in sequential files must be accessed in order, one record after the other (unless you change your position in the file with the REWIND or BACKSPACE statements). Some methods of I/O are possible only with sequential files, including nonadvancing I/O, list-directed I/O, and namelist I/O. Internal files also must be sequential files. You must use sequential access for files associated with sequential devices.

A sequential device is a physical storage device that does not allow explicit motion (other than reading or writing). The keyboard, screen, and printer are all sequential devices.

Direct-Access Files

Data in direct-access files can be read or written to in any order. Records are numbered sequentially, starting with record number 1. All records have the length specified by the RECL option in the OPEN statement. Data in direct files is accessed by specifying the record you want within the file. If you need random access I/O, use direct-access files. A common example of a random-access application is a database.

All files are composed of records. Each record is one entry in the file. It can be a line from a terminal or a logical record on a magnetic tape or disk file. All records within one file are of the same type.

In Fortran, the number of bytes written to a record must be less than or equal to the record length. One record is written for each unformatted READ or WRITE statement. A formatted READ or WRITE statement can transfer more than one record using the slash (/) edit descriptor.

For binary files, a single READ or WRITE statement reads or writes as many records as needed to accommodate the number of bytes being transferred. On output, incomplete formatted records are padded with spaces. Incomplete unformatted and binary records are padded with undefined bytes (zeros).

The remainder of this section contains information about:

Record Types

Microsoft Fortran PowerStation Compatible Files (when /fpscomp was specified during compilation)

WRITE

Statement: Transfers output data to external sequential, direct-access, or internal records.

Syntax

Sequential

Formatted:

WRITE (eunit, format [, advance] [, iostat] [, err]) [io-list]

Formatted - List-Directed:

WRITE (eunit, * [, iostat] [, err]) [io-list]

Formatted - Namelist:

WRITE (eunit, nml-group [, iostat] [, err])

Unformatted:

WRITE (eunit [, iostat] [, err]) [io-list]

Direct-Access

Formatted:

WRITE (eunit, format, rec [, iostat] [, err]) [io-list]

Unformatted:

WRITE (eunit, rec [, iostat] [, err]) [io-list]

Indexed (VMS only)

Formatted:

WRITE (eunit, format, [, iostat] [, err]) [io-list]

Unformatted:

WRITE (eunit, [, iostat] [, err]) [io-list]

Internal

WRITE (iunit, format [, iostat] [, err]) [io-list]

eunit

Is an external unit specifier, optionally prefaced by UNIT=. UNIT= is required if eunit is not the first specifier in the list.

format

Is a format specifier. It is optionally prefaced by FMT= if format is the second specifier in the list and the first specifier indicates a logical or internal unit specifier without the optional keyword UNIT=. For internal WRITEs, an asterisk (*) indicates list-directed formatting. For direct-access WRITEs, an asterisk is not permitted.

advance

Is an advance specifier (ADVANCE=c-expr). If the value of c-expr is 'YES', the statement uses advancing input; if the value is 'NO', the statement uses nonadvancing input. The default value is 'YES'.

iostat

Is the name of a variable to contain the completion status of the I/O operation. Optionally prefaced by IOSTAT=.

err

Are branch specifiers if an error (ERR=label) condition occurs.

io-list

Is an I/O list: the names of the variables, arrays, array elements, or character substrings from which or to which data will be transferred. Optionally an implied-DO list.

form

Is the nonkeyword form of a format specifier (no FMT=).

*

Is the format specifier indicating list-directed formatting. (It can also be specified as FMT=*.)

nml-group

Is the namelist group specification for namelist I/O. Optionally prefaced by NML=. NML= is required if nml-group is not the second I/O specifier.

rec

Is the cell number of a record to be accessed directly. Optionally prefaced by REC=.

iunit

Is an internal unit specifier, optionally prefaced by UNIT=. UNIT= is required if iunit is not the first specifier in the list. It must be a character variable. It must not be an array section with a vector subscript.

If a parameter of the WRITE statement is an expression that calls a function, that function must not execute an I/O statement or the EOF intrinsic function, because the results are unpredictable.

Best Wishes

Mike

it is a sample from M. Peric

C################################################# ##################

SUBROUTINE SRES(K) C################################################# ##################

C This routine writes out the results onto a file C so that re-start is possible at a later stage. C================================================= ==================

C

WRITE(FILRES,'(A6,3H.re,I1)') NAME,K

OPEN (UNIT=3,FILE=FILRES,FORM='UNFORMATTED')

REWIND 3 C

IJST=IJGR(K)+1

IJEN=IJGR(K)+NIGR(K)*NJGR(K)

WRITE(3) K,IJST,IJEN,ITIM,TIME,(F1 IJ),IJ=IJST,IJEN),

* (F2(IJ),IJ=IJST,IJEN),(U(IJ),IJ=IJST,IJEN),

* (V(IJ),IJ=IJST,IJEN),(P(IJ),IJ=IJST,IJEN),

* (T(IJ),IJ=IJST,IJEN),

* (FMOC(I),I=IOCS(K)+1,IOCS(K)+NOC(K))

IF(LTIME) WRITE(3) (UO(IJ),IJ=IJST,IJEN),(VO(IJ),IJ=IJST,IJEN),

* (TO(IJ),IJ=IJST,IJEN) C

CLOSE(UNIT=3) C

RETURN

END

I don't see the need to close and reopen the file for every time step. Just open it initially and keep it open throughout the computation so all new data is automatically appended. Use option "unformatted" to save time a disk space.

Open and close takes a negligible amount of processor time compared to all the computation that a CFD time step requires.

If the file is open when your program terminates unexpectedly (that is, it crashes!), the last results on the file will be lost! Closing a file forces the I/O buffer to empty onto the storage media.

Also, rewind is an ancient command associated with magnetic tape storage. Probably not very useful if you're saving to a hard drive.

I am not worried about the time it takes to open and close a file and I also didn't say anything about 'rewind'.

I am just saying, it doesn't make sense to close and reopen (regardless of the time it takes), and then re-read the old data every time you open the file (which obviously takes a long time).

You can keep the file open throughout the computation,write the new data as soon as it's available (when the new time step is finished). Fortunately, there are ways to flush the buffer to the disk without crudely closing a file, and you can do that whenever needed, completely under your control.

not every OS and/or compiler(at least in fortran) has a working "flush" command.

I don't know of any operating system that doesn't let you flush the buffer. which one is that? even the 'close' command wouldn't be able to flush if the OS itself prohibits it.

the flush command may not be standard fortran but the major compilers that I know of (maybe excluding gnu) do have it. if it's not available with your compiler, you could always implement a C subroutine to do that for you (if you think it's worth it).

"and I also didn't say anything about 'rewind'."

Michial mentioned rewind as a part of his quote from Peric's book. We're trying to help hawk, and throwing a "rewind" in won't do that.

"I am just saying, it doesn't make sense to close and reopen (regardless of the time it takes), and then re-read the old data every time you open the file (which obviously takes a long time)."

You don't re-read the data if you use position='append' in the open statement. The write statement will begin at the end of the previous file.

"You can keep the file open throughout the computation,write the new data as soon as it's available (when the new time step is finished)."

"Fortunately, there are ways to flush the buffer to the disk without crudely closing a file, and you can do that whenever needed, completely under your control."

Good. Wish you'd said that there are other ways to flush the buffer in your previous post. Hawk (and I) will want to know what they are.

>Michial mentioned rewind as a part of his quote from
:Peric's book. We're trying to help hawk, and throwing
:a "rewind" in won't do that.

No, it won't and I didn't say it would.

>You don't re-read the data if you use
osition='append' in the open statement. The write >statement will begin at the end of the previous file.

That's absolutely correct, too, but again, I wasn't talking about 'append'. My reply refers to hawk's original post and nothing else. He did reread the data every time, instead of using append or keeping the file open. Just because I don't reiterate everything that has been said in previous posts does not mean I disagree with it. I am merely offering an alternative. Don't you think there is usually more than one way to solve a problem?

For flush, use the fortran routine FLUSH(id), which many compilers offer, alternatively do it with a C subroutine, and there may be other ways I don't know of.