[Gerry Kan]

Howdy Foamers:

I am wondering if there is a way to have OpenFOAM (line) samples *not* generate a dataset for each output interval.

For my validation case I would like to determine basic turbulent statistics (i.e., mean and fluctuation). For this purpose I take defined sampling line in the domain within controlDict. This will generate a dataset for each output interval, in my case each time step.

However, to match my LDA measurements, I need to sample my output at a very high frequency for a fairly long time (like 1 kHz for 1 hour). This is to ensure both the mean (duration) and fluctuations (frequency) are correctly represented in my openFOAM run.

The caveat is that, since with samples, openFOAM creates a directory for each sampling interval, I will end up having 3.6 million directories. I don't know if our Linux Cluster (on the OS side) would allow so many directories to be created, not to mention the time it would take to go through the files in each directory (a lesser problem, I would argue).

Is there another way to have openFOAM concatenate output sample data so that I don't end up having millions of directories, if this is at all possible in any OS?

Thanks in advance, Gerry.

[HPE]

I am not sure that sample creates a new directory and file at each write step in OpenFOAM; but, in any case, you can call a systemCall FO which can concatenate two files, and subsequently delete the former one at each time-step. Does it make sense?

FYI: I am sure that the cluster will not allow you more than 100k-500k files/dirs.

[Gerry Kan]

Dear Herpes:

A system call in a cluster environment is probably not a very good idea. Lags in disk I/O, and the fact that computation might take place elsewhere and copied back to your working directory, would pose serious challenge in sychronizing your system call with the output. I know that not all clusters do that, some of the ones I use do.

Not to mention, doing anything that involves deleting output during a computation, you risk losing data. You will find yourself pulling out your hair very quickly, especially if your jobs take weeks to complete.

Having said that, I think it might be a better choice to write a script to monitor progress of the OpenFOAM output and move them to sub-folders accordingly. The combination and deletion of the data can take place afterwards, when I know everything is in a safe place.

Gerry.

P.S. - Judging from the links you attached in your signature on your posts, I assume you are commercially affiliated with ESI, and you answering user questions on CFD-online forms part of your regular support activity? If that is the case, I would strongly suggest you revise the way you respond to them. The helpfulness of your comments notwithstanding, they do come across as being rather condescending.

[HPE]

Edited:

Nobody is paying me to be here (all free work - free time to try to help others as much as I can), and I must refuse your perception/opinion for the 95% of my posts, I'm afraid. Most of the time, it is just international English: see an example potentialFoam, erasing the U and p fields.

[Tobermory]

Perhaps consider taking a different tack. Presumably this is an LES or DNS simulation? How about using the fieldAverage functionObject to calculate mean and mean square statistics, and then when you have averaged for long enough, run the sample as a postProcessing exercise on the mean and rms fields.

Good luck!

[Gerry Kan]

Dear Tobermory:

Thanks. Yes, it is an LES simulation. The fieldAverage function will come in handy, but at the moment I need to extract point measurements for mean and fluctuations along different line stations. I am still in the process of figuring out what to do.

Thanks, Gerry.

[Tobermory]

Understood. Here's the thing though (and I only realised this recently) - you can always run your sampling as a post processing activity, choosing what time step(s) you want to run it on. You don't have to do your sampling during the run (unless you want to save the time history of the samples). For example, with pisoFoam, you can run pisoFoam -postProcess. When you do that, instead of continuing the run, it just runs any postprocessing function objects.

So, if you have run your LES simulation for a while and have developed a time-averaged field at say time t1, you can then play around with as many different samples as you wish, all on the t1 data fields, as a postprocessing activity.

Just a suggestion. May not work for you, but have a think.

[Gerry Kan]

Originally Posted by Tobermory

Understood. Here's the thing though (and I only realised this recently) - you can always run your sampling as a post processing activity, choosing what time step(s) you want to run it on.

Hi Tobermory:

I have used this feature before, and this requires you to have the solution already computed and saved, so that it is available for sampling / probing for the time requested. If you are talking about 1 ms for an hour, it's going to be a lot of data. In fact, much worse than having just 3.6 milliion line sample data.

In the meantime I talked to my cluster manager. He believes that the OS *should* be able to handle 3.6 million directories, but the only issue is how long it takes to list all of them. Previously I had a case with only 360k directories (I think, I was sampling at 10 ms) and it took "a long time" (again I didn't take a stop watch, but it was very long) for an ls operation in my script to retrieve the listing.

The compromise seems to be to have another process monitor the postProcessing output and move them to a directory tree structure, each with a more manageable number of subdirectories containing the sampled data.

Thanks for your suggestions, though, Gerry.

[Tobermory]

Understood - sounds like you have a good plan. Good luck!