[fs82]

Dear Foam Developers,

I am currently experiencing a little problem with reconstructPar. The statistics of OpenFoam (LES) are limited to <Ui> and <Ui'Uj'>. For most of the eningeering purposes this is enough. But for meteorological applications one needs correlations, high order statistics and more. The most used method is to use a high number of snapshots (>900). I have done so and my case (11 million grid points) runs very good on 640 cores of a AMD Bulldozer Cluster. I saved 1800 snapshots and now they are stored in the processor directories: one file for each field and time and processor. One reconstructPar job for all times is much to slow for all of the 1800 snapshots, one reconstructPar job for each time is a big problem for the file system and does not work also. Limiting the number of reconstructPar jobs to 32 (still one job for one time) gives a annoying behavior: some went through within 1 hour, some not within 12 hours. I have to say, I am not alone on this machine and I have to share it. Therefore exists a queuing system which limits the execution time.
Most of the other codes I know uses one result file per field even in parallel runs. The fields of the processors are combined during runtime and than stored. May be it is possible to include such a feature into OpenFoam? This would reduce the amount of time for postprocessing very much?

Kind regards,
Fabian

[wyldckat]

Greetings Fabian,

Wow, that's a whole lot of data. And I think you're not alone when it comes to post-processing this amount of OpenFOAM data from clusters, so until someone else more experience can give his/her very welcome hints, here are mine...

OK, what kind of post-processing are you going to need?

• If it's all done with ParaView and memory-wise the case fits in a single workstation, then you can use the internal ".foam" plugin for opening the files. As of OpenFOAM 2.0.0 and/or ParaView 3.8, you can run like this:

  paraFoam -builtin

  If this doesn't work, simply run:

  touch case.foam
  paraview

  And then open the file "case.foam".
• If you don't have a workstation that has that much memory and disk space, as well as only requiring ParaView... then it's possible but it's going to be a bit crazy to do.... perhaps it's best to see the idea after this list. Nonetheless, if this is the case, you can ask how to do it here as well
• If you need to do post-processing with something other than ParaView... things are going to get tricky. If the post-processing tools also use OpenFOAM code, then you should also use them in parallel.

OK, the final idea - when it comes to supercomputing and OpenFOAM, it's useful to do either one or both of the following:

• Go to one of the more extensive workshops about OpenFOAM.
• Request commercial support from anyone that properly supports OpenFOAM on clusters.

Last but not least, if you want the official OpenFOAM developers to see your idea, either:

• Post it as a feature request on the bug tracker: http://www.openfoam.org/mantisbt/my_view_page.php
• Contact support: http://www.openfoam.com/support/index.php

If you want this to be done "for free" (takes a whole lot longer in most cases, if at all) by the community, you can either try posting the request at:

• OpenFOAM Programming & Development
• http://www.extend-project.de/
  • This project's bug tracker: http://sourceforge.net/apps/mantisbt.../main_page.php

Best regards,
Bruno

[fs82]

Thx Bruno for your detailed answer. I am using OF now for 3 years always on HPCs and I am still very happy with it. I am with you that visiting a workshop or buying support would be the best solution. Nevertheless I am working at the university (TU Dresden) and as you might know money for support, licences and workshops is always a big problem. Therefore I have to help myself or ask the forum, which helped me a lot in the past.

Originally Posted by wyldckat

Wow, that's a whole lot of data.

Yes it is, but I think is looks also more complicated than it is. I need high order statistics (skewness, kurtosis, correlations) for postprocessing. This is not implemented so far but if one has enough snapshots this is an easy OF utility ... I hope so. I use Tecplot for visualization, which would be the answer if paraview would be an option.

Originally Posted by wyldckat

OK, what kind of post-processing are you going to need?

I am only interested in statistics of the velocity field. This means I only need the instantaneous velocity field of the snapshots. I will do the postprocessing on a single core linux machine with 96GB memory. I think this should be enough. OF library provides the basic IO (mesh loading, reading fields of varios times, writing and basic mathematics) and I just have to combine it to a little tool. I hope it would be so easy as I imagine but I will see.

Originally Posted by wyldckat

If you want this to be done "for free" (takes a whole lot longer in most cases, if at all) by the community, you can either try posting the request at:

• OpenFOAM Programming & Development
• http://www.extend-project.de/
• This project's bug tracker: http://sourceforge.net/apps/mantisbt.../main_page.php

This is was one of the reasons, why I posted it in this subforum. If I have some time I could write a bug report. This was a question to the forum if somebody has the same problems and a solution. Also it was a question to the developers if it is possible to add a advanced parallel IO which gives the user the requested field at the and of the computational run. Most of the academic codes I know (semtex, lesocc) have such a feature. May be it is possible to add it to OF in the future.

Now something new: I found a workaround yesterday. Our queuing-system allows array jobs and one can specify the number of jobs running at the same time. Also reconstructPar support time ranges. Combining such a time range with an array job with limited jobs running parallel reduces the load for the file system and the jobs seems to have not such a high difference in the runtime.

kind regards,
Fabian