Postprocessing large data sets in parallel

lakeat · June 10, 2011, 02:14

Quote:

Originally Posted by arjun

just an update to my comment.

I implemented iso surface generation part. It is working well. So now for generating animation of Q criteria for 3billion cells case, I only need 1 processor paraview. Post processing files are very small and thus, i can save them with high frequency and it also means quality of animation would be much better than what I would get from saved calculation data.

Sweet success. :-D

Congrats! You are man!

Well, mind sharing your code, it must be a great code, how did you do that..

lakeat · June 10, 2011, 02:16

Quote:

Originally Posted by sandy

Hi lakeat, I get the error information again. Maybe I should try:

Danie -help

?

"Danie -help" would result an error msg, but "Daniel -help" will not

arjun · June 10, 2011, 03:22

I can outline what I did, so if someone wants to impliment it in openFOAM he can do it too.

first of all it is not difficult thing to do. Implementation does depend on data structure of solver so efficiency might be different from solver to solver.

First, lets assume that solver is has control volumes and each control volume is made of some faces. Each face in turn is made up of some nodes spread in 3d space. Face could be polygonal with nf nodes numbered as n1 n2 .... nf.

Take variable phi whose iso surface you want to generate. Now first step is to transfer this variable to nodes. That is calculate the values of this variable on nodes. This you anyway do if you export data for post processing , as most of post processors want node based data.

Second step is to loop through all the cells one by one. Lets say you are at cell c. For this cell c collect all the faces. Since each face is just a polygon , check its edges ie n1->n2, n2->n3 , nf-1 -> nf , where the iso value falls if it falls. Could be done with the help of equation of line. If iso value happens, store this new node (xi,yi,zi) .

so after you loop through all the faces of cell c, you will have say m new nodes generated. They make a polygon that you have to visualise. Store it and finally when all the cells are looped through export this in a vtk file or some format that your program can understand.

For parallel run, each process would generate a part of this file. So for each iteration you would have MAX Np files (Np = number of processors).

Write a small code that will read all these files, merge them and create a single file in any format that post processor can understand. I am using vtk polygons.

Side Note: For cell , m nodes are generated , you would have to arrange them in an order that when it is drawn by post processor it does not look mangled.

Even for very large calculations, each file is only few thousand polygons, and for this reason 1 processor post processor is good enough to create animations.

I compared results with fieldview and they look very similar to it.

I will be adding this into parallel unstructured grid based solver that i am working on. Will be good addition, I guess.

sandy · June 10, 2011, 22:04

stunned .... what means ... pls make clear ...

arjun · June 10, 2011, 23:42

Quote:

Originally Posted by sandy

stunned .... what means ... pls make clear ...

ooops.........

arjun · June 20, 2011, 22:58

In the middle of sept we are going to present our calculation results at http://www.j-focus.or.jp/ along with other presentors. The animations would be from the method i mentioned in this thread.

Before us there is a presentation where they will present a calculation involving 2 billion cells using 10000 cores (software is called frontflow).

This got me thinking, is anyone aware of largest calculation that is done with openFOAM?? And how many processors it used. Is there any guess for it?? Is it 100million 500 million or 1 billion or so??

lakeat · June 21, 2011, 15:09

Quote:

Originally Posted by arjun

In the middle of sept we are going to present our calculation results at http://www.j-focus.or.jp/ along with other presentors. The animations would be from the method i mentioned in this thread.

Before us there is a presentation where they will present a calculation involving 2 billion cells using 10000 cores (software is called frontflow).

This got me thinking, is anyone aware of largest calculation that is done with openFOAM?? And how many processors it used. Is there any guess for it?? Is it 100million 500 million or 1 billion or so??

I recall csc of Finland has done some huge jobs..

arjun · June 22, 2011, 04:34

Quote:

Originally Posted by lakeat

I recall csc of Finland has done some huge jobs..

thanks for this. Their website talks about testing openfoam for 8million , 13.8 million and 22million cells. So I gather they have not run anything higher than 100million cells .

few months ago, someone mentioned about running a case with 80million cells openfoam forum, i wonder how it went.

anyway thanks again.

potentialFoam · June 28, 2016, 04:43

Dear Foamers,

I have another problem with 'Postprocessing large data sets in parallel'. I have a simple case (pimpleFoam) decomposed to 48 processor directories. For postprocessing, I run 'sample' in parallel like here

Code:

mpirun -np 48 sample -parallel -time 1 > log.sample

to extract the velocity 'U' on several lines specified in system/sampleDict.

Usually, everything is fine. But sometimes some lines are only extracted partly. It seems as if some processor-directories just do not contribute their data.
If I reconstruct the case and re-decompose it, the problem still remains the same. But the reconstructed-solution shows no problems (like 'empty' regions) if I watch it wich paraFoam.

Do you have any idea or did you had a similar error once?

June 28, 2016, 04:43		#29
potentialFoam Member Peter Join Date: Nov 2015 Location: Hamburg, Germany Posts: 57 Rep Power: 11	Dear Foamers, I have another problem with 'Postprocessing large data sets in parallel'. I have a simple case (pimpleFoam) decomposed to 48 processor directories. For postprocessing, I run 'sample' in parallel like here Code: mpirun -np 48 sample -parallel -time 1 > log.sample to extract the velocity 'U' on several lines specified in system/sampleDict. Usually, everything is fine. But sometimes some lines are only extracted partly. It seems as if some processor-directories just do not contribute their data. If I reconstruct the case and re-decompose it, the problem still remains the same. But the reconstructed-solution shows no problems (like 'empty' regions) if I watch it wich paraFoam. Do you have any idea or did you had a similar error once?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
PostChannel in parallel	maka	OpenFOAM Post-Processing	2	November 5, 2015 00:05
Smallest binary file format to save large data	Zonexo	Main CFD Forum	2	June 2, 2008 21:25
Large data file to manage	fluid	FLUENT	0	December 13, 2007 13:46
Postprocessing Pressure data spanwise over a wing	Rick	FLUENT	1	February 3, 2006 10:28
How to update polyPatchbs localPoints	liu	OpenFOAM Running, Solving & CFD	6	December 30, 2005 18:27

June 10, 2011, 03:22		#23
arjun Senior Member Arjun Join Date: Mar 2009 Location: Nurenberg, Germany Posts: 1,291 Rep Power: 35	I can outline what I did, so if someone wants to impliment it in openFOAM he can do it too. first of all it is not difficult thing to do. Implementation does depend on data structure of solver so efficiency might be different from solver to solver. First, lets assume that solver is has control volumes and each control volume is made of some faces. Each face in turn is made up of some nodes spread in 3d space. Face could be polygonal with nf nodes numbered as n1 n2 .... nf. Take variable phi whose iso surface you want to generate. Now first step is to transfer this variable to nodes. That is calculate the values of this variable on nodes. This you anyway do if you export data for post processing , as most of post processors want node based data. Second step is to loop through all the cells one by one. Lets say you are at cell c. For this cell c collect all the faces. Since each face is just a polygon , check its edges ie n1->n2, n2->n3 , nf-1 -> nf , where the iso value falls if it falls. Could be done with the help of equation of line. If iso value happens, store this new node (xi,yi,zi) . so after you loop through all the faces of cell c, you will have say m new nodes generated. They make a polygon that you have to visualise. Store it and finally when all the cells are looped through export this in a vtk file or some format that your program can understand. For parallel run, each process would generate a part of this file. So for each iteration you would have MAX Np files (Np = number of processors). Write a small code that will read all these files, merge them and create a single file in any format that post processor can understand. I am using vtk polygons. Side Note: For cell , m nodes are generated , you would have to arrange them in an order that when it is drawn by post processor it does not look mangled. Even for very large calculations, each file is only few thousand polygons, and for this reason 1 processor post processor is good enough to create animations. I compared results with fieldview and they look very similar to it. I will be adding this into parallel unstructured grid based solver that i am working on. Will be good addition, I guess.

June 10, 2011, 22:04		#24
sandy Senior Member Sandy Lee Join Date: Mar 2009 Posts: 213 Rep Power: 18	stunned .... what means ... pls make clear ...

June 20, 2011, 22:58		#26
arjun Senior Member Arjun Join Date: Mar 2009 Location: Nurenberg, Germany Posts: 1,291 Rep Power: 35	In the middle of sept we are going to present our calculation results at http://www.j-focus.or.jp/ along with other presentors. The animations would be from the method i mentioned in this thread. Before us there is a presentation where they will present a calculation involving 2 billion cells using 10000 cores (software is called frontflow). This got me thinking, is anyone aware of largest calculation that is done with openFOAM?? And how many processors it used. Is there any guess for it?? Is it 100million 500 million or 1 billion or so??