[DominicTNC]

Dear Foamers,

Would anyone know if it is possible to do reconstruction in parallel in OF 2.4.0?

Currently, I need to reconstruct a huge case after running snappyHexMesh in 264 processors. When I do reconstructParMesh, about 40GB RAM is needed, and 15GB of them is swapped. So, the reconstruction take really long time.

Any suggestion to reconstruct such huge case?

Thank you! Any comments are appreciated!

[Antimony]

Hi,

Is there a reason you want to do the reconstruction? If it is to visualize results, you can do the same in parallel.

AFAIK, reconstruct (ParMesh or Par) is a serial process, and I don't think it can be done in parallel.

Cheers,
Antimony

[DominicTNC]

Thanks for you quick reply.

Since I'm still new to OF. I used to reconstructPar after runing simulation and then use the "foamToVTK" command to convert the format to *.VTK, and download the VTK file to my desktop from the server (Since paraView cannot be used on the server, I have to download the result to my desktop).

So could you share some more details on visualizing the result without reconstructPar/reconstructParMesh ?

Thank you very much!

Dominic

[Antimony]

Hi,

In your case folder, if you have a .foam file (just an empty file with the extension .foam), you can use that to open your case in Paraview. Simply choose the case type as Decomposed instead of Reconstructed and you can visualize a case that was run in parallel without having to reconstruct it.

Hope this helps.

Cheers,
Antimony

[DominicTNC]

HI,

Is it just creating the empty *.foam file by myself?? Where can I choose the case type "Decomposed" ??

Thank you very much!

Dominic

[Antimony]

Hi,

Yes. In your case folder create an empty/dummy file called (for example) case.foam

In Linux, you could do this by typing "touch case.foam" in the terminal

When you load the case.foam into Paraview, and scroll down under "Properties" you should find one that says "Case Type"

Cheers,
Antimony

[DominicTNC]

Thanks! Really appreciate your help!

[Armin.Sh]

Hi Dominic,

for this purpose I would use the attached code. Just save it in your case and remove .txt before using. For more information, please visit the following link:

bash script for pseudo-parallel usage of reconstructPar

Regards,
Armin

[DominicTNC]

Hi Antimony,

I just find out that if i don't reconstructParMesh after running snappyHexMesh, there will be an error message telling me that boundary conditions of some patch fields cannot be found in processor*/0/ directory when I start to run the solver.

And I noticed that all the missing patch fields are those from the surface generated by snappyHexMesh. In the past, I reconstructParMesh and decomposePar again before running the solver, everything work well. So, any ideas how to solve it?

Many thanks!

[DominicTNC]

Hi Armin,

Thank you for your kindly help.

But the script seems to be useful when I have many time directories. Now I only have a few directories, but lot of processors directories. Reconstruction takes so much memory.

[Antimony]

Hi,

So whenever I do snappy in parallel, the first thing I do is to rename the 0 folder to 0.org. In the 0.org, I make sure that I have the names of all the patches I know for certain will be present in the final mesh (including the processor ones - remember that wildcards are allowed, so you could simply put in "proc*.*" and OF will take care of it for you).

Once snappy is done in parallel, I rename 0.org to 0 and copy it into the processor directories (in terminal, it is just a single line) and start my simulations.

Hope this helps.

Cheers,
Antimony

[Armin.Sh]

Hi Dominic,

firstly about the script: It's very useful and fast for the cases like yours with over 200 processors. What you should mind is, that it's not for the reconstructParMesh, but for the reconstructPar (your future work after running a solver). Since your thread title is "reconstructPar/reconstructParMesh in parallel" I would recommend to use this script for the first part of your title (reconstructPar). With it you can save a lot of time in future!

And secondly about the meshing phase, where you are now: I would like to know, if you use the writeCompression option in the controlDict. If you switch it to "on", you could reduce your data volume to half and therefore do the reconstructParMesh faster.

Regards,
Armin

[DominicTNC]

Hi Armin,

Really?! I am so glad to hear that!

But when I tried to run the script, I got the following error message, hope you could shed some light on this. Thanks.

running reconstructPar in pseudo-parallel mode on 6 processors
reconstructing 1 time directories
making temp dir
Starting Job 1 - reconstructing time = 68 through 0
Job started with PID 9345


--> FOAM FATAL ERROR:
No times selected

    From function reconstructPar
    in file reconstructPar.C at line 210.

FOAM exiting

1 directories remaining...
finished

And I notice that "writeCompression" is off. I have just turn it on.

Many thanks.

[DominicTNC]

Hi Antimony,

Thank you for your guidance. The problem was solved.

Dominic

[Armin.Sh]

Hi Dominic,

you're welcome.

Could you please write your command in the terminal, which led to this error?

Regards,
Armin

[DominicTNC]

Hi Armin,

I typed "sh parReconstructPar.sh -n 6" to try to run the script.

[Armin.Sh]

Hi Dominic,

Please type the following command and inform me about the result:

./parReconstructPar.sh -n 6

Regards,
Armin

[DominicTNC]

Hi Armin,

This gives me permission error. I used to run script using "sh" instead of "./", though I don't know why this works.

Thanks.

Dominic

[Armin.Sh]

Hi Dominic,

okay. Then please give this first:

chmod +x parReconstructPar.sh

and afterwards this:

./parReconstructPar.sh -n 6

I hope it works now!

Regards,
Armin

[DominicTNC]

Hi Armin,

Thank you very much! It works!

On the other hand, do you have any experience running this script on LSF platform? Because I will need to do so in the future.

Many Thanks.

Dominic