CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM

Parallel refineMesh with Cyclic Boundaries

Register Blogs Community New Posts Updated Threads Search

Like Tree6Likes
  • 1 Post By wyldckat
  • 2 Post By mchurchf
  • 2 Post By mchurchf
  • 1 Post By wyldckat

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   July 20, 2011, 15:46
Default Parallel refineMesh with Cyclic Boundaries
  #1
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 19
mchurchf is on a distinguished road
I want to make a really large block mesh with uniform grid resolution throughout and cyclic boundaries. When I say really large, I mean hundreds of millions or a billion cells. My machines do not have enough memory to run blockMesh and decomposePar, both of which are serial executable only.

Therefore, I have been experimenting with using blockMesh to make a coarse mesh, decomposing that, and then using refineMesh over the entire domain in parallel to refine each piece of the mesh to create the really large mesh. This makes the memory demands on blockMesh and decomposePar smaller. refineMesh is run in parallel, so memory is distributed over a lot of nodes making the job manageable.

The problem is when there are cyclic boundaries. refineMesh seems to not handle cyclic boundaries in parallel. Any suggestions on what I'm trying to do?
mchurchf is offline   Reply With Quote

Old   July 20, 2011, 19:22
Default
  #2
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,981
Blog Entries: 45
Rep Power: 128
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Greetings Matthew,

I'm sorry for not going into details, but here goes:
  1. Preserve cyclic boundaries: Cyclic patches and parallel postprocessing problems post #8
  2. Now I can't remember very well, but I vaguely remember something about tolerance for compatibility between cyclic patches... maybe this one can shed some light: http://www.cfd-online.com/Forums/ope...tml#post276931 post #2
  3. Here's another, this time a whole thread dedicated to this: http://www.cfd-online.com/Forums/ope...lic-patch.html
For more, search the forum for something like "tolerance cyclic patches".

Good luck!
Bruno
Pagoda likes this.
__________________
wyldckat is offline   Reply With Quote

Old   June 6, 2012, 12:49
Default
  #3
Member
 
Kim Yusik
Join Date: Dec 2009
Posts: 39
Rep Power: 17
impecca is on a distinguished road
Hi, Matthew.

Even you can refine the mesh using 'refineMesh' with parallel computing, presumably you still need to construct the subdomains in the end using 'reconstructPar'. and this is not running in parallel either as far as I know.

I am also strugling with deomposing and reconstructing large mesh. So I am wondering that you have an alternative solution to deal with a large mesh? (apart from buying more memory).
In my case, I can use upto 35 million cells with 48gb memory.
Regards

Yusik
impecca is offline   Reply With Quote

Old   June 6, 2012, 13:19
Default A Working Method for Creating Large Meshes in Parallel
  #4
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 19
mchurchf is on a distinguished road
Yusik,

I do have a method that works now. I create a coarse mesh with blockMesh. Then I decompose the domain with decomposePar, making sure to list the cyclic boundaries in the "preservePatches" keyword. Then I run refineMesh or refineHexMesh in parallel until I get to the desired grid resolution. I do not have the computer memory to reconstruct the domain with reconstructPar. I deal with that by sampling slices, probes, etc. while the solver is running. If I really need to explore the full 3D flow field, Paraview's OpenFOAM reader will read in each processor directory and display the flow field as one contiguous field. You need to copy the system/controlDict to system/controlDict.foam. After you open Paraview, you choose the controlDict.foam file as the file to open, which invokes the OpenFOAM reader. Then you'll have the option to view the decomposed domain (where each processor directory is read in). If you have a very large domain that requires a lot of memory, then you can launch Paraview in parallel to spread the memory requirements over multiple compute nodes.

Matt
wyldckat and mgg like this.
mchurchf is offline   Reply With Quote

Old   June 6, 2012, 17:47
Default
  #5
Member
 
Kim Yusik
Join Date: Dec 2009
Posts: 39
Rep Power: 17
impecca is on a distinguished road
Thank you for quick response. and I also agree that will be a good solution.
I did a quick test as you suggested and I have a related question.
I decomposed a coarse mesh into subdomains and then I ran

topoSet -parallel (for local refinement)
setsToZones -noFlipMap -parallel
refineMesh -dict -parallel

upto here, there is no problem. Then I realized that the refined mesh is
generated in a new time step within subdomains, e.g.
processor#/'newtimestep'/polyMesh

This means that the mesh is refined and decomposed but the fields
(U, p etc) are not. they remain the same in the old time step folder. So I am
wondering if there is a way to refine fields together with mesh? I've tried
to use couple of options such as '-overwrite' but it hasn't been successful so far.
Thank you in advance
Yusik
impecca is offline   Reply With Quote

Old   June 7, 2012, 10:31
Default
  #6
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 19
mchurchf is on a distinguished road
Yusik,

Yes, try using refineHexMesh. It is the newer refinement tool, and it will refine the solution fields, too. It's use is similar to refineMesh.

Matt
wyldckat and mgg like this.
mchurchf is offline   Reply With Quote

Old   June 20, 2012, 14:06
Default
  #7
Member
 
Kim Yusik
Join Date: Dec 2009
Posts: 39
Rep Power: 17
impecca is on a distinguished road
Thanks for your valuable comments.
I forgot to mention this.
impecca is offline   Reply With Quote

Old   November 28, 2018, 02:22
Default
  #8
New Member
 
David Smith
Join Date: Jul 2013
Posts: 9
Rep Power: 13
minh khang is on a distinguished road
Quote:
Originally Posted by impecca View Post
Hi, Matthew.

Even you can refine the mesh using 'refineMesh' with parallel computing, presumably you still need to construct the subdomains in the end using 'reconstructPar'. and this is not running in parallel either as far as I know.

I am also strugling with deomposing and reconstructing large mesh. So I am wondering that you have an alternative solution to deal with a large mesh? (apart from buying more memory).
In my case, I can use upto 35 million cells with 48gb memory.
Regards

Yusik
"'reconstructPar'. and this is not running in parallel either as far as I know." : So if we reconstructPar in parallel and then what we will have? have to reconstructPar in single processor to merge data together again?
minh khang is offline   Reply With Quote

Old   December 22, 2018, 12:11
Default
  #9
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,981
Blog Entries: 45
Rep Power: 128
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Quick answer: reconstructPar is sort-of the opposite of decomposePar:
  • decomposePar is meant to run in serial mode (not parallel) and it will decompose the domain into subdomains, so that the case can then be run in parallel; it decomposes both the mesh and the fields.
  • reconstructPar is also meant to run in serial mode and it will reconstruct the subdomains back into the main domain, but it will only reconstruct the fields and a few other things. But it will not reconstruct the mesh, for that you must use reconstructParMesh.
lukasf likes this.
__________________
wyldckat is offline   Reply With Quote

Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with cyclic boundaries in Openfoam 1.5 fs82 OpenFOAM 37 November 29, 2024 11:15
Cyclic patches and parallel postprocessing problems askjak OpenFOAM Bugs 18 October 27, 2010 04:35
[mesh manipulation] refineMesh -parallel crashes (in cavity tutorial) andras OpenFOAM Meshing & Mesh Conversion 0 August 8, 2010 15:52
Cyclic patch in parallel calculations didomenico OpenFOAM Running, Solving & CFD 4 March 7, 2007 06:46
Cyclic Boundaries -> Match Option -> Arbitrary Derek Siemens 1 August 4, 2004 23:06


All times are GMT -4. The time now is 13:56.