manual decomposition for topological changes

kdneroorkar · September 1, 2011, 11:20

I have read some posts that 1.7-ext might have topological changes implemented in parallel. I am currently using 1.6-ext and
I was wondering if there was a way in which I could manually decompose the mesh so that all the topological changes were handled by one processor and the rest of the mesh is handled by other processors.

deepsterblue · September 1, 2011, 13:44

1.6-ext already has topo-changes in parallel. Nevertheless, you can always partition a mesh in specific ways by supplying a processor addressing list during decomposePar (i.e., processor ID for each cell in the mesh - like an output from Metis). However, even if topo-changes occur on a single processor, the processor patch addressing can (and usually does) change, which requires a global sync prior to solution restart. Keeping all topo-changes specific to a single processor is also a bad idea from a load-balancing stand-point.

kdneroorkar · September 1, 2011, 14:11

Thanks Sandeep for that information. Based on the thread
http://www.cfd-online.com/Forums/ope...changes-4.html

I thought that parallel topo changes were going to be in the next release.
I do agree that what I am suggesting will be bad from load balancing point of view, but I am trying to run a pretty big case, and I wanted to know if there was any way of running it in parallel.

Peter_600 · September 6, 2011, 15:40

Hi, I ve followed this thread am I am also interested in parallel computing with topochanger. How can we remedy the issue with the addressing?
I ve noticed that after a topological change the mesh has not anymore the same cell and face zones as the initial one. I am doing engine simulations as well and by using for instance the engineTopoChangerMesh class, each time folder during a run contains a folder with the new mesh. Unfortunately, it does not anymore contain the nicely created zones of my initial mesh. I am aware of it that this will cause problems if I wanna simulate in parallel.
So, is there a way to get back the cell and face zones on the fly?

thx

deepsterblue · September 6, 2011, 15:46

Not really sure what you're talking about - what issues with addressing? If there's an error generated, please post that, so that we have an idea about what you're referring to. If the zone addressing is not being preserved after topo-changes, and you feel that something is wrong, please refer the problem to Hrv, who's written the relevant bits of code.

kdneroorkar · September 6, 2011, 16:12

Hi Peter
I am using sliding interfaces and layering in my simulation. And the zones seem to be being tracked correctly during the serial simulation. When I do foamToVTK, I can see these zones in paraview too.

Is it possible that there is something else wrong in your simulation?

Peter_600 · September 7, 2011, 08:34

Sorry I expressed myself a little bit unclear.
Everything works fine. I just want to know if there are already some ways to decompose the mesh nicely to use topochanger in parallel.

deepsterblue · September 7, 2011, 10:06

Define "nicely"

Peter_600 · September 7, 2011, 14:13

Hi Sandeep

I don t know how I can decompose the mesh from my existing zones.
Considering an engine, I would like to have the ports and the cylinder seperated. You may have heard about enginescotch.
http://www.openfoamworkshop.org/6th_...sak_slides.pdf -> slide 15
I would like to decompose my case that way.
Thanks in advance
Peter

kdneroorkar · September 9, 2011, 11:02

Hi Peter
This looks really nice, thanks for sending this presentation. I also would be interested in this. If you find out anything more, please let me know. my email address is kneroork@engin.umass.edu
Thanks
Kshitij

Peter_600 · September 10, 2011, 08:41

Unfortunately not. I was thinking of writing my own decomposition method like the engineScotch.
Any suggestions how to start?

deepsterblue · September 12, 2011, 18:10

As far as I can tell, it looks like a situation where CPU id's are specifically assigned based on mesh connectivity (or even geometry). Once this assignment is done, fvMeshDistribute can do the re-distribution bit quite easily. It wouldn't hold much water from a load-balancing point of view, but it gets the job done.

Peter_600 · September 13, 2011, 03:34

Thx, for the hints. I ll give it a try.
Peter

kdneroorkar · December 9, 2011, 10:42

Hi Peter
I found out with Kalle's help that if the processor boundaries are perpendicular to the face where layerAR is working, then the topochanger will work in parallel (look at the picture on page 15 of the presentation). To do this you need to do a manual decomposition of the domain.
Hope this helps
Kshitij

Peter_600 · December 10, 2011, 05:28

Hi,
thank you for the answer. Indeed, it runs

Peter_600 · December 10, 2011, 07:28

I was using scotch for the decomposition of the mesh. How do you decompose manually?

September 1, 2011, 11:20	manual decomposition for topological changes	#1
kdneroorkar Member kshitij neroorkar Join Date: Mar 2009 Location: Michigan, USA Posts: 32 Rep Power: 17	I have read some posts that 1.7-ext might have topological changes implemented in parallel. I am currently using 1.6-ext and I was wondering if there was a way in which I could manually decompose the mesh so that all the topological changes were handled by one processor and the rest of the mesh is handled by other processors.

September 1, 2011, 13:44		#2
deepsterblue Senior Member Sandeep Menon Join Date: Mar 2009 Location: Amherst, MA Posts: 403 Rep Power: 25	1.6-ext already has topo-changes in parallel. Nevertheless, you can always partition a mesh in specific ways by supplying a processor addressing list during decomposePar (i.e., processor ID for each cell in the mesh - like an output from Metis). However, even if topo-changes occur on a single processor, the processor patch addressing can (and usually does) change, which requires a global sync prior to solution restart. Keeping all topo-changes specific to a single processor is also a bad idea from a load-balancing stand-point. __________________ Sandeep Menon University of Massachusetts Amherst https://github.com/smenon

September 6, 2011, 15:46		#5
deepsterblue Senior Member Sandeep Menon Join Date: Mar 2009 Location: Amherst, MA Posts: 403 Rep Power: 25	Not really sure what you're talking about - what issues with addressing? If there's an error generated, please post that, so that we have an idea about what you're referring to. If the zone addressing is not being preserved after topo-changes, and you feel that something is wrong, please refer the problem to Hrv, who's written the relevant bits of code. __________________ Sandeep Menon University of Massachusetts Amherst https://github.com/smenon

September 7, 2011, 10:06		#8
deepsterblue Senior Member Sandeep Menon Join Date: Mar 2009 Location: Amherst, MA Posts: 403 Rep Power: 25	Define "nicely" __________________ Sandeep Menon University of Massachusetts Amherst https://github.com/smenon

September 12, 2011, 18:10		#12
deepsterblue Senior Member Sandeep Menon Join Date: Mar 2009 Location: Amherst, MA Posts: 403 Rep Power: 25	As far as I can tell, it looks like a situation where CPU id's are specifically assigned based on mesh connectivity (or even geometry). Once this assignment is done, fvMeshDistribute can do the re-distribution bit quite easily. It wouldn't hold much water from a load-balancing point of view, but it gets the job done. __________________ Sandeep Menon University of Massachusetts Amherst https://github.com/smenon

September 1, 2011, 14:11		#3
kdneroorkar Member kshitij neroorkar Join Date: Mar 2009 Location: Michigan, USA Posts: 32 Rep Power: 17	Thanks Sandeep for that information. Based on the thread http://www.cfd-online.com/Forums/ope...changes-4.html I thought that parallel topo changes were going to be in the next release. I do agree that what I am suggesting will be bad from load balancing point of view, but I am trying to run a pretty big case, and I wanted to know if there was any way of running it in parallel.

September 6, 2011, 15:40		#4
Peter_600 Senior Member Join Date: Oct 2009 Posts: 140 Rep Power: 17	Hi, I ve followed this thread am I am also interested in parallel computing with topochanger. How can we remedy the issue with the addressing? I ve noticed that after a topological change the mesh has not anymore the same cell and face zones as the initial one. I am doing engine simulations as well and by using for instance the engineTopoChangerMesh class, each time folder during a run contains a folder with the new mesh. Unfortunately, it does not anymore contain the nicely created zones of my initial mesh. I am aware of it that this will cause problems if I wanna simulate in parallel. So, is there a way to get back the cell and face zones on the fly? thx

September 6, 2011, 16:12		#6
kdneroorkar Member kshitij neroorkar Join Date: Mar 2009 Location: Michigan, USA Posts: 32 Rep Power: 17	Hi Peter I am using sliding interfaces and layering in my simulation. And the zones seem to be being tracked correctly during the serial simulation. When I do foamToVTK, I can see these zones in paraview too. Is it possible that there is something else wrong in your simulation?

September 7, 2011, 08:34		#7
Peter_600 Senior Member Join Date: Oct 2009 Posts: 140 Rep Power: 17	Sorry I expressed myself a little bit unclear. Everything works fine. I just want to know if there are already some ways to decompose the mesh nicely to use topochanger in parallel.

September 7, 2011, 14:13		#9
Peter_600 Senior Member Join Date: Oct 2009 Posts: 140 Rep Power: 17	Hi Sandeep I don t know how I can decompose the mesh from my existing zones. Considering an engine, I would like to have the ports and the cylinder seperated. You may have heard about enginescotch. http://www.openfoamworkshop.org/6th_...sak_slides.pdf -> slide 15 I would like to decompose my case that way. Thanks in advance Peter

September 9, 2011, 11:02		#10
kdneroorkar Member kshitij neroorkar Join Date: Mar 2009 Location: Michigan, USA Posts: 32 Rep Power: 17	Hi Peter This looks really nice, thanks for sending this presentation. I also would be interested in this. If you find out anything more, please let me know. my email address is kneroork@engin.umass.edu Thanks Kshitij

September 10, 2011, 08:41		#11
Peter_600 Senior Member Join Date: Oct 2009 Posts: 140 Rep Power: 17	Unfortunately not. I was thinking of writing my own decomposition method like the engineScotch. Any suggestions how to start?

September 13, 2011, 03:34		#13
Peter_600 Senior Member Join Date: Oct 2009 Posts: 140 Rep Power: 17	Thx, for the hints. I ll give it a try. Peter

December 9, 2011, 10:42		#14
kdneroorkar Member kshitij neroorkar Join Date: Mar 2009 Location: Michigan, USA Posts: 32 Rep Power: 17	Hi Peter I found out with Kalle's help that if the processor boundaries are perpendicular to the face where layerAR is working, then the topochanger will work in parallel (look at the picture on page 15 of the presentation). To do this you need to do a manual decomposition of the domain. Hope this helps Kshitij

December 10, 2011, 05:28		#15
Peter_600 Senior Member Join Date: Oct 2009 Posts: 140 Rep Power: 17	Hi, thank you for the answer. Indeed, it runs

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Description file for manual decomposition	A.Devesa	OpenFOAM Running, Solving & CFD	2	July 4, 2011 09:09
Flow path control manual edit in autogrid	njiang	Fidelity CFD	0	June 22, 2011 19:03
CCM+ and manual decomposition	William Blake	Siemens	3	June 29, 2006 17:43
Can somebody send me a Fluent 6 UDF manual??	KKLAU	FLUENT	4	April 14, 2004 17:37

December 10, 2011, 07:28		#16
Peter_600 Senior Member Join Date: Oct 2009 Posts: 140 Rep Power: 17	I was using scotch for the decomposition of the mesh. How do you decompose manually?