decomposePar over several disks

mturcios777 · October 17, 2011, 16:15

Greeting Foam Users,

I'm working on parallelization some of our cases and solvers and was looking to ask questions about distributing data over several disks. I have two multi-core machines I have made into "cluster" and am testing decomposed cases. Just using the built in network cards to communicate (these are Dell Precision T5500 with Broadcom BCM5761 and an ethernet crossover cable). what are some tips to reduce the overhead so that communication quick, or some inexpensive modifications that I could make to improve performance.

I was also curious about different methods for domain decomposition. Specifically, how the manual option works (with examples if possible), and if the simple/hierarchical methods can be controlled by subdomain (ie, after the first division of a domain in two equal parts, can I further subdivide one of the sub-domains)?

Thanks!

wyldckat · October 18, 2011, 18:12

Hi Marco,

I'm going to have to be quick... so here goes...

The documented version of said dictionary can be found this way:

Code:

find $WM_PROJECT_DIR/applications -name decomposeParDict

Here you'll find a lot about this dictionary

All other examples can be found this way:

Code:

find $WM_PROJECT_DIR/tutorials -name decomposeParDict

As for dumping the cell<->processor association, I believe this should help: http://openfoamwiki.net/index.php/Ti...gisteredObject

As for a whole lot of other notes about OpenFOAM+parallel: http://www.cfd-online.com/Forums/blo...-parallel.html

Best regards,
Bruno

mturcios777 · October 19, 2011, 13:59

Many thanks!

mturcios777 · October 20, 2011, 19:07

There was a message here about ParaView having a "decomposed case" option, but that has since been deleted. I looked at my current version of ParaView (3.10.1) and was not able to finde the decomposed case option. Does anyone else know about this?

With the paraFoam -touch option, I am able to load all the processors into one instance of ParaView and do post-processing on the entire domain. The only problem is that application of filters has to be done on each processor in turn. Fine for domains of 5 processors or less, but this could get unwieldy (at least the way I'm doing it). We can apply multiple filters to single sources, can we apply a single filter and take as input multiple sources?

wyldckat · October 20, 2011, 19:44

Hi Marco,

Quote:

Originally Posted by mturcios777

There was a message here about ParaView having a "decomposed case" option, but that has since been deleted. I looked at my current version of ParaView (3.10.1) and was not able to finde the decomposed case option. Does anyone else know about this?

Strange... why would it be deleted? Wasn't it moved?
But if you are using OpenFOAM 2.0, run paraFoam like this:

Code:

paraFoam -builtin

It will open the case with the internal reader that ParaView has, namely using the file extension ".foam", instead of ".OpenFOAM".

Quote:

Originally Posted by mturcios777

With the paraFoam -touch option, I am able to load all the processors into one instance of ParaView and do post-processing on the entire domain. The only problem is that application of filters has to be done on each processor in turn. Fine for domains of 5 processors or less, but this could get unwieldy (at least the way I'm doing it). We can apply multiple filters to single sources, can we apply a single filter and take as input multiple sources?

There's a filter that will join multiple sources into a single source... I think it's "Group Datasets". Then you can simply manipulate the resulting object.

Best regards,
Bruno

mturcios777 · October 21, 2011, 13:57

Beautiful. I need to play with that feature a bit more, as it seems to have some trouble synchronizing the time directories (it only gives me 0, 0.1111... and 1). The group datasets feature does what I need, albeit with a little bit more work; I'll use it until I can get the built-in reader to work properly.

Much thanks!

October 17, 2011, 16:15	decomposePar over several disks	#1
mturcios777 Senior Member Marco A. Turcios Join Date: Mar 2009 Location: Vancouver, BC, Canada Posts: 740 Rep Power: 28	Greeting Foam Users, I'm working on parallelization some of our cases and solvers and was looking to ask questions about distributing data over several disks. I have two multi-core machines I have made into "cluster" and am testing decomposed cases. Just using the built in network cards to communicate (these are Dell Precision T5500 with Broadcom BCM5761 and an ethernet crossover cable). what are some tips to reduce the overhead so that communication quick, or some inexpensive modifications that I could make to improve performance. I was also curious about different methods for domain decomposition. Specifically, how the manual option works (with examples if possible), and if the simple/hierarchical methods can be controlled by subdomain (ie, after the first division of a domain in two equal parts, can I further subdivide one of the sub-domains)? Thanks! Last edited by mturcios777; October 17, 2011 at 21:03. Reason: Added question about manual method and sub-domain control

October 18, 2011, 18:12		#2
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,981 Blog Entries: 45 Rep Power: 128	Hi Marco, I'm going to have to be quick... so here goes... The documented version of said dictionary can be found this way: Code: find $WM_PROJECT_DIR/applications -name decomposeParDict Here you'll find a lot about this dictionary All other examples can be found this way: Code: find $WM_PROJECT_DIR/tutorials -name decomposeParDict As for dumping the cell<->processor association, I believe this should help: http://openfoamwiki.net/index.php/Ti...gisteredObject As for a whole lot of other notes about OpenFOAM+parallel: http://www.cfd-online.com/Forums/blo...-parallel.html Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
decomposePar gives errors	of_user_	OpenFOAM	1	July 4, 2011 06:27
decomposePar: can use this decomposition method only for the whole mesh	aloeven	OpenFOAM Bugs	0	March 16, 2011 11:15
Strange behaviour 1.6 decomposePar vs 1.7 decomposePar	BlueyTheDog	OpenFOAM	7	January 16, 2011 19:12
DecomposePar links against liblamso0 with OpenMPI	jens_klostermann	OpenFOAM Bugs	11	June 28, 2007 18:51
axisymmetric model of two rotating disks cavity	liaolingling	FLUENT	0	April 27, 2005 05:24

October 19, 2011, 13:59		#3
mturcios777 Senior Member Marco A. Turcios Join Date: Mar 2009 Location: Vancouver, BC, Canada Posts: 740 Rep Power: 28	Many thanks!

October 20, 2011, 19:07		#4
mturcios777 Senior Member Marco A. Turcios Join Date: Mar 2009 Location: Vancouver, BC, Canada Posts: 740 Rep Power: 28	There was a message here about ParaView having a "decomposed case" option, but that has since been deleted. I looked at my current version of ParaView (3.10.1) and was not able to finde the decomposed case option. Does anyone else know about this? With the paraFoam -touch option, I am able to load all the processors into one instance of ParaView and do post-processing on the entire domain. The only problem is that application of filters has to be done on each processor in turn. Fine for domains of 5 processors or less, but this could get unwieldy (at least the way I'm doing it). We can apply multiple filters to single sources, can we apply a single filter and take as input multiple sources?

October 21, 2011, 13:57		#6
mturcios777 Senior Member Marco A. Turcios Join Date: Mar 2009 Location: Vancouver, BC, Canada Posts: 740 Rep Power: 28	Beautiful. I need to play with that feature a bit more, as it seems to have some trouble synchronizing the time directories (it only gives me 0, 0.1111... and 1). The group datasets feature does what I need, albeit with a little bit more work; I'll use it until I can get the built-in reader to work properly. Much thanks!