[maka]

I'm decomposing a channel (with periodic boundary conditions. The mesh is done blockMesh) that is 64x24x64. The decomposition takes place in the streamwise direction. I expected that for any processor i the no of faces shared with processor j is equal for all processors. wall boundary faces can off course differ in the number of cells.

What I got is not normal. Note the shared no of cells is halfed when going from 2 to 3 processor. Can any body help me explain why this happens? Thanks in advance.

2 processors:
Processor 0
Number of cells = 98304
Number of faces shared with processor 1 = 6144
Number of boundary faces = 7168
Processor 1
Number of cells = 98304
Number of faces shared with processor 0 = 6144
Number of boundary faces = 7168
--------------------------------------------------
3 processors
Processor 0
Number of cells = 65536
Number of faces shared with processor 1 = 3188
Number of faces shared with processor 2 = 3072
Number of boundary faces = 4746
Processor 1
Number of cells = 65536
Number of faces shared with processor 0 = 3188
Number of faces shared with processor 2 = 3188
Number of boundary faces = 4652
Processor 2
Number of cells = 65536
Number of faces shared with processor 1 = 3188
Number of faces shared with processor 0 = 3072
Number of boundary faces = 4746
--------------------------------------------------
4 processors:
Processor 0
Number of cells = 49152
Number of faces shared with processor 1 = 3072
Number of faces shared with processor 3 = 3072
Number of boundary faces = 3584
Processor 1
Number of cells = 49152
Number of faces shared with processor 0 = 3072
Number of faces shared with processor 2 = 3072
Number of boundary faces = 3584
Processor 2
Number of cells = 49152
Number of faces shared with processor 1 = 3072
Number of faces shared with processor 3 = 3072
Number of boundary faces = 3584
Processor 3
Number of cells = 49152
Number of faces shared with processor 2 = 3072
Number of faces shared with processor 0 = 3072
Number of boundary faces = 3584
--------------------------------------------------
5 processors:
Processor 0
Number of cells = 39322
Number of faces shared with processor 1 = 3188
Number of faces shared with processor 4 = 3072
Number of boundary faces = 2790

Processor 1
Number of cells = 39322
Number of faces shared with processor 0 = 3188
Number of faces shared with processor 2 = 3189
Number of boundary faces = 2791
Processor 2
Number of cells = 39322
Number of faces shared with processor 1 = 3189
Number of faces shared with processor 3 = 3188
Number of boundary faces = 2791
Processor 3
Number of cells = 39321
Number of faces shared with processor 2 = 3188
Number of faces shared with processor 4 = 3188
Number of boundary faces = 2790
Processor 4
Number of cells = 39321
Number of faces shared with processor 3 = 3188
Number of faces shared with processor 0 = 3072
Number of boundary faces = 2790

--------------------------------------------------

[eugene]

Because periodic boundaries become processor boundaries if they have to communicate between processors, i.e. the two sides of the cyclic boundary no longer reside on the same processor.

[maka]

I think you refer to the cyclic boundary that is normal to the direction I decompose in (x). I can not see why other cyclic boundries does not belong to the same processor as long as they are parllel to the x direction, since they will be communicated within the same processor. To make my point more clear, I expected that, when decomposing in x direction, Number of faces shared with processor i = Ny*Nz = 24*64 = 1536 = constant (no matter how many domain cuts I have). Please detail more for me to understand. Thanks for your reply

[eugene]

Here is a 1d domain:

|--------|

It contains 8 cells and 2 cyclic boundaries.

Here is the same domain decomposed into 2:

|----||----|

Each domain now talks to the other domain through 2 boundary faces, the ones in the middle and the cyclics which now have become "processor" boundaries.

Here is the same domain decomposed into 4:

|--||--||--||--|

As you can see each domain communicates with its neighbour through only a single shared face. This inlcudes the first and the last domains which talk to each other through the cyclic-processor boundaries.

No matter how many decompositions you make after 2 the number of faces will remain constant. Two domains is different because the cyclics act as processor boundaries as well, doubling the number of interprocessor faces.

For the channel flow example the boundaries in both the x and z directions are cyclics.

As to the small variations in the number of shared faces at higher processor counts, this is due to tolerance issues that cause an unequal number of cells to be distributed. Try hierarchical instead of simple decomposition method and/or decrease the decomposition tolerance "delta".

Clear?

[maka]

Yes. Many thanks.

[msang]

Hi All,

I am having problems running decomposePar with merged patchs. The trace is from a simple test case. This test case is just the icoFoam cavity with two blocks merged (it works fine with one processor). I am using OF 1.4. Does anyone have any ideas what my problem is?

Thanks in advance

Mark

m3:run$decomposePar . cavity-2blocks
/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.4 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : decomposePar . cavity-2blocks
Date : Jul 21 2007
Time : 17:12:48
Host : m3
PID : 8943
Root : /home/marks/OpenFOAM/marks-1.4/run
Case : cavity-2blocks
Nprocs : 1
Create time

Time = 0
Create mesh


Calculating distribution of cells
Selecting decompositionMethod simple

Finished decomposition in 0 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Calculating processor boundary addressing

Distributing points to processors

Constructing processor meshes
*** glibc detected *** decomposePar: malloc(): memory corruption: 0x0000000000843520 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2af1aa1958fe]
/lib64/libc.so.6[0x2af1aa197787]
/lib64/libc.so.6(__libc_malloc+0x86)[0x2af1aa199386]
/usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x2af1a9a80fcd]
/usr/lib64/libstdc++.so.6(_Znam+0x9)[0x2af1a9a810e9]
decomposePar(_ZN4Foam4ListIiEC1EiRKi+0x5c)[0x43e2bc]
decomposePar[0x4d8f1f]
decomposePar[0x43ffab]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2af1aa146ae4]
decomposePar(_ZN4Foam6fvMesh10readUpdateEv+0x169)[0x43ad49]
======= Memory map: ========
00400000-00511000 r-xp 00000000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar
00711000-00719000 r--p 00111000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar
00719000-0071a000 rw-p 00119000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar
0071a000-0086c000 rw-p 0071a000 00:00 0 [heap]
2af1a7b4a000-2af1a7b66000 r-xp 00000000 08:05 2002754 /lib64/ld-2.5.so
2af1a7b66000-2af1a7b68000 rw-p 2af1a7b66000 00:00 0
2af1a7d66000-2af1a7d68000 rw-p 0001c000 08:05 2002754 /lib64/ld-2.5.so
2af1a7d68000-2af1a8741000 r-xp 00000000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a8741000-2af1a8940000 ---p 009d9000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a8940000-2af1a8968000 r--p 009d8000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a8968000-2af1a896f000 rw-p 00a00000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a896f000-2af1a8973000 rw-p 2af1a896f000 00:00 0
2af1a8973000-2af1a8999000 r-xp 00000000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o
2af1a8999000-2af1a8a98000 ---p 00026000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o
2af1a8a98000-2af1a8a9a000 rw-p 00025000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o
2af1a8a9a000-2af1a8a9c000 r-xp 00000000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8a9c000-2af1a8c9b000 ---p 00002000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8c9b000-2af1a8c9c000 r--p 00001000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8c9c000-2af1a8c9d000 rw-p 00002000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8c9d000-2af1a8c9e000 rw-p 2af1a8c9d000 00:00 0
2af1a8c9e000-2af1a8e16000 r-xp 00000000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a8e16000-2af1a9015000 ---p 00178000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a9015000-2af1a9019000 r--p 00177000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a9019000-2af1a901c000 rw-p 0017b000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a901c000-2af1a901d000 rw-p 2af1a901c000 00:00 0
2af1a901d000-2af1a901e000 r-xp 00000000 08:06 970184 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfoamUtil.so
2af1a901e000-2af1a921d000 ---p 00001000 08:06 970184 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfoamUtil.so
2af1a921d000-2af1a921e000 r--p 00000000 08:06 Aborted
m3:run$

[andrea]

Hi Marks,
hope you're already solved the problem, but maybe it's useful for others.
I solved just by deleting non essential files as a cellZone and the sets folder.

Andrea