[romant]

Hej,

the cases that we want to simulate contains three different domains, one fluid and two solid domains. The solid domains are rather small compared to the fluid domain, this is also true for the cell count. When I want to decompose the case over let's say 100 cores, then I automatically decompose also the solid domains over 100 cores, resulting in a large number of interfaces which needs to send data back and forth and the amount of cells on every core is very small (much less than the recommended 20k to 50k per core), slowing down the overall computations.

I was wondering if there is a way to decompose the water over the 100 cores and then for example decompose the solid domains on only 8 cores for example? Has anybody experience with decomposition of multi domain cases for large simulations?

[timo_IHS]

Hi Roman,

I do not have experience with multidomain cases, but you could try a manual decomposition. Make a decomposition with e.g. 100 and then reduce the amount of solids to e.g. 2.

[KlasJ]

Hi Roman,

Perhaps you have already solved this issue, but if not perhaps this could be of help.

My experience with multidomain cases is that it is much easier to use a manual decomposition. What I usually do is to produce a list of all the cells for all the regions in my simulation. Then I supply the lists (one per region) to the manual mode of decompasePar. The decomposeParDict would read:

numberOfSubdomains N;
method          manual;
manualCoeffs
{
        dataFile "cellDecomposition";
}
distributed no;

and the cellDecomposition would look something like:

FoamFile
{
    version     2.0;
    format      ascii;
    class       labelList;
    location    "constant";
    object      cellDecomposition;
}

ZZ
(
X
....
X
)

where ZZ are the number of cells for this region and X are the processor numbers where you want this cell to be placed.

Now, the interesting part is how you produce such lists. When the problems have very specific geometries or specific interfaces I often prefer to determine the processor boundaries somewhat manual. For example, if you have repeating geometry (like a set of fuel pins in a reactor) the point of automatic decomposition is much smaller.

I use a utility application to perform the splitting using topoSets. The application reads some splitting coordinates from a dictionary and then computes the processor for each of the cells. If you are interested I could share this small application.

Best regards,

Klas

[romant]

Originally Posted by KlasJ

Hi Roman,

Perhaps you have already solved this issue, but if not perhaps this could be of help.

My experience with multidomain cases is that it is much easier to use a manual decomposition. What I usually do is to produce a list of all the cells for all the regions in my simulation. Then I supply the lists (one per region) to the manual mode of decompasePar. The decomposeParDict would read:

numberOfSubdomains N;
method          manual;
manualCoeffs
{
        dataFile "cellDecomposition";
}
distributed no;

and the cellDecomposition would look something like:

FoamFile
{
    version     2.0;
    format      ascii;
    class       labelList;
    location    "constant";
    object      cellDecomposition;
}

ZZ
(
X
....
X
)

where ZZ are the number of cells for this region and X are the processor numbers where you want this cell to be placed.

Now, the interesting part is how you produce such lists. When the problems have very specific geometries or specific interfaces I often prefer to determine the processor boundaries somewhat manual. For example, if you have repeating geometry (like a set of fuel pins in a reactor) the point of automatic decomposition is much smaller.

I use a utility application to perform the splitting using topoSets. The application reads some splitting coordinates from a dictionary and then computes the processor for each of the cells. If you are interested I could share this small application.

Best regards,

Klas

Hej Klas,

I would be very much interested in that application if you could share here that would be very nice.

[styleworker]

I'm very interested in this application, too. Would you mind sharing this application?

[ipedrojm]

Hello to you all,

I have a similar problem with OpenFOAM. I am working with chtMultiRegionFoam solver and everything works fine in parallel when I use the "same" number of processors for "each" region, e.g. 4 for the fluid region and 4 for the solid region and run the application with mpirun -n 4 chtMultiRegionFoam -parallel.

However, I am interested in running this solver with a different number of processors per region, e.g. 2 processors for the fluid region (left) and 1 processor for the solid region (right). I attached a figure in order to illustrate this: the fluid and solid regions are separated by a vertical line at the middle of the computational domain. Each colour designs a processor.

I decompose the computational domain as follows:

1) I use the "scotch" mode in the fluid region (left) with 2 processors with the -cellDist option. This creates a file "cellDecomposition" similar to the one described by Klas in "constant/fluid" directory, which is ready to be used with a "manual" decomposition method. The OF command is: decomposePar -region fluid -cellDist

2) I run the decomposePar command but now using the "manual" mode (using the cellDecomposition file created in previous step) and overwriting the previous decomposition. OF command: decomposePar -region fluid -force.

3) I repeat previous steps with the solid region (right), specifying only 1 processor for this region during step 1), and then specifying 2 processors in step 2), which gives a warning message as the second processor has 0 cells assigned. By doing this, I can run the solver using 2 processors and let 1 processor idle when solving for this region.

After doing that I can visualize the mesh, which gives the image I attached. I run the simulation using 2 processors, as the fluid (left) needs them, with the command: mpirun -n 2 chtMultiRegionFoam -parallel. However, the simulation fails because of MPI communications errors. This error makes sense to me as the chtMultiRegionFoam is solving "sequentially" (partitioned approach) each region and, therefore, if the user runs the application specifying 2 processors (using the mpirun command), each region "should" be using 2 processors...

Could you please, Klas and Roman or anyone else, help me with this issue? It seems that you do not encounter this problem in your cases. Probably I am missing something. Any help will be highly appreciated.

Regards,

[dasith0001]

Originally Posted by KlasJ

Hi Roman,

Perhaps you have already solved this issue, but if not perhaps this could be of help.

My experience with multidomain cases is that it is much easier to use a manual decomposition. What I usually do is to produce a list of all the cells for all the regions in my simulation. Then I supply the lists (one per region) to the manual mode of decompasePar. The decomposeParDict would read:

numberOfSubdomains N;
method          manual;
manualCoeffs
{
        dataFile "cellDecomposition";
}
distributed no;

and the cellDecomposition would look something like:

FoamFile
{
    version     2.0;
    format      ascii;
    class       labelList;
    location    "constant";
    object      cellDecomposition;
}

ZZ
(
X
....
X
)

where ZZ are the number of cells for this region and X are the processor numbers where you want this cell to be placed.

Now, the interesting part is how you produce such lists. When the problems have very specific geometries or specific interfaces I often prefer to determine the processor boundaries somewhat manual. For example, if you have repeating geometry (like a set of fuel pins in a reactor) the point of automatic decomposition is much smaller.

I use a utility application to perform the splitting using topoSets. The application reads some splitting coordinates from a dictionary and then computes the processor for each of the cells. If you are interested I could share this small application.

Best regards,

Klas

Hi Klas,

Would you be able to share the application? would be really really helpful.
Thank you

Dasith