[6863523]

Dear All,
I am trying to run cases in OpenFOAM, here is the configuration of the server I am using.

 ywang@srv-calmipo01 ~/OpenFOAM/ywang-3.0.1/run/UCUCUC/Re17500 $ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Stepping:              1
CPU MHz:               2399.976
BogoMIPS:              4396.31
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39

Q1: Here is the decomposeParDict I want to use, and I want to make the most use of the server. What should I modify about file?

 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  3.0.1                                 |
|   \\  /    A nd           | Web:      [IMG]file:///C:\Users\wyb\AppData\Local\Temp\%W@GJ$ACOF(TYDYECOKVDYB.png[/IMG]www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains  12;

method          scotch;


multiLevelCoeffs
{
    
    level0
    {
        numberOfSubdomains  64;
         method scotch;
    }
    level1
    {
        numberOfSubdomains  4;
        method scotch;
    }
}

simpleCoeffs
{
    n           (2 1 1);
    delta       0.001;
}

hierarchicalCoeffs
{
    n           (1 2 1);
    delta       0.001;
    order       xyz;
}

metisCoeffs
{
 /*
    processorWeights
    (
        1
        1
        1
        1
    );
  */
}

scotchCoeffs
{
   
}
manualCoeffs
{
    dataFile    "decompositionData";
}

structuredCoeffs
{
    patches     (movingWall);
    method      scotch;
}

// ************************************************************************* //

After the decomposePar command, I entered
mpirun -np 12 icoFoam -parallel > log
Q2: This running process will be saved in the log file, but I also want to see them running on the screen. What should I add to the command?

I want to store the solution according to the controlDict file below

 // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application     myIcoFoam;

startFrom       startTime;

startTime       0;

stopAt          endTime;

endTime         4000;

deltaT          0.0125;

writeControl    timeStep;

writeInterval   20000;

purgeWrite      0;

writeFormat     ascii;

writePrecision  6;

writeCompression off;

timeFormat      general;

timePrecision   6;

runTimeModifiable true;

Q3: But if I run in parallel, the files will be saved automatically in separate folders, named Processor0, processor1, ...What is the difference between these separate files and those files obtained without parallel?

Q4: How can I continue the simulations based on these files, if I do not use parallel running?

Thank you in advance.
Best Regards,
Bill

[GerhardHolzinger]

Originally Posted by 6863523

Dear All,
I am trying to run cases in OpenFOAM, here is the configuration of the server I am using.
Q1: Here is the decomposeParDict I want to use, and I want to make the most use of the server. What should I modify about file?

Q2: This running process will be saved in the log file, but I also want to see them running on the screen. What should I add to the command?

Q3: But if I run in parallel, the files will be saved automatically in separate folders, named Processor0, processor1, ...What is the difference between these separate files and those files obtained without parallel?

Q4: How can I continue the simulations based on these files, if I do not use parallel running?

Thank you in advance.
Best Regards,
Bill

Q1: The setting numberOfSubdomains controls how many parallel processes will be used in the simulation. You can tune parallel performance by playing around with the decomposition. Dividing the load (the number of processes) is one thing to consider, the extra amount of work (communication between the parallel processes) is another topic. Both have influence, but the latter can be very tricky. It is hard to give a general answer, other than do your own testing and do more research.

Q2: Having the solver output permanently on the screen seems to slow the simulation down. Redirect the solver output to a file, and display the solver output only on demand, e.g. by the tail command or by pyFoamPlotWatcher.py

Q3: A parallel simulation is a large task done is smaller chunks. This is why you do domain decomposition. The sum of the content of the processor* directories is your original case. After you finished, you can run reconstructPar to recombine the small chunks into your large simulation again.

Q4: What is the point of continuing a simulation you did in parallel before, in serial (one process)? How you do it? Run reconstructPar, make sure startTime in controlDict is set to latestTime, check the settings for ending the simulation (stopAt and stopTime), and run the solver.
What makes more sense is to change the number of parallel processes, e.g. because you initially used to little or too much. Recombine, decompose with the new number of subdomains, and continue the parallel run with the new number of parallel processes.

[6863523]

Dear Dr. Holzinger,
Thanks for the quick reply.
Q1: I will test to see what is the best for my cases.
Q2: Thank you for the suggestions.
Q3: That's correct!!! I looked through the forum and found reconstructPar function. I know what to do after parallel running.
Q4: Based on the answer for Q3, I understand my problem now and thanks for the instructions.

Thanks for the detailed instructions. If I understand your point and the computer configuration correctly, I may start from

 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  3.0.1                                 |
|   \\  /    A nd           | Web:       [IMG]file:///C:\Users\wyb\AppData\Local\Temp\%W@GJ$ACOF(TYDYECOKVDYB.png[/IMG]www.OpenFOAM.org                       |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 20;
method          scotch;
multiLevelCoeffs
{    
    level0
    {
        numberOfSubdomains  10; //changed from 64 to 10
         method scotch;
    }
    level1
    {
        numberOfSubdomains  8; //changed from 4 to 8
        method scotch;
    }
}

simpleCoeffs
{
    n           (2 1 1);
    delta       0.001;
}

hierarchicalCoeffs
{
    n           (1 2 1);
    delta       0.001;
    order       xyz;
}

metisCoeffs
{
 /*
    processorWeights
    (
        1
        1
        1
        1
    );
  */
}

scotchCoeffs
{
   
}
manualCoeffs
{
    dataFile    "decompositionData";
}

structuredCoeffs
{
    patches     (movingWall);
    method      scotch;
}

// ************************************************************************* //

Actually, I am confused on how to link the numberOfSubdomains to the computer configurations, as shown in the red parts.
Thank you.
Best Regards,
Bill

[GerhardHolzinger]

Originally Posted by 6863523

Dear Dr. Holzinger,
Thanks for the quick reply.
Q1: I will test to see what is the best for my cases.
Q2: Thank you for the suggestions.
Q3: That's correct!!! I looked through the forum and found reconstructPar function. I know what to do after parallel running.
Q4: Based on the answer for Q3, I understand my problem now and thanks for the instructions.

Thanks for the detailed instructions. If I understand your point and the computer configuration correctly, I may start from

 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  3.0.1                                 |
|   \\  /    A nd           | Web:       [IMG]file:///C:\Users\wyb\AppData\Local\Temp\%W@GJ$ACOF(TYDYECOKVDYB.png[/IMG]www.OpenFOAM.org                       |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 20;
method          scotch;

scotchCoeffs
{
}


// ************************************************************************* //

Actually, I am confused on how to link the numberOfSubdomains to the computer configurations, as shown in the red parts.
Thank you.
Best Regards,
Bill

The numberOfSubdomains should be less or equal to the number of CPU cores.
Since you use the scotch method for decomposition, you can delete all other entries in the decomposeParDict.

When you use a method X, then the XCoeffs are relevant. Using scotch and changing the multiLevelCoeffs makes no sense and will change nothing.

First, you may want to stick to the scotch method, as it handles decomposition for you. Try different numbers of processes to see where the sweetspot lies. If the case is large enough, conducting parallel simulation will speed-up your calculations. As you increase the number of parallel processes the speed-up will increase up to the point, when the additional work (communication between the processes) will become dominant.

[6863523]

Dear Dr. Holzinger,,
Thank you for your clear explanations, they are really very helpful to me!!!
I start from

numberofSubdomains 40;

and set all scotchCoeffs to 1 to make a test.

Since I used cyclic boundary, the decompose method is recommended here: https://github.com/OpenFOAM/OpenFOAM...composeParDict
Thank you in advance.
Best Regards,
Bill

[6863523]

Dear,
I have tested the numberofSubdomains(n) with several numbers. Here are some results I have got.

n=8, for t=3.7s ClockTime=863s;
ClockTime/t ≈ 233;

n=12, for t=5s, ClockTime=916s;
         for t=7.538s, ClockTime=1363s;
         for t=10s, ClockTime=1803s;
         for t=10.425s, ClockTime=1898s;
ClockTime/t ≈ 180;

n=15, for t=5s, ClockTime=909s;
         for t=7.5s, ClockTime=1375s;
         for t=10s, ClockTime=1802s;
ClockTime/t ≈ 182

n=20, for t=2.325s, ClockTime=349s;         
         ClockTime/t ≈ 150
        for t=321.09s, ClockTime=21574s;
        ClockTime/t ≈ 67.2

It seems that when the numberofSubdomains=20, it works the fastest.

In addition, I would like to recommend the parallel tests mentioned here: Optimising the Parallelisation of OpenFOAM Simulations
And also the basic knowledge about parallel strategy here: http://quantumwise.com/documents/tut...rategy.systems
Thank you in advance.
Best Regards,
Bill