CFD Online Discussion Forums - Parallel Computing decomposePar

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM Pre-Processing (https://www.cfd-online.com/Forums/openfoam-pre-processing/)

- - Parallel Computing decomposePar (https://www.cfd-online.com/Forums/openfoam-pre-processing/95340-parallel-computing-decomposepar.html)

Hi Bruno,
Thanks for your reply. I run snappyHexMesh in OpenFOAM-2.1.0 on Tianhe-1A, a supercomputer. So the memory shouldn't be a problem.
OMG! I will re-download a OpenFOAM-2.1.0 and try again, hope that it will work. Thank you again!
Best wishes!
Xiaow-g

Quote:

Originally Posted by wyldckat (Post 354054)

Greetings to all!

@xiaow_g: I know I've had a similar problem, where snappyHexMesh kept running at 100% in parallel and wouldn't continue, but I can't remember what the problem was. It might be a memory issue like Chris said, but it could also be a bug you might be triggering somehow.

It would be useful to know the Linux distribution and architecture (uname -m) you are using, as well how much RAM is available to your Linux installation (in case you are running it inside a virtual machine).

The other possibility is that you need to upgrade to OpenFOAM 2.1.x. Several bugs have been fixed since 2.1.0 was released and I vaguely remember having problems with snappyHexMesh in parallel with one versions that wasn't from git...

Best regards,
Bruno

Hi, as I'm experiencing the same error (http://www.cfd-online.com/Forums/ope...ssorx-0-a.html) is there any way to set up a case from scratch in such a way that this error here is avoided without the need to run changeDict?

Thank you!

strange behaviour DecomposePar

Hello OF experts and users,
please assist me in struggling the following problem:

during run of icoFoam i included my utility:

Code:

....

        runTime.write();

    #include "utility.H";



        Info<< "ExecutionTime = " << runTime.elapsedCpuTime() << " s"

            << "  ClockTime = " << runTime.elapsedClockTime() << " s"

            << nl << endl;

......

"utility. H" consists of one loop, that prints values of U in cell centres along Y direction in the center of the channel (dimensions: x=0..6,Y=-1..1,Z=0..3; Nx=120,Ny=64,Nz=120):

Code:

scalar x,y,z;



forAll(mesh.cellCentres(),ii)

{

    x=y=z=0;

    x=mesh.cellCentres()[ii].x();

    y=mesh.cellCentres()[ii].y();

    z=mesh.cellCentres()[ii].z();



      if(x<3.10&&x>3.03&&z<1.58&&z>1.55)

    {

     Info<<"ii="<<ii<<" y="<<y<<" u="<<U[ii]<<endl;

    }



}

in single run output is perfect: 64 values of U

Code:

ii=315457 y=-0.996054 u=(-0.00620626 6.79366e-05 0.0108981)

ii=315585 y=-0.987859 u=(-0.00614572 -0.000243301 0.0243829)

ii=315713 y=-0.979032 u=(-0.00606576 -0.000949787 0.0380821)

ii=315841 y=-0.969524 u=(-0.00595958 -0.00210503 0.0521407)

ii=315969 y=-0.959283 u=(-0.00582333 -0.00378294 0.0664832)

ii=316097 y=-0.948252 u=(-0.00565297 -0.00606614 0.0810126)

ii=316225 y=-0.936371 u=(-0.00544439 -0.00904414 0.0956093)

ii=316353 y=-0.923574 u=(-0.00519362 -0.012812 0.110129)

ii=316481 y=-0.90979 u=(-0.00489682 -0.0174689 0.124403)

ii=316609 y=-0.894943 u=(-0.00455088 -0.0231161 0.13823)

ii=316737 y=-0.878951 u=(-0.00415365 -0.0298542 0.151383)

ii=316865 y=-0.861726 u=(-0.00370377 -0.0377802 0.163607)

ii=316993 y=-0.843174 u=(-0.00320252 -0.0469805 0.174613)

ii=317121 y=-0.82319 u=(-0.002653 -0.0575266 0.184092)

ii=317249 y=-0.801666 u=(-0.00206075 -0.0694652 0.191718)

ii=317377 y=-0.778482 u=(-0.00143554 -0.0828093 0.197153)

ii=317505 y=-0.753511 u=(-0.000789686 -0.0975293 0.200063)

ii=317633 y=-0.726614 u=(-0.000139452 -0.113541 0.200139)

ii=317761 y=-0.697644 u=(0.000496615 -0.130699 0.197118)

ii=317889 y=-0.66644 u=(0.00109615 -0.148783 0.190809)

ii=318017 y=-0.632829 u=(0.00163618 -0.167488 0.181132)

ii=318145 y=-0.596627 u=(0.00209476 -0.186432 0.168159)

ii=318273 y=-0.557634 u=(0.00245017 -0.205152 0.152146)

ii=318401 y=-0.515634 u=(0.00268635 -0.223128 0.133575)

ii=318529 y=-0.470396 u=(0.00279119 -0.239814 0.113173)

ii=318657 y=-0.42167 u=(0.00276001 -0.254686 0.0919138)

ii=318785 y=-0.369186 u=(0.002593 -0.267305 0.0709739)

ii=318913 y=-0.312656 u=(0.00229841 -0.277386 0.0516354)

ii=319041 y=-0.251767 u=(0.00188806 -0.284874 0.0351015)

ii=319169 y=-0.186184 u=(0.00137428 -0.289967 0.0222259)

ii=319297 y=-0.115543 u=(0.000771825 -0.293112 0.0131436)

ii=319425 y=-0.0394559 u=(0.000105613 -0.294848 0.0068923)

ii=839745 y=0.0394559 u=(-0.000582071 -0.295449 0.00134095)

ii=839873 y=0.115543 u=(-0.00122611 -0.294747 -0.00626435)

ii=840001 y=0.186184 u=(-0.00177453 -0.292263 -0.0177436)

ii=840129 y=0.251767 u=(-0.00220134 -0.287433 -0.0335546)

ii=840257 y=0.312656 u=(-0.00249681 -0.279854 -0.053115)

ii=840385 y=0.369186 u=(-0.00265327 -0.269405 -0.075247)

ii=840513 y=0.42167 u=(-0.00266883 -0.256237 -0.0985297)

ii=840641 y=0.470396 u=(-0.00254961 -0.240723 -0.121556)

ii=840769 y=0.515634 u=(-0.00230948 -0.223372 -0.143094)

ii=840897 y=0.557634 u=(-0.00196405 -0.204763 -0.162169)

ii=841025 y=0.596627 u=(-0.00153323 -0.185474 -0.178093)

ii=841153 y=0.632829 u=(-0.00103735 -0.166047 -0.190452)

ii=841281 y=0.66644 u=(-0.000497004 -0.146948 -0.199071)

ii=841409 y=0.697644 u=(7.11539e-05 -0.128564 -0.203973)

ii=841537 y=0.726614 u=(0.000650192 -0.111193 -0.205332)

ii=841665 y=0.753511 u=(0.00122649 -0.0950506 -0.203423)

ii=841793 y=0.778482 u=(0.00178946 -0.0802759 -0.198595)

ii=841921 y=0.801666 u=(0.00233001 -0.0669428 -0.191226)

ii=842049 y=0.82319 u=(0.0028412 -0.0550705 -0.181708)

ii=842177 y=0.843174 u=(0.00331821 -0.0446353 -0.170421)

ii=842305 y=0.861726 u=(0.00375808 -0.0355805 -0.15773)

ii=842433 y=0.878951 u=(0.00415926 -0.027827 -0.143969)

ii=842561 y=0.894943 u=(0.00452063 -0.0212799 -0.129439)

ii=842689 y=0.90979 u=(0.00484285 -0.0158363 -0.114408)

ii=842817 y=0.923574 u=(0.00512709 -0.0113894 -0.099104)

ii=842945 y=0.936371 u=(0.00537485 -0.00783297 -0.0837236)

ii=843073 y=0.948252 u=(0.00558836 -0.00506389 -0.0684298)

ii=843201 y=0.959283 u=(0.00576992 -0.00298402 -0.0533565)

ii=843329 y=0.969524 u=(0.00592204 -0.00150175 -0.0386111)

ii=843457 y=0.979032 u=(0.00604735 -0.00053254 -0.0242779)

ii=843585 y=0.987859 u=(0.00614837 -5.53222e-07 -0.0104258)

ii=843713 y=0.996054 u=(0.00622505 0.000147164 0.00250257)

But parrallel run for 6 proc outputs only 11 values!

Code:

ii=103398 y=-0.996054 u=(-0.00619963 6.84788e-05 0.0108932)

ii=103526 y=-0.987859 u=(-0.00613874 -0.000242565 0.0243827)

ii=103654 y=-0.979032 u=(-0.00605887 -0.000950786 0.0380804)

ii=103782 y=-0.969524 u=(-0.00595275 -0.00210967 0.0521367)

ii=103910 y=-0.959283 u=(-0.00581652 -0.00379168 0.0664762)

ii=104038 y=-0.948252 u=(-0.00564615 -0.00607851 0.0810015)

ii=104166 y=-0.936371 u=(-0.00543755 -0.00905979 0.0955933)

ii=104294 y=-0.923574 u=(-0.00518676 -0.0128303 0.110108)

ii=104422 y=-0.90979 u=(-0.00488997 -0.0174897 0.124375)

ii=104550 y=-0.894943 u=(-0.00454407 -0.0231361 0.138196)

ii=104678 y=-0.878951 u=(-0.00414708 -0.0298703 0.151337)

But i expect the same result as single run...Why it is not the same?

DecomposePar Dict the following:

Code:

numberOfSubdomains 6;



method          simple;



simpleCoeffs

{

    n               ( 1 6 1 );

    delta           0.00001;

}



hierarchicalCoeffs

{

    n               ( 1 2 2 );

    delta           0.001;

    order           xyz;

}



manualCoeffs

{

    dataFile        "cellDecomposition";

}



scotchCoeffs

{

}

Also the result is sensitive for the number of proc. i am using...

Greetings Lev,

It's quite simple:

Info is for single process output on the master thread; or for each slave, when each slave is launched with it's own shell, which can be achieved by using foamJobDebug if I'm not mistaken.
Run:

Code:

foamJobDebug -help

for more information.
PInfo will gather all Info's and output on the master thread! (edit: apparently PInfo doesn't exist... I guess my memory isn't all that well...)

Best regards,
Bruno

Hello, Bruno
unfortunately command "PInfo" give a compilation error as "PInfo was not declared in this scope ".
But the problem is not only at the screen output, but also i do not know how to order the code to analyze whole mesh that is spread to many processors (as is working with single run). And i need the results to be collected and to be written in the log file.
Like this command in single run

Code:

OFstream k(runTime.path()/"k.dat");

....

k<<"velocity "<<U[ii]<<endl;

Where can i read about this topics?

Regards
Lev

Hi Lev,

Sorry about that. Apparently PInfo doesn't exist. But I vaguely remember reading about it... weird :confused:

OK, by the description of what you're trying to do, it looks like to me that the probe or sampling utilities and/or function objects would be more suitable for what you want to do.
And then there is also "swak4Foam": http://openfoamwiki.net/index.php/Contrib/swak4Foam

Other than this... try searching this forum for more answers... I know they're in here somewhere...

Best regards,
Bruno

Greetings!

I saw just now the solution and remembered about this thread.

Quote:

Originally Posted by wyldckat (Post 366526)

Sorry about that. Apparently PInfo doesn't exist. But I vaguely remember reading about it... weird :confused:

Apparently my memory mangled the mnemonic incorrectly:

It's not: Info + Pout - cout => PInfo
It's: Info-Info + Parallel + cout => Pout

Kudos to mfmohdyasin:

Quote:

Originally Posted by mfmohdyasin (Post 378825)

Just figured out that I supposed to use "Pout" instead of "Info" to get output from each processor.

Best regards,
Bruno

Hi all,

I'm following up with the issue that decomposePar does not preserve some boundaries and later on the solver complains about missing boundary definitions.

Furthermore potentialFoam complains about the processor-boundaries.

My question is: Apart from the workarounds that have been constructed in the meanwhile to solve this: is there a out-of-box-solution for OpenFOAM 2.1.1 that I am using out of box on ubuntu???

my script looks like this (without reconstructing)

Code:

#!/bin/sh

# Source tutorial run functions

. $WM_PROJECT_DIR/bin/tools/RunFunctions



cp -r 0.org 0 > /dev/null 2>&1



runApplication blockMesh

cp system/decomposeParDict.simple system/decomposeParDict

runApplication decomposePar

runParallel snappyHexMesh 2 -overwrite -parallel



find . -type f -iname "*level*" -exec rm {} \;



ls -d processor* | xargs -i cp -r 0.org/* ./{}/0/ $1



runParallel potentialFoam 2 -initialiseUBCs -noFunctionObjects -writep



runParallel `getApplication` 2

Thanks very much!!
Marco

Greetings Marco,

Well... there are 2... 2.5 tricks that I know of:

The one used in the tutorial "incompressible/pisoFoam/les/motorBike/motorBike", which you seem to be basing your case on.
The one used in the case examples available here: http://code.google.com/p/bluecfd-sin...untimes202_211 - those rely on reconstructing the mesh and then decomposing all of it once again.
And the half-trick still relies on mesh reconstruction, but decomposes only the fields based on the existing decomposition - the following can be used to replace the tutorial "incompressible/pisoFoam/les/motorBike/motorBike/Allrun":

Code:

#!/bin/sh cd ${0%/*} || exit 1 # run from this directory # Source tutorial run functions . $WM_PROJECT_DIR/bin/tools/RunFunctions # copy motorbike surface from resources folder cp $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/ #make sure the write tolerance is upped for mesh reconstruction sed -i -e 's=writePrecision 6;=writePrecision 7;=' system/controlDict # Copy 0.org to 0 cp -r 0.org 0 runApplication blockMesh cp system/decomposeParDict.hierarchical system/decomposeParDict runApplication decomposePar mv log.decomposePar log.decomposePar.1 cp system/decomposeParDict.ptscotch system/decomposeParDict runParallel snappyHexMesh 8 -overwrite -parallel find . -type f -iname "*level*" -exec rm {} \; #ls -d processor* | xargs -i cp -r 0.org/* ./{}/0/ $1 runParallel renumberMesh 8 -overwrite #Need to reconstruct the mesh, in order to be able to re-decompose the fields runApplication reconstructParMesh -constant runApplication decomposePar -fields runParallel potentialFoam 8 -initialiseUBCs -noFunctionObjects runParallel `getApplication` 8 # ----------------------------------------------------------------- end-of-file

Anything beyond this depends on how you defined your boundary condition patch names in the first place! ;)

Best regards,
Bruno

Hey Bruno,

great help, thanks! I took version 2, which works fine.

Thanks
marco

Hi dear FOAMuser

I have 3 machin include 12 core. I used scotch method for 12 processor, its ok on 1 machine. but when I used 2 or 3 machine for more processor, it dosent work without any error or warning. :confused:

Quote:

Originally Posted by m.goudarzi (Post 582448)

I have 3 machin include 12 core. I used scotch method for 12 processor, its ok on 1 machine. but when I used 2 or 3 machine for more processor, it dosent work without any error or warning. :confused:

Quick request: Please provide more details, so that we can reproduce the same error. The details I need are as follows:

Which installation instructions did you follow for installing OpenFOAM?
Which version of OpenFOAM are you using?
Can you reproduce the same problem with one of OpenFOAM's tutorials?
- For example, the tutorial "incompressible/simpleFoam/motorBike"?
Which are the exact commands you are using for launching the solver in parallel?

Hi dear Bruno
Thank you for your reply
actually i dont know how the OpenFOAM installed, This system with OpenFOAM has been gave me.
I use OpenFOAM-2.3.1.
I reproduce the problem with motor bike and have a same problem with scotch method.
when I using simple method ( for decompose) for cavity case, its work with 36 processor but didnt work with scotch method.
I try to run simple method for my 3D asymmetric diffuser but similar to scotch method it didnt work.
my command for run is:
mpirun -np 36 $HOME/OpenFOAM/OpenFOAM-2.3.1/foamExec pimpleFoam -parallel

Thanks
Marzieh

Quote:

Originally Posted by wyldckat (Post 583110)

Quick request: Please provide more details, so that we can reproduce the same error. The details I need are as follows:

Which installation instructions did you follow for installing OpenFOAM?
Which version of OpenFOAM are you using?
Can you reproduce the same problem with one of OpenFOAM's tutorials?
- For example, the tutorial "incompressible/simpleFoam/motorBike"?
Which are the exact commands you are using for launching the solver in parallel?

In addition I use Gambit mesh for the geometry.

Quick answer: Sorry Marzieh, I've been having long work weeks, along with other responsibilities and I haven't managed during weekends to answer as many questions as I wanted here on the forum :(.

Let me refer you to this blog post of mine, which I forgot the other day to mention about it: Notes about running OpenFOAM in parallel

In addition, I forgot to ask the following questions:

What does the following command give you?

Code:

mpirun --version
And what about this one?

Code:

uname -a

Quote:

Originally Posted by wyldckat (Post 585151)

What does the following command give you?

Code:

mpirun --version
And what about this one?

Code:

uname -a

Your welcome dear bruno. Thank you for taking the time to reply me.

1. mpirun version 1.8.2

2. Linux ubuntu00 3.13.0-35-generic #62~precise1-Ubuntu SMP Mon Aug 18 14:52:04 UTC 2014 x86_64 x86_64 GNU/Linux

Quick answer: I've search through my memories and tried to deduce the problem... and something doesn't add up.
This could be a problem with Open-MPI 1.8.2 or it could be due to a problem with incompatible MPI versions on each one of the 3 machines... but it's hard to prove that the issue is only reproducible if you decompose the mesh with scotch and run with pimpleFoam.

I need the following details:

What do the following commands give you?

Code:

mpirun -np 36 $(which foamExec) which mpirun mpirun -np 36 $(which foamExec) mpirun --version mpirun -np 36 $(which foamExec) which pimpleFoam mpirun -np 36 $(which foamExec) wmakePrintBuild mpirun -np 36 $(which foamExec) ldd $(which pimpleFoam) mpirun -np 36 uname -a
What are the exact steps you take to prepare, mesh, decompose and run the case?
- Because something seems to be missing or is being incorrectly used...

Hi,
I am using openFoam 3.0.1 , and I am trying to run my case using the following steps:

$ decomposePar
its running ok.
$ mpirun –np 6 renumberMesh –overwrite -parallel

gives me the following error:

Code:

--------------------------------------------------------------------------

mpirun was unable to launch the specified application as it could not find an executable:



Executable: –np

Node: basharhpc



while attempting to start process rank 0.

--------------------------------------------------------------------------

I was using "mpirun -np" on the same machine with pimpleFoam with no problems but when I tried to use it with "renumberMesh" I get this error . This is the first time for me to use the "renumberMesh".

Bashar

Try -n instead of -np:

mpirun -n 6 renumberMesh -overwrite -parallel

Quote:

Originally Posted by Persia (Post 622038)

Try -n instead of -np:

mpirun -n 6 renumberMesh -overwrite -parallel

Thanks alot for the help.
I just used

Code:

mpirun -n 6 renumberMesh -overwrite -parallel

and it works for me.My last question is that when I will run the case in parallel which one is more appropriate to use the -np 6 or the -n 6 , with mpirun? since I used the -n 6 , in renumberMesh?
Reagrds,
bashar