[flying]

Hey Foamers:

It is very my pleasure to discuss the parallel running of icoLagrangianFoam.

When I use icoLagrangianFoam, at first I use the command' decomposePar' and there are two folders of 'processor0' and 'processor1' to be created. And then I used the command of ''mpriun -np icoLagrangianFoam'.

The case is run, but there is no result to be saved in the folder of 'processor1'and'processor0', and the result is saved under the folder of case.

I feel a little confused whether the icoLangrangianFoam could be run paralleled or the result is saved under the folder of case other in the 'processor*`. Would anyone like to help me for the question?

Thanks and best wishes!

[gschaider]

Originally Posted by flying

Hey Foamers:

It is very my pleasure to discuss the parallel running of icoLagrangianFoam.

When I use icoLagrangianFoam, at first I use the command' decomposePar' and there are two folders of 'processor0' and 'processor1' to be created. And then I used the command of ''mpriun -np icoLagrangianFoam'.

The case is run, but there is no result to be saved in the folder of 'processor1'and'processor0', and the result is saved under the folder of case.

I feel a little confused whether the icoLangrangianFoam could be run paralleled or the result is saved under the folder of case other in the 'processor*`. Would anyone like to help me for the question?

Thanks and best wishes!

Try again with
mpriun -np 2 icoLagrangianFoam -parallel
(what you did was run two serial runs in parallel

Bernhard

[flying]

Sincerely thanks! And it is work now. But for running paralleled with two processors is even slower than one processor.

[gschaider]

Originally Posted by flying

Sincerely thanks! And it is work now. But for running paralleled with two processors is even slower than one processor.

For small cases that is not surprising, because the communication needs more time than the actual calculation

[flying]

Thanks a lot

[flying]

Long time no see. And wish all foamers are fine.

There are big bugs on "icoLagrangianFoam" in the case of paralleling computing. There is no problem for the case are computed using one process. If the mesh is decomposed in the z direction for cavity, the particle could be moved, but the result is different with case of using one process, it is the same situation for the y direction. The worst thing is that the particles can not be moved if it is decomposed in x direction after the computation starts some steps. It is absolutely there is some problem for the particle transfered between processors.

Does anyone also find the bug?
It is very sad I know few about the part of transfer field of particles. Who would like to give me some advice on it?

Thanks and best wishes!

[gschaider]

Originally Posted by flying

Long time no see. And wish all foamers are fine.

There are big bugs on "icoLagrangianFoam" in the case of paralleling computing. There is no problem for the case are computed using one process. If the mesh is decomposed in the z direction for cavity, the particle could be moved, but the result is different with case of using one process, it is the same situation for the y direction. The worst thing is that the particles can not be moved if it is decomposed in x direction after the computation starts some steps. It is absolutely there is some problem for the particle transfered between processors.

Does anyone also find the bug?
It is very sad I know few about the part of transfer field of particles. Who would like to give me some advice on it?

Thanks and best wishes!

Before we start talking: which version? (OpenFOAM and icoLagrangianFoam)

I just had a quick look at the latest version and noticed that in method HardBallParticle::hitProcessorPatch it reads
td.switchProcessor=false;
where in my opinion it should be
td.switchProcessor=true;
but I'm not 100% sure whether this will fix it and I havn't got the time to test it

Bernhard

[flying]

It is icoLangrangianFoam for OpenFoam 1.5.

In the HardBallParticle.C, there is no lines in the member function of HardBallParticle::hitProcessorPatch. Maybe I should add these lines. I will try it. Thanks a lot.

[gschaider]

Originally Posted by flying

It is icoLangrangianFoam for OpenFoam 1.5.

In the HardBallParticle.C, there is no lines in the member function of HardBallParticle::hitProcessorPatch. Maybe I should add these lines. I will try it. Thanks a lot.

There are two such methods. One is empty. The other not.

[foamWang]

Hi Bernhard,

Yes, this solves the problem.
However, if I set useMomentumSource to 1, i.e. two-way coupling of solid and fluid phase, a extremely high value of Co will be achieved. This obviously not physical. This may be dedicated to the suddenly appearance of particles in a new processor.

Any idea to solve this?

Thanks.

Roro

Originally Posted by gschaider

Before we start talking: which version? (OpenFOAM and icoLagrangianFoam)

I just had a quick look at the latest version and noticed that in method HardBallParticle::hitProcessorPatch it reads
td.switchProcessor=false;
where in my opinion it should be
td.switchProcessor=true;
but I'm not 100% sure whether this will fix it and I havn't got the time to test it

Bernhard

[gschaider]

Originally Posted by foamWang

Yes, this solves the problem.
However, if I set useMomentumSource to 1, i.e. two-way coupling of solid and fluid phase, a extremely high value of Co will be achieved. This obviously not physical. This may be dedicated to the suddenly appearance of particles in a new processor.

Any idea to solve this?

Does this happen only in parallel runs? (I doubt it)

I think it IS physical: particles with a momentum appear out of nowhere (OK. THAT is unphysical). What should the fluid do? Suffer silently (that would be unphysical) or try to incorporate the additional momentum (that is what you're seeing)

Solutions all have to do with the injector:
- inject less (volume fraction of the particles should be well below 10% for the solver to be valid)
- inject slower
- write a different injector (on the patch, one that injects with no relative velocity to the fluid)

Bernhard

[foamWang]

Hi, Bernard,

Yes, it's not the parallel problem. Particles can smoothly pass the processorPatch.

Thanks.

Roro

Originally Posted by gschaider

Does this happen only in parallel runs? (I doubt it)

I think it IS physical: particles with a momentum appear out of nowhere (OK. THAT is unphysical). What should the fluid do? Suffer silently (that would be unphysical) or try to incorporate the additional momentum (that is what you're seeing)

Solutions all have to do with the injector:
- inject less (volume fraction of the particles should be well below 10% for the solver to be valid)
- inject slower
- write a different injector (on the patch, one that injects with no relative velocity to the fluid)

Bernhard

[chegdan]

Hi All,

I downloaded the icoLagrangianFoam from (http://openfoam-extend.svn.sourcefor...agrangianFoam/) and everything compiled fine in OF-1.5-dev

I tried to run icoLagrangianFoam in parallel and it failed, giving the following error messages:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.5-dev                               |
|   \\  /    A nd           | Revision: 1664                                  |
|    \\/     M anipulation  | Web:      http://www.OpenFOAM.org               |
\*---------------------------------------------------------------------------*/
Exec   : /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam -parallel
Date   : Oct 22 2010
Time   : 15:17:32
Host   : aris
PID    : 5532
Case   : /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/icoLagrangianFoam/channelParticles
nProcs : 2
Slaves : 
1
(
aris.5533
)

Pstream initialized with:
    floatTransfer     : 0
    nProcsSimpleSum   : 0
    commsType         : nonBlocking

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi


Reading environmentalProperties
Constructing kinematicCloud
--> FOAM Warning : 
    From function Cloud<ParticleType>::initCloud(const bool checkClass)
    in file /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/src/lagrangian/basic/lnInclude/CloudIO.C at line 51
    Cannot read particle positions file 
    "/home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/icoLagrangianFoam/channelParticles/processor0/0/lagrangian/kinematicCloud"
    assuming the initial cloud contains 0 particles.
Selecting DispersionModel NoDispersion
Selecting DragModel SphereDrag
Selecting InjectionModel ConeInjection
Selecting pdfType RosinRammler
Selecting WallInteractionModel StandardWallInteraction
Selecting U IntegrationScheme Euler

Starting time loop

Time = 0.001

Courant Number mean: 0 max: 0.2 velocity magnitude: 1
Evolving kinematicCloud
[aris:05532] *** Process received signal ***
[aris:05532] Signal: Segmentation fault (11)
[aris:05532] Signal code:  (-6)
[aris:05532] Failing at address: 0x3e80000159c
[aris:05532] [ 0] /lib/libc.so.6 [0x7f24fce05530]
[aris:05532] [ 1] /lib/libc.so.6(gsignal+0x35) [0x7f24fce054b5]
[aris:05532] [ 2] /lib/libc.so.6 [0x7f24fce05530]
[aris:05532] [ 3] /lib/libc.so.6 [0x7f24fce4afc2]
[aris:05532] [ 4] /lib/libc.so.6(__libc_malloc+0x6e) [0x7f24fce4cd4e]
[aris:05532] [ 5] /usr/lib/libstdc++.so.6(_Znwm+0x1d) [0x7f24fd6a464d]
[aris:05532] [ 6] /usr/lib/libstdc++.so.6(_Znam+0x9) [0x7f24fd6a4769]
[aris:05532] [ 7] /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam4ListIcE7setSizeEi+0x33) [0x7f24fdcb48a3]
[aris:05532] [ 8] /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8OPstreamC1ENS_7Pstream10commsTypesEiiNS_8IOstream12streamFormatENS3_13versionNumberE+0xce) [0x7f24fdcb3b7e]
[aris:05532] [ 9] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x43a220]
[aris:05532] [10] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x448ede]
[aris:05532] [11] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x449cde]
[aris:05532] [12] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x419353]
[aris:05532] [13] /lib/libc.so.6(__libc_start_main+0xfd) [0x7f24fcdf0abd]
[aris:05532] [14] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x416b29]
[aris:05532] *** End of error message ***
mpirun noticed that job rank 0 with PID 5532 on node aris exited on signal 11 (Segmentation fault). 
1 additional process aborted (not shown)

I thought it might be a recent update of my kernel, but a quick parallel run of interFoam on the damBreak case finished without any problems.

I ran a parallel run using

mpirun -np 2 `which icoLagrangianFoam` -parallel < /dev/null >& log.icoLagrangianFoam &

Im sure its something I'm doing wrong. Any help?

Dan

[gschaider]

Originally Posted by chegdan

Hi All,

I downloaded the icoLagrangianFoam from (http://openfoam-extend.svn.sourcefor...agrangianFoam/) and everything compiled fine in OF-1.5-dev

I tried to run icoLagrangianFoam in parallel and it failed, giving the following error messages:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.5-dev                               |
|   \\  /    A nd           | Revision: 1664                                  |
|    \\/     M anipulation  | Web:      http://www.OpenFOAM.org               |
\*---------------------------------------------------------------------------*/
Exec   : /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam -parallel
Date   : Oct 22 2010
Time   : 15:17:32
Host   : aris
PID    : 5532
Case   : /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/icoLagrangianFoam/channelParticles
nProcs : 2
Slaves : 
1
(
aris.5533
)

Pstream initialized with:
    floatTransfer     : 0
    nProcsSimpleSum   : 0
    commsType         : nonBlocking

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi


Reading environmentalProperties
Constructing kinematicCloud
--> FOAM Warning : 
    From function Cloud<ParticleType>::initCloud(const bool checkClass)
    in file /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/src/lagrangian/basic/lnInclude/CloudIO.C at line 51
    Cannot read particle positions file 
    "/home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/icoLagrangianFoam/channelParticles/processor0/0/lagrangian/kinematicCloud"
    assuming the initial cloud contains 0 particles.
Selecting DispersionModel NoDispersion
Selecting DragModel SphereDrag
Selecting InjectionModel ConeInjection
Selecting pdfType RosinRammler
Selecting WallInteractionModel StandardWallInteraction
Selecting U IntegrationScheme Euler

Starting time loop

Time = 0.001

Courant Number mean: 0 max: 0.2 velocity magnitude: 1
Evolving kinematicCloud
[aris:05532] *** Process received signal ***
[aris:05532] Signal: Segmentation fault (11)
[aris:05532] Signal code:  (-6)
[aris:05532] Failing at address: 0x3e80000159c
[aris:05532] [ 0] /lib/libc.so.6 [0x7f24fce05530]
[aris:05532] [ 1] /lib/libc.so.6(gsignal+0x35) [0x7f24fce054b5]
[aris:05532] [ 2] /lib/libc.so.6 [0x7f24fce05530]
[aris:05532] [ 3] /lib/libc.so.6 [0x7f24fce4afc2]
[aris:05532] [ 4] /lib/libc.so.6(__libc_malloc+0x6e) [0x7f24fce4cd4e]
[aris:05532] [ 5] /usr/lib/libstdc++.so.6(_Znwm+0x1d) [0x7f24fd6a464d]
[aris:05532] [ 6] /usr/lib/libstdc++.so.6(_Znam+0x9) [0x7f24fd6a4769]
[aris:05532] [ 7] /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam4ListIcE7setSizeEi+0x33) [0x7f24fdcb48a3]
[aris:05532] [ 8] /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8OPstreamC1ENS_7Pstream10commsTypesEiiNS_8IOstream12streamFormatENS3_13versionNumberE+0xce) [0x7f24fdcb3b7e]
[aris:05532] [ 9] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x43a220]
[aris:05532] [10] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x448ede]
[aris:05532] [11] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x449cde]
[aris:05532] [12] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x419353]
[aris:05532] [13] /lib/libc.so.6(__libc_start_main+0xfd) [0x7f24fcdf0abd]
[aris:05532] [14] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/icoLagrangianFoam [0x416b29]
[aris:05532] *** End of error message ***
mpirun noticed that job rank 0 with PID 5532 on node aris exited on signal 11 (Segmentation fault). 
1 additional process aborted (not shown)

I thought it might be a recent update of my kernel, but a quick parallel run of interFoam on the damBreak case finished without any problems.

I ran a parallel run using

mpirun -np 2 `which icoLagrangianFoam` -parallel < /dev/null >& log.icoLagrangianFoam &

Im sure its something I'm doing wrong. Any help?

Dan

Hi Dan!

Could you try the same with the rhoTurbTwinParcelFoam? Just to make sure whether the problem is with the solver or with the library?

Bernhard

[chegdan]

Thanks for the reply, I tried the rhoTurbTwinParcelFoam solver and it compiled fine. When i ran the test case received the error:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.5-dev                               |
|   \\  /    A nd           | Revision: 1664                                  |
|    \\/     M anipulation  | Web:      http://www.OpenFOAM.org               |
\*---------------------------------------------------------------------------*/
Exec   : /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/rhoTurbTwinParcelFoam -parallel
Date   : Oct 24 2010
Time   : 15:32:19
Host   : aris
PID    : 3572
Case   : /home/dcombest/OpenFOAM/dcombest-1.5-dev/run/tutorials/rhoTurbTwinParcelFoam/simplifiedSiwek
nProcs : 2
Slaves : 
1
(
aris.3573
)

Pstream initialized with:
    floatTransfer     : 0
    nProcsSimpleSum   : 0
    commsType         : nonBlocking

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0


Reading environmentalProperties
Reading thermophysical properties

Selecting thermodynamics package hThermo<pureMixture<constTransport<specieThermo<hConstThermo<perfectGas>>>>>

Reading field U

Reading/calculating face flux field phi

Creating turbulence model

Selecting RAS turbulence model kEpsilon
kEpsilonCoeffs
{
    Cmu             0.09;
    C1              1.44;
    C2              1.92;
    C3              -0.33;
    alphah          1;
    alphak          1;
    alphaEps        0.76923;
}

Creating field DpDt

Constructing thermoCloud1
--> FOAM Warning : 
    From function Cloud<ParticleType>::initCloud(const bool checkClass)
    in file /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/src/lagrangian/basic/lnInclude/CloudIO.C at line 51
    Cannot read particle positions file 
    "/home/dcombest/OpenFOAM/dcombest-1.5-dev/run/tutorials/rhoTurbTwinParcelFoam/simplifiedSiwek/processor0/0/lagrangian/thermoCloud1"
    assuming the initial cloud contains 0 particles.
Selecting DispersionModel StochasticDispersionRAS
Selecting DragModel SphereDrag
Selecting InjectionModel ManualInjection
Selecting pdfType RosinRammler
Selecting WallInteractionModel StandardWallInteraction
Selecting U IntegrationScheme Euler
Selecting HeatTransferModel RanzMarshall
Selecting T IntegrationScheme Analytical
Constructing kinematicCloud1
--> FOAM Warning : 
    From function Cloud<ParticleType>::initCloud(const bool checkClass)
    in file /home/dcombest/OpenFOAM/OpenFOAM-1.5-dev/src/lagrangian/basic/lnInclude/CloudIO.C at line 51
    Cannot read particle positions file 
    "/home/dcombest/OpenFOAM/dcombest-1.5-dev/run/tutorials/rhoTurbTwinParcelFoam/simplifiedSiwek/processor0/0/lagrangian/kinematicCloud1"
    assuming the initial cloud contains 0 particles.
Selecting DispersionModel StochasticDispersionRAS
Selecting DragModel SphereDrag
Selecting InjectionModel ManualInjection
Selecting pdfType RosinRammler
Selecting WallInteractionModel StandardWallInteraction
Selecting U IntegrationScheme Euler
Courant Number mean: 0 max: 0

Starting time loop

Courant Number mean: 0 max: 0
deltaT = 0.000119047619
Time = 0.000119048

Evolving thermoCloud1
[aris:03572] *** Process received signal ***
[aris:03572] Signal: Segmentation fault (11)
[aris:03572] Signal code:  (-6)
[aris:03572] Failing at address: 0x3e800000df4
[aris:03572] [ 0] /lib/libc.so.6 [0x7fdde01b9530]
[aris:03572] [ 1] /lib/libc.so.6(gsignal+0x35) [0x7fdde01b94b5]
[aris:03572] [ 2] /lib/libc.so.6 [0x7fdde01b9530]
[aris:03572] [ 3] /lib/libc.so.6 [0x7fdde01feaf0]
[aris:03572] [ 4] /lib/libc.so.6(__libc_malloc+0x6e) [0x7fdde0200d4e]
[aris:03572] [ 5] /usr/lib/libstdc++.so.6(_Znwm+0x1d) [0x7fdde0a5864d]
[aris:03572] [ 6] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/rhoTurbTwinParcelFoam [0x465b9c]
[aris:03572] [ 7] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/rhoTurbTwinParcelFoam [0x46650f]
[aris:03572] [ 8] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/rhoTurbTwinParcelFoam [0x41b8e7]
[aris:03572] [ 9] /lib/libc.so.6(__libc_start_main+0xfd) [0x7fdde01a4abd]
[aris:03572] [10] /home/dcombest/OpenFOAM/dcombest-1.5-dev/applications/bin/linux64GccDPOpt/rhoTurbTwinParcelFoam [0x418eb9]
[aris:03572] *** End of error message ***
mpirun noticed that job rank 0 with PID 3572 on node aris exited on signal 11 (Segmentation fault). 
1 additional process aborted (not shown)

the code works fine in serial.

I then went to the lagrangian folder in the $FOAM_SRC and performed and "svn update" and then recompiled the lagrangian files with the ./Allwmake script. Still the same error. Is there another folder I should update? Should I do a complete update of 1.5-dev?

Dan

[chegdan]

I just wanted to make sure it wasn't something else that was out of date and needed to be updated. I updated all of of-1.5-dev through svn and recompiled the changes, tried to run parallel again with icoLagrangianFoam and same error messages.

I'm working on 64bit ubuntu 9.10

Dan

[gschaider]

Originally Posted by chegdan

I just wanted to make sure it wasn't something else that was out of date and needed to be updated. I updated all of of-1.5-dev through svn and recompiled the changes, tried to run parallel again with icoLagrangianFoam and same error messages.

I'm working on 64bit ubuntu 9.10

Dan

This seems to be a general problem with the 1.5-dev (I'm getting it too). The nature of the stack-trace indicates that somewhere memory is freed twice somewhere

Bernhard

[preichl]

Hi All,

Does anyone know how to get around the failure with resulting stack trace problem when running icoLagrangianFoam in parallel?.
I am also getting this with OF 1.5-dev. It works fine on one processor, but as soon as I try to run it in parallel I also get this error.

I also noted that icoLagrangianFoam is not included in OF 1.6-ext. I tried to move it across but when I tried to compile it with (wmake icoLagrangianFoam ) it complains about a missing CintDefs.H file.

Thanks in advance (again),

Paul.

[gschaider]

Originally Posted by preichl

Hi All,

Does anyone know how to get around the failure with resulting stack trace problem when running icoLagrangianFoam in parallel?.
I am also getting this with OF 1.5-dev. It works fine on one processor, but as soon as I try to run it in parallel I also get this error.

I also noted that icoLagrangianFoam is not included in OF 1.6-ext. I tried to move it across but when I tried to compile it with (wmake icoLagrangianFoam ) it complains about a missing CintDefs.H file.

Thanks in advance (again),

Paul.

Hi Paul!

icoLagrangianFoam should be in 1.6-ext, but not in the solver directory but in $FOAM_TUTORIALS/lagrangian

Bernhard

[preichl]

Hi Bernhard,

I compiled the icoLagrangianFoam files in $FOAM_TUTORIALS/lagrangian of OF 1.6-ext and everything now works.

Thanks again,

Paul.