[piterw82]

Dear OpenFOAM users,

I am trying to run "bed2" tutorial (twoPhaseEulerFoam) on 2 CPU's using OpenFOAM-1.6. It works on single CPU ( i reach 2s of the flow) but fails on more than one CPU (it crashes at 0.114s of the flow). I run the case on local machine that has 4CPU's. Did anybody hve the same problems?
My operating system is OpenSuse 11.1 (x86_64)

The error message I get is:

[0] #0 Foam::error:rintStack(Foam::Ostream&) in "/home/OpenFOAM /OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #1 Foam::sigFpe::sigFpeHandler(int) in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #2 ?? in "/lib64/libc.so.6"
[0] #3 Foam::GAMGSolver::scalingFactor(Foam::Field<double >&, Foam::Field<double> const&, Foam::Field<double> const&, Foam::Field<double> const&) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #4 Foam::GAMGSolver::scalingFactor(Foam::Field<double >&, Foam::lduMatrix const&, Foam::Field<double>&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, Foam::Field<double> const&, unsigned char) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #5 Foam::GAMGSolver::Vcycle(Foam::PtrList<Foam::lduMa trix::smoother> const&, Foam::Field<double>&, Foam::Field<double> const&, Foam::Field<double>&, Foam::Field<double>&, Foam::Field<double>&, Foam::PtrList<Foam::Field<double> >&, Foam::PtrList<Foam::Field<double> >&, unsigned char) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #6 Foam::GAMGSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #7 Foam::fvMatrix<double>::solve(Foam::dictionary const&) in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libfiniteVolume.so"
[0] #8 main in "/root/OpenFOAM/root-1.6/applications/bin/linux64GccDPOpt/twoPhaseEulerFoam"
[0] #9 __libc_start_main in "/lib64/libc.so.6"
[0] #10 _start at /usr/src/packages/BUILD/glibc-2.9/csu/../sysdeps/x86_64/elf/start.S:116
[peter:15350] *** Process received signal ***
[peter:15350] Signal: Floating point exception (8)
[peter:15350] Signal code: (-6)
[peter:15350] Failing at address: 0x3bf6
[peter:15350] [ 0] /lib64/libc.so.6 [0x7f388a8196e0]
[peter:15350] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f388a819645]
[peter:15350] [ 2] /lib64/libc.so.6 [0x7f388a8196e0]
[peter:15350] [ 3] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver13scalingFacto rERNS_5FieldIdEERKS2_S5_S5_+0x6d) [0x7f388b69391d]
[peter:15350] [ 4] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver13scalingFacto rERNS_5FieldIdEERKNS_9lduMatrixES3_RKNS_10FieldFie ldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEE RKS2_h+0xaa) [0x7f388b693d2a]
[peter:15350] [ 5] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver6VcycleERKNS_7 PtrListINS_9lduMatrix8smootherEEERNS_5FieldIdEERKS 8_S9_S9_S9_RNS1_IS8_EESD_h+0x1294) [0x7f388b696b34]
[peter:15350] [ 6] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver5solveERNS_5Fi eldIdEERKS2_h+0x46f) [0x7f388b697d1f]
[peter:15350] [ 7] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS _10dictionaryE+0x158) [0x7f388cc28498]
[peter:15350] [ 8] twoPhaseEulerFoam [0x432158]
[peter:15350] [ 9] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f388a805586]
[peter:15350] [10] twoPhaseEulerFoam [0x42ae29]
[peter:15350] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 15350 on node peter exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------

Regards,
Peter

[wyldckat]

Greetings Peter,

I've just tested the same tutorial case you've reported about, and I ran with a recent (pulled today) OpenFOAM 1.6.x with 2 cores under KUbuntu 10.04 64bit using gcc 4.4.3 and it crashed later on during the run at "Time = 1.04"s, instead of 0.114s.
Also ran the same tutorial case using the pre-built OpenFOAM 1.6 in the same machine and it crashed the very same way, but at "Time = 1.0105"s.
edit: a minor note: with OpenFOAM 1.6, both mpi launched processes crashed with the same sigFpe error!

I've used the simple decomposition method with 2 subdomains and split it in 2 over the 1st coordinate.

I thought this could be a bug already fixed, but apparently it is yet to be fixed. Please report it on the bug report section of this forum, since it was you who found about it : OpenFOAM Bugs
I think you can post a short description there and put a link to this thread, in order to minimize entropy

Best regards,
Bruno

[alberto]

Simply use a conjugate gradient method instead than GAMG. In my experience it is more robust in multiphase simulations.

Best,

[piterw82]

Hi Alberto,

I have switched GAMG to PCG solver for the pressure but the only result was to get further time step. I also set the tolerances for pressure and alpha to 1e-15 and it also did not help. Maybe you have some other sugestions?

Is this tutorial working for you in parallel?

Regards,
Peter

[alberto]

Originally Posted by piterw82

Hi Alberto,

I have switched GAMG to PCG solver for the pressure but the only result was to get further time step. I also set the tolerances for pressure and alpha to 1e-15 and it also did not help. Maybe you have some other sugestions?

Is the code crashing for alpha being too high?

Best,

[piterw82]

Originally Posted by alberto

Is the code crashing for alpha being too high?

Best,

yes, the value for alpha reach very high level (sometime min and max are -1e-5 and 1e+5). below is the printout from my log file (here alpha goes only from -0.68 up to 1.1). The strange thing is that this case is working fine for single processor. Do you have any idea why this is happening for multi-processor?

Courant Number mean: 0.0280557 max: 0.36132
Max Ur Courant Number = 0.461666
Calculating averages

Time = 0.9655

DILUPBiCG: Solving for alpha, Initial residual = 0.00161798, Final residual = 3.43713e-11, No Iterations 4
Dispersed phase volume fraction = 0.275303 Min(alpha) = -2.12575e-18 Max(alpha) = 0.802811
DILUPBiCG: Solving for alpha, Initial residual = 0.000113185, Final residual = 5.23969e-11, No Iterations 3
Dispersed phase volume fraction = 0.275303 Min(alpha) = -2.11835e-18 Max(alpha) = 0.802806
GAMG: Solving for p, Initial residual = 0.000399759, Final residual = 8.44465e-09, No Iterations 12
time step continuity errors : sum local = 3.62591e-09, global = -7.10158e-10, cumulative = 2.70152e-07
DILUPBiCG: Solving for alpha, Initial residual = 8.22487e-05, Final residual = 6.99567e-11, No Iterations 3
Dispersed phase volume fraction = 0.275303 Min(alpha) = -2.11584e-18 Max(alpha) = 0.802824
DILUPBiCG: Solving for alpha, Initial residual = 8.09589e-06, Final residual = 0.00755801, No Iterations 1001
Dispersed phase volume fraction = 0.275245 Min(alpha) = -0.684932 Max(alpha) = 1.10975
GAMG: Solving for p, Initial residual = 0.00143021, Final residual = 7.14171e-09, No Iterations 19
time step continuity errors : sum local = 3.07254e-09, global = 5.55504e-10, cumulative = 2.70707e-07
DILUPBiCG: Solving for epsilon, Initial residual = 0.00673291, Final residual = 2.54617e-06, No Iterations 5
DILUPBiCG: Solving for k, Initial residual = 0.00751057, Final residual = 3.37159e-06, No Iterations 5

Regards,
Peter

[alberto]

Originally Posted by piterw82

yes, the value for alpha reach very high level (sometime min and max are -1e-5 and 1e+5). below is the printout from my log file (here alpha goes only from -0.68 up to 1.1). The strange thing is that this case is working fine for single processor. Do you have any idea why this is happening for multi-processor?

The problem with the particle pressure in twoPhaseEulerFoam is due to how the particle pressure is treated, and to the structure of the solution algorithm, which should be more robust.

I don't know why this happens in parallel. Probably it has something to do with how the domain is decomposed. For example, decompose with the simple method, and try to keep the particle phase in the same processor patch (I know it's not convenient, but it might help understanding).

Best,

[piterw82]

Originally Posted by alberto

The problem with the particle pressure in twoPhaseEulerFoam is due to how the particle pressure is treated, and to the structure of the solution algorithm, which should be more robust.

I don't know why this happens in parallel. Probably it has something to do with how the domain is decomposed. For example, decompose with the simple method, and try to keep the particle phase in the same processor patch (I know it's not convenient, but it might help understanding).

Best,

Hello Alberto,

As the domain is rectangular I do the simple method for decomposition. Here is the printout after the decomposeParDic:

/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 1.6 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 1.6-f802ff2d6c5a
Exec : decomposePar
Date : Jun 23 2010
Time : 10:02:14
Host : root
PID : 32387
Case : /root/OpenFOAM/root-1.6/run/test_4_bed2
nProcs : 1
SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Time = 0
Create mesh

Calculating distribution of cells
Selecting decompositionMethod simple

Finished decomposition in 0.01 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Calculating processor boundary addressing

Distributing points to processors

Constructing processor meshes

Processor 0
Number of cells = 3000
Number of faces shared with processor 1 = 30
Number of processor patches = 1
Number of processor faces = 30
Number of boundary faces = 6230

Processor 1
Number of cells = 3000
Number of faces shared with processor 0 = 30
Number of processor patches = 1
Number of processor faces = 30
Number of boundary faces = 6230

Number of processor faces = 30
Max number of processor patches = 1
Max number of faces between processors = 30

Processor 0: field transfer
Processor 1: field transfer

End.



I have run the test (as you suggested) with alpha phase being always in one processor (I have extended the hight of the test case to 2m instead of 1m). This simulation was running without crushing. Maybe there is a problem with sharing the information of alpha between the processors? If this is the case how can you run parallel simulation with Cirulating Fluidized Bed. Do you have any suggestion what can be done?

Regards,
Peter

[alberto]

There is either some missing "correctBoundaryConditions()", which I doubt, or something changes between serial and parallel runs (just guessing: interpolation of ppMagf?).

I run CFB's in the past in parallel, but it was long ago (OF 1.4). Stability problems are extremely common with twoPhaseEulerFoam. As I told you, the algorithm should be revised to make the management of the particle pressure more robust.

Best,

[piterw82]

Originally Posted by alberto

There is either some missing "correctBoundaryConditions()", which I doubt, or something changes between serial and parallel runs (just guessing: interpolation of ppMagf?).

I run CFB's in the past in parallel, but it was long ago (OF 1.4). Stability problems are extremely common with twoPhaseEulerFoam. As I told you, the algorithm should be revised to make the management of the particle pressure more robust.

Best,

I have reported the problem with parallel run for "bed2" tutorial in OpenFOAM Bugs, as Bruno advise me to do. I guess there is a change when decomposing the mesh which generates the problem. That could explain why the solver is running on single CPU and crashes with multi processor.