[sunnysun]

Hi, All,

I am runing a transient case with both icoFoam and turbFoam, the mesh composed of 300,000 elements.

I use parallel computing with 15 processors(64 bits), but it turns out to be very slow. adaptive time step is used. Below is part of output from 2 solvers.

ICOFOAM:
Time = 0.090534016

Courant Number mean: 0.0042189121 max: 0.60001494
deltaT = 5.051708e-06
DILUPBiCG: Solving for Ux, Initial residual = 1.5933376e-05, Final residual = 9.3428658e-09, No Iterations 2
DILUPBiCG: Solving for Uy, Initial residual = 1.7857106e-05, Final residual = 1.1072651e-09, No Iterations 2
DILUPBiCG: Solving for Uz, Initial residual = 1.6937772e-05, Final residual = 4.5817176e-09, No Iterations 2
DICPCG: Solving for p, Initial residual = 3.6714912e-05, Final residual = 9.753555e-09, No Iterations 937
DICPCG: Solving for p, Initial residual = 0.00022323789, Final residual = 9.9495147e-09, No Iterations 870
time step continuity errors : sum local = 1.1644498e-12, global = -3.4127869e-14, cumulative = 5.5155785e-13
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
ExecutionTime = 139686.95 s ClockTime = 140261 s

Time = 0.090539068

Courant Number mean: 0.0042189072 max: 0.60001581
deltaT = 5.0515749e-06
DILUPBiCG: Solving for Ux, Initial residual = 1.6457677e-05, Final residual = 4.6470092e-09, No Iterations 3
DILUPBiCG: Solving for Uy, Initial residual = 1.8183019e-05, Final residual = 1.1097131e-09, No Iterations 3
DILUPBiCG: Solving for Uz, Initial residual = 1.7496236e-05, Final residual = 8.7850725e-09, No Iterations 2
DICPCG: Solving for p, Initial residual = 3.916581e-05, Final residual = 9.5537303e-09, No Iterations 934
DICPCG: Solving for p, Initial residual = 0.00022626984, Final residual = 9.5267804e-09, No Iterations 914
time step continuity errors : sum local = 1.1548003e-12, global = -1.7614837e-14, cumulative = 5.3394302e-13
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
ExecutionTime = 139696.76 s ClockTime = 140271 s

Time = 0.090544119

Courant Number mean: 0.0042188909 max: 0.60001492
deltaT = 5.0514493e-06
DILUPBiCG: Solving for Ux, Initial residual = 1.5920686e-05, Final residual = 9.2325137e-09, No Iterations 2
DILUPBiCG: Solving for Uy, Initial residual = 1.7845416e-05, Final residual = 9.8299531e-10, No Iterations 2
DILUPBiCG: Solving for Uz, Initial residual = 1.6922093e-05, Final residual = 4.5258186e-09, No Iterations 2
DICPCG: Solving for p, Initial residual = 3.8415655e-05, Final residual = 9.4206439e-09, No Iterations 938
DICPCG: Solving for p, Initial residual = 0.00023378581, Final residual = 9.4035601e-09, No Iterations 864
time step continuity errors : sum local = 1.1591606e-12, global = 2.6889353e-15, cumulative = 5.3663195e-13
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
ExecutionTime = 139706.31 s ClockTime = 140281 s

Time = 0.090549171

Courant Number mean: 0.0042188863 max: 0.60001584
deltaT = 5.051316e-06
DILUPBiCG: Solving for Ux, Initial residual = 1.6469925e-05, Final residual = 4.7423517e-09, No Iterations 3
DILUPBiCG: Solving for Uy, Initial residual = 1.8189916e-05, Final residual = 1.2159212e-09, No Iterations 3
DILUPBiCG: Solving for Uz, Initial residual = 1.7507658e-05, Final residual = 8.9428823e-09, No Iterations 2
DICPCG: Solving for p, Initial residual = 3.9920706e-05, Final residual = 9.8779321e-09, No Iterations 933
DICPCG: Solving for p, Initial residual = 0.00023749663, Final residual = 9.958994e-09, No Iterations 914
time step continuity errors : sum local = 1.1595706e-12, global = 1.3994702e-14, cumulative = 5.5062665e-13
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
ExecutionTime = 139716.1 s ClockTime = 140291 s


TURBFOAM:


Time = 0.011246492

Courant Number mean: 0.0021497253 max: 0.29183874
deltaT = 4.1666667e-06
DILUPBiCG: Solving for Ux, Initial residual = 2.2243458e-05, Final residual = 6.4891902e-08, No Iterations 1
DILUPBiCG: Solving for Uy, Initial residual = 2.7101833e-05, Final residual = 1.6199912e-07, No Iterations 1
DILUPBiCG: Solving for Uz, Initial residual = 2.3161504e-05, Final residual = 1.6636072e-07, No Iterations 1
DICPCG: Solving for p, Initial residual = 6.4919684e-05, Final residual = 9.5536623e-07, No Iterations 870
DICPCG: Solving for p, Initial residual = 6.7920792e-05, Final residual = 9.9707514e-07, No Iterations 13
DICPCG: Solving for p, Initial residual = 8.065135e-06, Final residual = 7.5610997e-07, No Iterations 3
time step continuity errors : sum local = 4.3769708e-12, global = -4.630939e-14, cumulative = -3.7206299e-11
DICPCG: Solving for p, Initial residual = 4.9009229e-06, Final residual = 9.478687e-07, No Iterations 65
DICPCG: Solving for p, Initial residual = 7.1595084e-06, Final residual = 8.0333983e-07, No Iterations 2
DICPCG: Solving for p, Initial residual = 1.1750355e-06, Final residual = 5.0730276e-07, No Iterations 1
time step continuity errors : sum local = 2.93423e-12, global = -1.9983413e-13, cumulative = -3.7406133e-11
DILUPBiCG: Solving for epsilon, Initial residual = 1.0007713e-05, Final residual = 4.9980981e-08, No Iterations 1
DILUPBiCG: Solving for k, Initial residual = 3.374637e-05, Final residual = 2.9900197e-07, No Iterations 1
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
ExecutionTime = 17468.16 s ClockTime = 157562 s

Time = 0.011250659

Courant Number mean: 0.0021497677 max: 0.29185204
deltaT = 4.1666667e-06
DILUPBiCG: Solving for Ux, Initial residual = 2.2222925e-05, Final residual = 6.6488393e-08, No Iterations 1
DILUPBiCG: Solving for Uy, Initial residual = 2.7102149e-05, Final residual = 1.639788e-07, No Iterations 1
DILUPBiCG: Solving for Uz, Initial residual = 2.3168021e-05, Final residual = 1.8278849e-07, No Iterations 1
DICPCG: Solving for p, Initial residual = 5.2848402e-05, Final residual = 9.6086881e-07, No Iterations 867
DICPCG: Solving for p, Initial residual = 6.600024e-05, Final residual = 9.7140246e-07, No Iterations 13
DICPCG: Solving for p, Initial residual = 7.8676029e-06, Final residual = 7.3652595e-07, No Iterations 3
time step continuity errors : sum local = 4.270429e-12, global = 4.5538476e-14, cumulative = -3.7360594e-11
DICPCG: Solving for p, Initial residual = 4.7932553e-06, Final residual = 9.1380874e-07, No Iterations 24
DICPCG: Solving for p, Initial residual = 4.0275424e-06, Final residual = 7.9633809e-07, No Iterations 1
DICPCG: Solving for p, Initial residual = 9.6547429e-07, Final residual = 9.6547429e-07, No Iterations 0
time step continuity errors : sum local = 5.3984369e-12, global = 5.6480473e-13, cumulative = -3.679579e-11
DILUPBiCG: Solving for epsilon, Initial residual = 9.9681975e-06, Final residual = 9.9681975e-06, No Iterations 0
DILUPBiCG: Solving for k, Initial residual = 3.3737499e-05, Final residual = 2.9891437e-07, No Iterations 1
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
ExecutionTime = 17474.9 s ClockTime = 157626 s

Time = 0.011254826

Courant Number mean: 0.0021498106 max: 0.29186488
deltaT = 4.1666667e-06
DILUPBiCG: Solving for Ux, Initial residual = 2.2233614e-05, Final residual = 6.5251044e-08, No Iterations 1
DILUPBiCG: Solving for Uy, Initial residual = 2.7089605e-05, Final residual = 1.6233944e-07, No Iterations 1
DILUPBiCG: Solving for Uz, Initial residual = 2.3153049e-05, Final residual = 1.6663021e-07, No Iterations 1
DICPCG: Solving for p, Initial residual = 6.453333e-05, Final residual = 9.5711004e-07, No Iterations 870
DICPCG: Solving for p, Initial residual = 6.8593867e-05, Final residual = 9.8837497e-07, No Iterations 16
DICPCG: Solving for p, Initial residual = 8.3200747e-06, Final residual = 7.3665399e-07, No Iterations 3
time step continuity errors : sum local = 4.2746891e-12, global = -1.250224e-13, cumulative = -3.6920812e-11
DICPCG: Solving for p, Initial residual = 4.9091131e-06, Final residual = 9.0890313e-07, No Iterations 65
DICPCG: Solving for p, Initial residual = 6.9693982e-06, Final residual = 7.8703718e-07, No Iterations 2
DICPCG: Solving for p, Initial residual = 1.1489768e-06, Final residual = 4.9787673e-07, No Iterations 1
time step continuity errors : sum local = 2.8983148e-12, global = -1.6075718e-13, cumulative = -3.7081569e-11
DILUPBiCG: Solving for epsilon, Initial residual = 1.0005115e-05, Final residual = 5.0014711e-08, No Iterations 1
DILUPBiCG: Solving for k, Initial residual = 3.3738596e-05, Final residual = 2.9894466e-07, No Iterations 1
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
DILUPBiCG: Solving for T, Initial residual = 0, Final residual = 0, No Iterations 0
ExecutionTime = 17482.23 s ClockTime = 157694 s

Is there any way to improve the computational performence?

Thanks in advance!

Vivien

[gcollecutt]

Hi Vivien,

On occasions I have forgotten to put the -parallel option on the command line and have run 8 or more simultaneous copies of the single process version. How I got my PhD I don't know! I doubt you have made this mistake.

I also see that the DICPCG solver is working pretty hard to solve the pressure field. Normally I would expect it to take less than 10 iterations to converge when running a courant number of 0.3. Sometimes I find the GAMG solver better for stubborn fields, but I have noted that this solver does not parallelise very well.

How well does it run when you use a constant inlet condition (rather than the time varying one I think you are using)?

Greg.

[sunnysun]

Hi, Greg,

I did put the "-parallel" command...

May I know how fine is your mesh and how long is your simulation? Which solver did you use?

Many thanks!!

Vivien

[hellorishi]

Hi Vivien,

I read somewhere on the forum that for efficient parallel execution, each processor should get atleast 100k cells. In your case 300k/15cpu = 20k cells per cpu. So there is a lot of communication involved and possibly the network speed is bottleneck. Variant of Conjugate Gradient(CG) solvers need global data in each iteration step.

My suggestion:
1. Run the case on 1CPU and then 2 or 4 cpus which are on the same node(depending if you have dual cores or quad) and check the speedup, if any.

2. Increase the domain size to so that ~100cells/cpu

3. Can you post your decomposePar logfile. It will give information about the communication required.

Rishi

[gcollecutt]

Rishi, Vivien,

I am using a custom solver which is a variant of reactingFoam and includes a fair amount of lagranigian particle tracking and computation. In my case I have 300K cells + 200K particles and obtain near linear speed improvement all the way to 64 CPUs (i.e. less than 5000 cells per process). My mesh is long and thin and so I use simple decomposition on the long axis, so each processor mesh has at most two processor boundaries.

Greg.

[ivan_cozza]

Originally Posted by sunnysun

Hi, Greg,

I did put the "-parallel" command...

May I know how fine is your mesh and how long is your simulation? Which solver did you use?

Many thanks!!

Vivien

Hi guys!
In my opinion the problem is in the p solver, try tu use GAMG as linear solver, in my incompressible simulations it put the p iterations down from 900 up to 20...
Ivan