[msrinath80]

For my approx 1.2 million cells (single-phase) icoFoam vortex shedding case, here is the output after the first three time steps and the corresponding fvSolution pressure solver entries:

ICCG: (p ICCG 1e-06 0;)

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : /home/madhavan/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxAMD64Gcc4DPOpt/icoFoa m /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam/. one_sq_cyl_3d_unsteady
Date : Nov 07 2006
Time : 12:27:17
Host : masternode
PID : 21882
Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam/.
Case : one_sq_cyl_3d_unsteady
Nprocs : 1
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi

Reading probeLocations

Constructing probes


Starting time loop

Time = 0.1

Mean and max Courant Numbers = 0 0.0741732
BICCG: Solving for Ux, Initial residual = 1, Final residual = 7.60531e-08, No Iterations 3
BICCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
BICCG: Solving for Uz, Initial residual = 0, Final residual = 0, No Iterations 0
ICCG: Solving for p, Initial residual = 1, Final residual = 9.73006e-07, No Iterations 825
time step continuity errors : sum local = 4.55914e-10, global = -5.75299e-13, cumulative = -5.75299e-13
ICCG: Solving for p, Initial residual = 0.00108292, Final residual = 9.32198e-07, No Iterations 685
time step continuity errors : sum local = 3.75695e-07, global = -7.83026e-10, cumulative = -7.83601e-10

Wall patch = 0
Wall patch name = ChannelWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 2.01178e-05
pressureDragCoefficient = 0
viscDragCoefficient = 2.01178e-05
LiftCoefficient = 1.41566e-11


Wall patch = 1
Wall patch name = ObstacleWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 4.79068e-06
pressureDragCoefficient = 4.62468e-06
viscDragCoefficient = 1.66004e-07
LiftCoefficient = 8.15603e-14

ExecutionTime = 347.02 s ClockTime = 347 s

Time = 0.2

Mean and max Courant Numbers = 0.108516 4.04533
BICCG: Solving for Ux, Initial residual = 0.823434, Final residual = 4.45773e-06, No Iterations 3
BICCG: Solving for Uy, Initial residual = 0.329552, Final residual = 3.46137e-06, No Iterations 4
BICCG: Solving for Uz, Initial residual = 0.362595, Final residual = 4.31759e-06, No Iterations 3
ICCG: Solving for p, Initial residual = 0.00371997, Final residual = 9.84174e-07, No Iterations 719
time step continuity errors : sum local = 1.53182e-07, global = 3.88971e-10, cumulative = -3.9463e-10
ICCG: Solving for p, Initial residual = 0.00419475, Final residual = 9.92882e-07, No Iterations 709
time step continuity errors : sum local = 7.36119e-08, global = -2.94289e-10, cumulative = -6.88919e-10

Wall patch = 0
Wall patch name = ChannelWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 2.11014e-05
pressureDragCoefficient = 0
viscDragCoefficient = 2.11014e-05
LiftCoefficient = 2.18023e-11


Wall patch = 1
Wall patch name = ObstacleWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = -9.70849e-07
pressureDragCoefficient = -1.11824e-06
viscDragCoefficient = 1.47396e-07
LiftCoefficient = -2.70848e-11

ExecutionTime = 639.75 s ClockTime = 640 s

-----------------------------------------------
-----------------------------------------------

AMG (p AMG 1e-06 0 1000;):

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : /home/madhavan/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxAMD64Gcc4DPOpt/icoFoa m /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam/. one_sq_cyl_3d_unsteady
Date : Nov 07 2006
Time : 11:29:35
Host : masternode
PID : 21242
Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam/.
Case : one_sq_cyl_3d_unsteady
Nprocs : 1
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi

Reading probeLocations

Constructing probes


Starting time loop

Time = 0.1

Mean and max Courant Numbers = 0 0.0741732
BICCG: Solving for Ux, Initial residual = 1, Final residual = 7.60531e-08, No Iterations 3
BICCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
BICCG: Solving for Uz, Initial residual = 0, Final residual = 0, No Iterations 0
AMG: Solving for p, Initial residual = 1, Final residual = 9.80164e-07, No Iterations 105
time step continuity errors : sum local = 4.59248e-10, global = 2.47297e-14, cumulative = 2.47297e-14
AMG: Solving for p, Initial residual = 0.00108292, Final residual = 9.69421e-07, No Iterations 48
time step continuity errors : sum local = 3.90696e-07, global = -1.15826e-11, cumulative = -1.15579e-11

Wall patch = 0
Wall patch name = ChannelWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 2.01179e-05
pressureDragCoefficient = 0
viscDragCoefficient = 2.01179e-05
LiftCoefficient = -5.64623e-09


Wall patch = 1
Wall patch name = ObstacleWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 4.79158e-06
pressureDragCoefficient = 4.62556e-06
viscDragCoefficient = 1.6602e-07
LiftCoefficient = 1.45852e-10

ExecutionTime = 146.7 s ClockTime = 147 s

Time = 0.2

Mean and max Courant Numbers = 0.108519 4.04592
BICCG: Solving for Ux, Initial residual = 0.823441, Final residual = 4.46106e-06, No Iterations 3
BICCG: Solving for Uy, Initial residual = 0.329408, Final residual = 3.4621e-06, No Iterations 4
BICCG: Solving for Uz, Initial residual = 0.362669, Final residual = 4.339e-06, No Iterations 3
AMG: Solving for p, Initial residual = 0.0037213, Final residual = 9.14087e-07, No Iterations 82
time step continuity errors : sum local = 1.42274e-07, global = -4.78435e-13, cumulative = -1.20363e-11
AMG: Solving for p, Initial residual = 0.00419589, Final residual = 9.94429e-07, No Iterations 75
time step continuity errors : sum local = 7.37282e-08, global = 1.93804e-13, cumulative = -1.18425e-11

Wall patch = 0
Wall patch name = ChannelWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 2.11013e-05
pressureDragCoefficient = 0
viscDragCoefficient = 2.11013e-05
LiftCoefficient = 8.69834e-09


Wall patch = 1
Wall patch name = ObstacleWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = -9.72347e-07
pressureDragCoefficient = -1.11973e-06
viscDragCoefficient = 1.47384e-07
LiftCoefficient = -2.83295e-10

ExecutionTime = 294.14 s ClockTime = 295 s

-------------------------------------------
-------------------------------------------


AMG (p AMG 1e-06 0 100;):

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : /home/madhavan/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxAMD64Gcc4DPOpt/icoFoa m /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam/. one_sq_cyl_3d_unsteady
Date : Nov 07 2006
Time : 12:20:05
Host : masternode
PID : 21848
Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam/.
Case : one_sq_cyl_3d_unsteady
Nprocs : 1
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi

Reading probeLocations

Constructing probes


Starting time loop

Time = 0.1

Mean and max Courant Numbers = 0 0.0741732
BICCG: Solving for Ux, Initial residual = 1, Final residual = 7.60531e-08, No Iterations 3
BICCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
BICCG: Solving for Uz, Initial residual = 0, Final residual = 0, No Iterations 0
AMG: Solving for p, Initial residual = 1, Final residual = 8.07098e-07, No Iterations 102
time step continuity errors : sum local = 3.78169e-10, global = 1.46199e-14, cumulative = 1.46199e-14
AMG: Solving for p, Initial residual = 0.00108292, Final residual = 8.70596e-07, No Iterations 48
time step continuity errors : sum local = 3.50868e-07, global = 3.39341e-10, cumulative = 3.39356e-10

Wall patch = 0
Wall patch name = ChannelWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 2.01179e-05
pressureDragCoefficient = 0
viscDragCoefficient = 2.01179e-05
LiftCoefficient = -4.35326e-09


Wall patch = 1
Wall patch name = ObstacleWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 4.79156e-06
pressureDragCoefficient = 4.62554e-06
viscDragCoefficient = 1.6602e-07
LiftCoefficient = 8.678e-11

ExecutionTime = 108.75 s ClockTime = 109 s

Time = 0.2

Mean and max Courant Numbers = 0.10852 4.04587
BICCG: Solving for Ux, Initial residual = 0.823445, Final residual = 4.46093e-06, No Iterations 3
BICCG: Solving for Uy, Initial residual = 0.329439, Final residual = 3.4626e-06, No Iterations 4
BICCG: Solving for Uz, Initial residual = 0.362502, Final residual = 4.33196e-06, No Iterations 3
AMG: Solving for p, Initial residual = 0.00372114, Final residual = 7.40435e-07, No Iterations 68
time step continuity errors : sum local = 1.15246e-07, global = -2.12212e-12, cumulative = 3.37234e-10
AMG: Solving for p, Initial residual = 0.00419577, Final residual = 9.12739e-07, No Iterations 60
time step continuity errors : sum local = 6.76724e-08, global = 1.6342e-11, cumulative = 3.53576e-10

Wall patch = 0
Wall patch name = ChannelWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = 2.11013e-05
pressureDragCoefficient = 0
viscDragCoefficient = 2.11013e-05
LiftCoefficient = 7.399e-09


Wall patch = 1
Wall patch name = ObstacleWalls
Uav = (1 0 0)
Aref = 1
nu = nu [0 2 -1 0 0 0 0] 1.004e-06
DragCoefficient = -9.72242e-07
pressureDragCoefficient = -1.11963e-06
viscDragCoefficient = 1.47384e-07
LiftCoefficient = -2.05608e-10

ExecutionTime = 191.77 s ClockTime = 192 s

Clearly, the AMG solver is better than ICCG for this case. Are there any guidelines on what number to use at the coarsest level for the AMG solver?

[hjasak]

On the top level of AMG you have a system of equations that needs to be solved to a tight tolerance and the question is how much work to leave for the top level. In serial runs, anything between a dozen and a couple of hundred cells will give you the same result (1000 is too much). In parallel, it is the communications that matter, which means you need to be more careful: make sure there's a sufficient number of cells for each CPU. You also need to worry about the case where you have multiple separate domains. In general, 20-30 cells per CPU in parallel will do you fine (until I do some more tricks to speed this up). :-)

Hrv

[msrinath80]

Thanks for that insightful reply Hrv. Much appreciated.