[dlahaye]

Greetings,

1/ Performance of face-addressing in linear algebra operations

a/ how does face-adressing in OpenFOAM affect the data-locality (cache-misses, memory indirections) of the sparse matrix-vector multiplication. And the application of the preconditioner? See e.g. [1];

b/ single processor LDU matrix-vector multiplication requires access to three arrays, nl., L, D and U. The multiplication L*u and U*u requires twice non-uniform access to the vector u? Parallel implementation involves multiple L and U matrices, and makes matters even worse.

c/ PETSc provides the DMPLEX data structure. Does the use of this data-structure allow to brigde conflicting requirements for discretization (face-adressing) and for linear system solve (compressed-row storage)?;

d/ how does OpenFOAM overlay computation and communication in the parallel implementation of the matrix-vector multiplication; See e.g. [2];


2/ Pressure Solve within S/PIMPLE iteration using IC as preconditioner for CG

a/ results on https://develop.openfoam.com/committees/hpc show that for the presssure solve PETSc-ICC-CG (light blue) is slower than FOAM-DIC-PCG (dark blue) on 32, 64 and 128 nodes. PETSc by default applied the Manteufel shift in ICC, see [2,3]. What values of shift do OpenFOAM and PETSc apply? Is the difference in performance explained by the shift? Can the shift in OpenFOAM and PETSc be turned off? PETSc allows to set shift values using https://petsc.org/release/docs/manua...SetShiftAmount? Could try this first on 2D Poisson tutorials that PETSc provides.

b/ How is the coefficient matrix for the pressure correction formed?

c/ What exactly is DIC and FDIC in OpenFOAM? See [4]. Can the "D" be dropped in cases in which the coefficient matrix is by construction SPD, as in the case of the pressure matrix within the SIMPLE iteration?

d/ Can the PETSc-ICC-CG be further accelerated allowing some fill-in after approximate minimal degree reordering of the coefficient matrix? See https://petsc.org/release/docs/manua...atOrderingType (see e.g. [5]). Could try this first on 2D Poisson tutorials that PETSc provides.

e/ run PETSc-ICC-CG with monitoring (using -log_summary or -log_view) to gain better understanding of where computation and communication is spend. How does OpenFOAM overlay computation and communication?

f/ OpenFoam typically calls equation relaxation prior to calling linear solvers. Is this still required when calling PETSc solvers?

3/ Pressure Solve within S/PIMPLE iteration using GAMG as solver and as preconditioner for CG

a/ how does GAMG perform coarsening in case or x-anisotropic diffusion equation? Can algebraic and geometric complexity be printed as indication of required computational cost? See e.g. [6];

b/ how does mesh anisotropy (local refinement or changing coefficients) affect the performance of GAMG?

c/ PETSc provides two algebraic multigrid variants (PCGAMG and PCHYPRE). What can be learned by comparing both?

4/ Velocity Solve within S/PIMPLE iteration

a/ Can the same approximate minimal degree reordering, incomplete LU factorization and Krylov acceleration be used to accelerate the velocities components solves?

5/ Recycling Information for Pressure and Velocity Solve within S/PIMPLE iteration

a/ Can the GAMG multigrid hierarchy be kept fixed (frozen) throughout the entire P/SIMPLE iteration? Is this a manner to implement a deflation technique using multigrid deflation vectors? Is this a manner to examine whether the non-scalability of GAMG on large processor count is due the GAMG setup or GAMG solve phase?

6/ Block Coupled Pressure-Velocity Solve

a/ Can PETSC4FOAM be extended to copy both the velocity and pressure matrix to a PETSC coupled matrix? Can the FieldSplit technology that PETSc provides be employed to solve the coupled formulation efficiently? (see PETSc manual)

7/ Scalar Transport using System of Non-Linear Equation Systems (SNES) Components in PETSc

a/ Can the non-linearity of scalarTransportFOAM be solved using a Newton iteration with Jacobian provided by automatic differentiation?

8/ Time Integration using Time Stepping (TS) Components in PETSc

a/ Can advanced time-stepping (higher order BDF or Runge-Kutta) be used to accelarate the time-stepping procedure?

References

[1] https://dl.acm.org/doi/pdf/10.1145/3...90a_6cZiK0284w

[2] https://netlib.org/linalg/html_templ...00000000000000

[3] https://petsc.org/release/docs/manua...ICC.html#PCICC

[4] https://netlib.org/templates/templates.pdf, Manteufel Shift, Section 3.4.1 Creating an Incomplete Factorization

[5] OpenFOAM Preconditioners

[6] R. Idema, G. Papaefthymiou, D. Lahaye, C. Vuik and L. van der Sluis, "Towards Faster Solution of Large Power Flow Problems," in IEEE Transactions on Power Systems, vol. 28, no. 4, pp. 4918-4925, Nov. 2013, doi: 10.1109/TPWRS.2013.2252631.

[7] https://github.com/JuliaLinearAlgebr...icMultigrid.jl

Best wishes, Domenico.