[JLinfield]

Hi there,

We are using SU2 (v6.0 Falcon release) to simulate the flow around a civil aircraft wing/body. No mesh adaption or optimisation. The mesh is a high quality quad dominant surface mesh with typical anisotropic refinement around areas of curvature, and 50 or so hex boundary layer elements (Y+<1) with a tetra far field, circa 60M elements.

We are running on a 160 core CentOS cluster with InfiniBand interconnect and Open MPI Version 2.1.1.

We have found that a robust method for reaching convergence is to carry out stages of solution control, starting with a fixed low CFL for circa 1500 iterations, followed by a CFL ramp to say 20 over another 2500 its, then if required we move on to a fixed CL mode.

The issue is that, somewhat randomly, we find that the solution diverges immediately after a restart……..Doesn’t do a single iteration. We have carried out tests where we extend the number of iterations in the phase before to check that the solution is not actually blowing up after that number of iterations, and it’s fine.

I’m not sure if this is associated with a cluster/MPI issue or SU2 bug when SU2 writes the binary restart file in parallel, and perhaps it is corrupted causing divergence at restart?

Has anyone else faced this issue, or does anyone have any advice?

Any help greatly appreciated!

Thanks
James

[tjim]

Hi James,

Did you figure out what caused this?

Thanks,
Tim

[JLinfield]

Hi Tim,

Yes, we did figure out the issue in this case.

Embarrassingly it was not an issue with SU2 at all. It was due to a problem with our compute cluster. One of the nodes would fall over while SU2 was writing the restart file, so the file was corrupted and caused divergence upon restart.

Thanks
James

[tjim]

Thanks for the follow-up!

Kind regards,
Tim