[Geoff67]

I have been encountering problems with runs crashing due to fatal overflow in the linear solver at seemingly random points in the transient solution, while running on multiple compute nodes on a HPC cluster.

I am doubting the model set up, time step, or mesh are to blame. The identical job appears to run fine if I use a local computer, a different single node compute server or a single node on the HPC cluster. I had been running similar similar cases without problems for a while now but on a single compute node.

The fatal overflow occurs well into the transient simulation. There is nothing physically unique happening then. when the run crashes and I restart it from a back up file it crashes at the exact same time step. If I restart on a different single node machine the simulation runs past the point it crashed at. If i stop that run that went past the crash point and use it to initialize another run on the HPC cluster using 2 nodes it will run for a little while longer then crash again.

I compared the output files of the single and multi node jobs and noticed; 'multi-step communication method for linear solver' was being activated when I used 2 nodes but not for a single node. I disable the multi-step communication method and tried another 2 node job. It appeared to solve the problem at first. It ran past the point where it crashed previously but it has now crashed later into the simulation.

I was reading that the partitioning method can behave differently with different numbers of core. Please correct me if I am wrong but if the partitioning method was the problem, the run would crash on the first time step.

Unfortunately it is more desirable for me to run one 2-node job than two 1-node jobs due to the scheduler on the HPC cluster. So I would really like to be able to get to the root of this problem.

[ghorrocks]

The first thing to try in situations like this is a different partitioning algorithm. Your statement saying that poor partitioning would cause the run to crash in the first time step is incorrect - when a region of very high gradients (eg a shock wave or a free surface) lies on top of a partition boundary this can cause convergence problems. This can happen anywhere in a simulation as shocks and free surfaces can move.

So try some of the other partitioning methods and see if that fixes it.

Also check that your HPC is stable as well - does a simulation which is known to run well (eg a tutorial example) run successfully in the same parallel mode?

[Opaque]

Since it is a sliding mesh simulation, can you tell if there is a relation to the relative configuration when it fails?

That is, will it fail every time it passes through the same relative configuration?

[Geoff67]

Thanks for the response. That is good to know about the partitioning methods. I seem to have resolved the problem the problem by specifying the compute nodes differently. Using 'user specified partitioning direction' seems to have improved my convergence rate slightly.