[merijn]

Hi,
I'm an HPC admin, trying to get SU2 to run in slurm on an opa fabric,
which seems to work succesfully, but produces no valid results.

so to test this I started the https://su2code.github.io/tutorials/Inviscid_ONERAM6/
example.

if I run it manually on one node like so:
SU2_CFD inv_ONERAM6.cfg
it produces a 29M restart_flow.dat and I can view the resulting surface_flow.vtk file in paraview for a visualization.

if I run the same simulation in parallel on two nodes x24 cores using
parallel_computation.py -n $SLURM_NTASKS -f inv_ONERAM6.cfg
it produces a clean log output, stating success, but the restart_flow is smaller, 15M restart_flow.dat (half the size of a manual 1 node run) and the .vtk file has no points to show.

on 8 nodes the filesize of restart_flow.dat becomes 3.6 MB
on 16 nodes the filesize of restart_flow.dat becomes 1.8MB

I do see the processes running on the allocated nodes.

im deducing, it breaks up the problem to multiple compute-nodes, and the restart_flow.dat will also contain only the partial solution of one compute-node. and the solition vtk file will be invalid. speculating, perhaps it just overwrites restart_flow.dat on each node.

I compiled SU2 with ./configure --prefix=/shared/Modules/SU2/6.1.0 --enable-mpi CFLAGS=-g -O3
with openmpi 4.0.0

how to debug this, how to solve this test job with multiple nodes?

merijn