[hand90]

Hi everyone

After mainly running RANS and URANS simulations, I am attempting to run an LES case with the WALE sub-grid model. I am aware that LES is computational much more expensive due to the mesh requirement and resulting time step of the simulation. However when running the simulation in ANSYS CFX the computational time of a single iteration, one inner loop, is unexpectedly long.

The mesh size is "only" 14M which is rather small for an LES case, running on 62 cores resulting in 225k elements per core. Now for a single inner loop the simulation takes ~49s.

I am just curious what the experience is of other users.

Just a little more back ground information on the simulation.
Simulation batch file:

echo "working directory = "$SLURM_SUBMIT_DIR
export SLURM_NODEFILE=nodes.$$
srun hostname -s | sort > $SLURM_NODEFILE
nodelist=`cat $SLURM_NODEFILE`
nodelist=` echo $nodelist | sed -e 's/ /,/g'`
echo "nodelist= "$nodelist

module load ansys/182
#SBATCH -cwd

cfx5solve -def LES.def -continue-from-file 500_full.bak -double -par-dist $nodelist -start-method "Intel MPI Distributed Parallel" -size 1.2 -sizepar 1.2

Simulation itself:

-Modeling of a stator turbine blade
-time step = 1e-6 (CFL≈1)
-GGI in y (spanwise) and z (cascade) direction (1:1 mesh connection).
-No transient results saved

[ghorrocks]

As LES models are very expensive it is worth doing a careful validation of everything before you commit to the big model. Have you checked you are getting a believable turbulence decay spectrum? Have you checked that you really need the time step size and convergence criteria you are using? Relaxing any of these things will speed the simulation up a lot.

Also, rather than targeting CFL =~1, you will probably find the simulation runs better if you target 3-5 coeff loops per iteration.

Finally, I would do a benchmark on your computer cluster to check that it is actually giving you a useful speedup at 62 cores. If your cluster just has ethernet connections then I don't think you will ever get a good speed up over 62 cores (you will need infiniband or another high-end interconnect). This will be even more important if you have lots of cores per compute node. For instance, I would estimate that a 60 core simulation run on 4x 15 core compute nodes connected with ethernet will be slower than a 32 core simulation run on 4x8 core compute nodes.