[saladbowl]

I am alternating between running a IDDES case on a local cluster and larger national lab cluster (HPC). This is because I only get so many chunks of time on the HPC cluster and in the meantime my idea is to run on the local cluster so as to minimize total clock time. However, I am bothered by convergence issues in the pressure solver when I move data from one cluster to the other.

On the local cluster I decompose the domain in 100 subdomains, with 100 cores. Sometimes I can get to 200 depending on the usage of the cluster. When I am ready to run on the HPC cluster, I copy the resulting time, constant and system folders to the HPC cluster with rsync. The I run `reconstructPar -latestTime` followed by `decomposePar -latestTime`, `renumberMesh -overwrite` and start the simulation using `startTime -latestTime` in controlDict. I use about 1000 cores on the HPC cluster. Running on more cores doesn't show favorable scaling and allotted CPU time is fixed.

Upon restart, the number of iterations for the pressure solver reaches `maxIter`, which is already set to a large value (5000). This goes on for many time steps, using up valuable computational time on the HPC cluster.

I should say that I could avoid all this running on the HPC cluster only, but the issue is that it is much easier to run different tests with solvers and such on the local cluster. I also was given time on the HPC cluster after I had been running some time on the local cluster. I am trying to evolve from a RANS steady state solution.

The OpenFOAM versions on the local cluster and HPC cluster are the same, however they were compiled with different gnu and openmpi versions. On both clusters OpenFOAM is compiled with WM_LABEL=64.

I suspect that when I decompose the domain into a different number of subdomains that there will be differences because most preconditioners are parallel inconsistent. However, I did not expect that upon restart the computational cost would be so high.

This issue is baffling me. I am running with pimpleFoam in piso mode. I use scotch for domain decomposition.

Any pointers are highly appreciated. Thanks!

[saladbowl]

Upon further tests the issue reported above was not reproduced. Not sure what may have been the culprit. In any case, now I find the expected behavior, i.e., upon restart with a different domain decomposition, there is an initial increase in iterations, but nothing abnormal.