[brahmsdr]

Hi,

I have an issue running my simulation case with the icoFOAM solver in parallel mode on the cluster. I am using the ESI version of OpenFOAM v2212. I used the 'scotch' mesh decomposition method. Here's a small trial and error I did on flow past a cylinder with a mesh count of ~200k:

32 cores (4 nodes), 256GB (64GB per node, 8GB per cpu/task): sim Time = 0.2 took 2927s (dt=0.005)

16 cores (2 nodes), 256GB (128GB per node, 16GB per cpu/task): sim Time = 0.2 took 2571s (dt=0.005)

8 cores (2 nodes), 256GB (128GB per node, 32GB per cpu/task): sim Time = 0.2 took 1285s (dt=0.005)

4 cores (2 nodes), 256GB (128GB per node, 64GB per cpu/task): sim Time = 0.2 took 784s (dt=0.005)

2 cores (2 nodes), 256GB (128GB per node, 128GB per cpu/task): sim Time = 0.2 took 34s (dt=0.005)

My trial-and-error cases here suggested that the issue might be related to insufficient memory because increasing memory per CPU helps to make the simulation faster. The fact that it is running faster with a small number of processors still puzzles me. One possibility here might be caused by an inefficiency that arises in the parallel communication for a case with a small mesh count. However, I tried running several tests with a mesh count of ~3mil and the same issue arose:

72 cores (2 nodes), 288GB (144GB per node, 4GB per cpu/task): Simulation Time = 0.1 failed (not enough memory) (dt=0.005)

4 cores (4 nodes), 256GB (128GB per node, 64GB per cpu/task): Simulation Time = 0.1 took 365s (dt=0.005)

The smaller number of processors with large memory assigned per processor still gives a faster run. This will be an issue for example for a larger number of mesh since it will increase the amount of memory needed. I would also expect that a higher number of processors increases the efficiency, however, this is not the case here. A similar case was also reported here, where a smaller number of processors gave faster running time (clock time) and so far, I couldn't find the answer why.

Does anyone know what might be the issue here? Am I implementing something incorrectly or is this a configuration problem?

Any recommendations or suggestions will be very helpful. Thank you!