[jtipton2]

Hello all,

I'm trying to simulate 2D flow through a channel with ribs to compare against published data. I'm using blockMesh and rhoSimpleFoam and have performed a mesh refinement study. Boundary conditions are relatively robust: fixed velocity inlet, neumann outlet, no slip walls, heat transfer on one wall, low-Re mesh at walls.

I've been able to run without problems on a single processor and obtain meaningful results.

I've then tried to run in parallel using "scotch" decomposition. 8 processors runs for a while and then abruptly ends with floating point exceptions. 9 processors runs to completion but gives spurious results.

I'm attaching residual plots which show odd behavior for the parallel run.

Has anyone seen this kind of behavior before? Any ideas as to the cause?


OpenFOAM-6

os:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

lscpu output:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          32
On-line CPU(s) list:             0-31
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           1
Model name:                      AMD Ryzen Threadripper 1950X 16-Core Processor
Stepping:                        1
CPU MHz:                         3181.944
BogoMIPS:                        6799.20
Virtualization:                  AMD-V
L1d cache:                       512 KiB
L1i cache:                       1 MiB
L2 cache:                        8 MiB
L3 cache:                        32 MiB

[Svetlana]

Does it work in serial and in parallel for an existing OpenFOAM tutorial that uses the same solver? Just asking in case you are able to narrow it down to either an installation problem or a problem with configuration of this particular simulation. Thanks!

[ybapat]

You can try manual decomposition in the flow direction instead of scotch.

[jtipton2]

Thanks for the good suggestions...

I tried the squareBend tutorial using rhoSimpleFoam for 3 cases:

1. Single Processor
2. mpirun 8 processors scotch method
3. mpirun 8 processors hierarchical method

I obtained identical results. So at least I know mpirun is working properly.

I'll test a manual decomposition on my problem next and report back.

[jtipton2]

As per the suggestion, I was able to run using "hierarchical" decomposition and subdividing along the axial direction.

I'm really surprised that the "scotch" method failed for what I thought was a simulation with robust boundary conditions...

Thanks so much for the suggestions.