[llirik]

Good day to all
need your help with motorBike OpenFOam benchmark
I have a prototype ARM cpu based cluster
125 separate nodes = 1000 cores in total
Each node hardware configuration is:
8cores (Cortex-A53) + 2GB LPDDR3 ram + 16Gb emmc
nodes are interconnected 1Gb ethernet
OpenFoam-v2006
Ubuntu 20.04 aarch64
mpich-3.3.2
GlusterFS mount point with all benchmarks

I've tried benchmarks such as:
HPLinpack 2.3 -- High-Performance Linpack benchmark
NAS Parallel Benchmarks 3.4

they all work fine on 1000 cores 125 nodes and I decided to test openfoam motorBike benchmark
I checked first on single node: 8cores - everything is OK
but when i start to run on 8nodes/16nodes/64nodes - 64cores/128cores/512cores respectively
I have a problem at this stage:

mpirun -np 64 -hostfile ~/mpi_hosts8 snappyHexMesh -overwrite -parallel

/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: v2006 |
| \\ / A nd | Website: www.openfoam.com |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : _b45f8f6f58-20200629 OPENFOAM=2006
Arch : "LSB;label=32;scalar=64"
Exec : snappyHexMesh -overwrite -parallel
Date : Dec 12 2020
Time : 15:03:08
Host : node00001
PID : 26455
I/O : uncollated
Case : /storage-pool/HPC/bench_template/run_64
nProcs : 64
Hosts :
(
(node00001 1)
(node00002 1)
(node00003 1)
(node00004 1)
----------------
----------------
----------------

Edge intersection testing:
Number of edges : 32256
Number of edges to retest : 0
Number of intersected edges : 26
Selecting decompositionMethod hierarchical [64]

----------------

Feature refinement iteration 0
Balanced mesh in = 5.82 s

Feature refinement iteration 1
Balanced mesh in = 6.47 s

Feature refinement iteration 2
.........
Refined mesh in = 0.3 s
After refinement feature refinement iteration 2 : cells:14951 faces:47952 points:18169
Cells per refinement level:
0 10175
1 359
2 841
3 3576

аnd after that the execution freezes, the processor load on the nodes drops from 100% to 70%
The next "Balanced mesh in = " result takes a long time 600-900s
What i am doing wrong?
I have no experience using OpenFOAM I am doing research and i need to show different benchmarks on my prototype
I will be grateful for any help or advice