|
[Sponsors] |
Worse parallel efficiency with openfoam running on HPC cluster |
![]() |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
![]() |
![]() |
#1 |
New Member
Jin Zhang
Join Date: May 2018
Location: Germany
Posts: 15
Rep Power: 8 ![]() |
Hi all,
I am now testing benchmark case (cavity) on our own HPC cluster. The total number of cells is 15,000,000. As it is newly installed, I found that it cannot perfrom good parallel effiency with increasing nodes. We use intel-2020 for compiler and IntelMPI. For HPC cluster, each node has 64 CPUs and we totally have 5 nodes. This is the jobfile for intel-2020 and also the speed-up plot. I dont't know whether it is due to any wrong settings for MPI. Do you have any suggestions? Thanks! Jin |
|
![]() |
![]() |
![]() |
![]() |
#2 |
Senior Member
Gerhard Holzinger
Join Date: Feb 2012
Location: Austria
Posts: 341
Rep Power: 28 ![]() ![]() |
This is normal behaviour. Parallel efficiency levels-off at some point. This is why you see a speed-up from using 1, 2 or 3 nodes. However, the reason why 5 nodes performs poorer than using 4 nodes, is the ever increasing communications-workload in contrast to the ever decreasing per-node computational workload.
When using 3 nodes (64 CPUs each), your per-node computational effort is around 78.000 cells for each parallel process. When using 5 nodes, every parallel process only deals with roughly 46.000 cells. So, while using more and more parallel process seems a good idea, we need to bear in mind, that the effort of communication scales super-linearly with the number of parallel processes, while the reduction of computational effort only scales linearly. If you double the number of CPUs devoted to a parallel simulation, the per-node computational size is cut in half, while the effort for communication between all these parallel processes increases by a factor of between 2 and 4 (i.e. the squre of 2). The factor for the incease oin communication depends on your decomposition. If you decompose badly, each parallel process needs to communicate with every other process. If you decompose well, each parallel process only needs to communicate with a few processes. |
|
![]() |
![]() |
![]() |
Tags |
hpc, intel-2020, intelmpi, parallel |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
poor pressure convergence upon restart of parallel case | saladbowl | OpenFOAM Running, Solving & CFD | 1 | February 7, 2022 04:55 |
[OpenFOAM.org] Problems with installing openfoam 5.0 on HPC Cluster | sjlouie91 | OpenFOAM Installation | 4 | January 20, 2019 15:35 |
How to run SU2 on HPC cluster in parallel on HPC cluster? | Samirs | Main CFD Forum | 0 | July 13, 2018 00:44 |
OpenFOAM can't be run in parallel in cluster | sibo | OpenFOAM Running, Solving & CFD | 4 | February 21, 2017 16:29 |
Fluent 14.0 file not running in parallel mode in cluster | tejakalva | FLUENT | 0 | February 4, 2015 07:02 |