|
[Sponsors] |
OpenFOAM running on two nodes (cluster) :confused: |
![]() |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
![]() |
![]() |
#1 |
Senior Member
Dongyue Li
Join Date: Jun 2012
Location: Beijing, China
Posts: 844
Rep Power: 18 ![]() |
Hi there, we use the mellanox infiniband cards to connect several nodes as a cluster. At the beginning, we found a very slow performance, we suspected it came from openmpi, since the communication rate was 20m/s, which is very slow. After several days attempt, we decided to use intelMPI. As usual, we run it by
mpirun -machinefile hostfiel -np 128 pisoFoam -parallel It showed similar slow performance as openmpi!!! Then we added the following parameters: mpirun -machinefile hostfiel -np 128 -env I_MPI_FABRICS shm:dapl pisoFoam -parallel It is much faster! However, we are still now sure if we are using our infiniband. Because it does not scale well. For one node with 64 cores, it takes 76s. For two nodes with 128 cores, it takes 51s. 2.5 million cells were used. How can we moniter the communication rate between these nodes? Thanks.
__________________
My OpenFOAM algorithm website: http://dyfluid.com By far the largest Chinese CFD-based forum: http://www.cfd-china.com/category/6/openfoam We provide lots of clusters to Chinese customers, and we are considering to do business overseas: http://dyfluid.com/DMCmodel.html |
|
![]() |
![]() |
![]() |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Adding compute nodes - new nodes faster than existing ones? | SLC | Hardware | 3 | November 14, 2019 15:35 |
[OpenFOAM.org] Problems with installing openfoam 5.0 on HPC Cluster | sjlouie91 | OpenFOAM Installation | 4 | January 20, 2019 15:35 |
OpenFOAM v3.0+ ?? | SBusch | OpenFOAM | 22 | December 26, 2016 14:24 |
[OpenFOAM.org] OpenFOAM Cluster Setup for Beginners | Ruli | OpenFOAM Installation | 7 | July 22, 2016 04:14 |
CFX4.3 -build analysis form | Chie Min | CFX | 5 | July 12, 2001 23:19 |