[sharonyue]

Hi there, we use the mellanox infiniband cards to connect several nodes as a cluster. At the beginning, we found a very slow performance, we suspected it came from openmpi, since the communication rate was 20m/s, which is very slow. After several days attempt, we decided to use intelMPI. As usual, we run it by

mpirun -machinefile hostfiel -np 128 pisoFoam -parallel

It showed similar slow performance as openmpi!!! Then we added the following parameters:

mpirun -machinefile hostfiel -np 128 -env I_MPI_FABRICS shm:dapl pisoFoam -parallel

It is much faster! However, we are still now sure if we are using our infiniband. Because it does not scale well. For one node with 64 cores, it takes 76s. For two nodes with 128 cores, it takes 51s. 2.5 million cells were used. How can we moniter the communication rate between these nodes?

Thanks.