[hiep.nguyentrong]

I have 3 computer node with 16 cores each, 1GE interconect.
I run fluent mpitest or fluent report bandwidth give me 100MB/s. But when i run a simple simulation, single phase, no interface, network monitor only show 10 to 15MB in/out.

Both intelmpi and openmpi give the same result.
How can i use all my network bandwidth in my simulation?

[naffrancois]

Hello,

This probably simply means that your run does not require a high network bandwidth. The amount of data transfered to the network is roughly speaking proportional to the number of cells located at the mesh partitions boundaries. If your mesh is not too big, if your number of partitions is not too big, the amount of data is not so big and 1GbE limit is not reached, which is good ! Note that high latency of 1GbE can also impact the performance, but I would not worry too much with only 3 nodes.

If you want to see if your network slows down your calculations, monitor the runtime using 1, then 2 and finally 3 nodes. If it stalls between 1 and 2 or 2 and 3, then you may have a network limiting the performance.

[wkernkamp]

Originally Posted by hiep.nguyentrong

I have 3 computer node with 16 cores each, 1GE interconect.
I run fluent mpitest or fluent report bandwidth give me 100MB/s. But when i run a simple simulation, single phase, no interface, network monitor only show 10 to 15MB in/out.

Both intelmpi and openmpi give the same result.
How can i use all my network bandwidth in my simulation?

In addition to the answer above, there may be another reason: The mpi communication happens between the nodes upon completion of an iteration step. So the majority of the time, the nodes are at work and not sending anything. So if for 15% of the time the nodes are communicating at maximum bandwith, you would see a bandwidth use of 15% * 100 MB/s = 15 MB/s

[hiep.nguyentrong]

Thank you for reply,
In my case, i have 10mil poly mesh and 48 cores. Fluent report that each iteration took 3.328s, data transfer per ite: 300MB, LE wall-clock/iter: 1.346(40%).

The mpi communication happens between the nodes upon completion of an iteration step

that's true. I mean the maximum data transfer is 15MB.
When i increase 3 computer node to 10, speed increase only 50%. That whh i think the network is the bottleneck

[hiep.nguyentrong]

http://upanh.tv/image/jPryQA

Here is the networks monitor

[wkernkamp]

Originally Posted by hiep.nguyentrong

Thank you for reply,
In my case, i have 10mil poly mesh and 48 cores. Fluent report that each iteration took 3.328s, data transfer per ite: 300MB, LE wall-clock/iter: 1.346(40%).
that's true. I mean the maximum data transfer is 15MB.
When i increase 3 computer node to 10, speed increase only 50%. That whh i think the network is the bottleneck

For 10 nodes yes. I have build a network with old ConnectX=3 cards. That should work better for you. If you get dual port cards, you can build a ring config. Still fast and no noisy hub. (Only for people with some vacation time due, haha)

[CFDfan]

Originally Posted by hiep.nguyentrong

http://upanh.tv/image/jPryQA

Here is the networks monitor

Hi, what's the name of the tool you've used for network traffic monitoring