[feacluster]

I am testing out the 1.7 million cell motorbike tutorial on Azure. Long story short, it solves in 190 seconds on one 36 cpu machine ( F72s_v2 ). When I run it on two machines ( 72 cpus ), it runs in 600 seconds!

When I do top on both machines, I see 32 threads of simplefoam running at 100% . So it seems to be working correctly.

Is the network too slow for mpi ? I know I have run openfoam on a cluster with 10gigE network and it seemed to scale well..

I am also aware there are HPC specific nodes on Azure. But I want to get my feet wet with the basics before trying those.

[simrego]

Hi!


That 72 cores is not too much for 1.7 million cells??? You will have more processor faces than internal... What if you try with 17 million cells? (But i think 72 cores will be still a lot for 17 million cells)

[feacluster]

Originally Posted by simrego

Hi!
That 72 cores is not too much for 1.7 million cells??? You will have more processor faces than internal... What if you try with 17 million cells? (But i think 72 cores will be still a lot for 17 million cells)

The same model scales well upto 160 cores on my bare-metal cluster . After 160 cores I see minimal speedup..

[feacluster]

I ran the Ohio State benchmarks to measure latency on Azure:

http://mvapich.cse.ohio-state.edu/benchmarks/

I got around 60 micro seconds which is quite high . For our infiniband cluster I get 1-2 micro seconds. I suppose that is the reason for my slowdown when using multiple nodes?

I know people run Openfoam on Amazon aws. I am not sure how their latency numbers are. I would imagine they are in the same ballpark?

[wyldckat]

Quick answer: 10 GigE is known to have a considerably larger latency than Infiniband. 60 ms vs 1-2 ms sound about right.


Searching online for "azure infiniband" does return several answers, namely how to use Infiniband on Azure.

[Christophe]

I have a bunch of high end workstations with a 10 GigE interconnect. Did a lot of benchmarking. Once I added a third machine to go above 50 cores there was a huge diminishing return. I also have a cluster w 56 GB infiniband and it scaled perfectly with the same model I tested on the desktops.

Would def stay away from 10 GigE if possible.

I think a lot of instances on Azure have at least 25 Gbps.

[akidess]

Originally Posted by Christophe

Would def stay away from 10 GigE if possible.

I think a lot of instances on Azure have at least 25 Gbps.

They could have 10000 Gbps and still show bad scaling. It's the latency that counts.

[feacluster]

I re-ran my Openfoam benchmark on the Azure HPC compute instances ( H16r ). The perfomance scaled well, same as my bare-metal cluster.

These are 16 core machines with infiniband network. The catch is they are only compatible with Intel MPI . So you need your own cluster edition Intel compiler. I was able to compile on my bare-metal cluster and copy the executable to Azure and it ran fine..

[hypersonic]

Hi

I'm running ANSYS on Azure (8*H16r), but can only get RDMA inifiband at 1XDRR (5Gbit)
Anyone had any luck getting 4XFDR (56Gbit) to work with Intel MPI - as they say is possible on the Azure HPC page?

Thanks

[feacluster]

Originally Posted by hypersonic

Hi

I'm running ANSYS on Azure (8*H16r), but can only get RDMA inifiband at 1XDRR (5Gbit)
Anyone had any luck getting 4XFDR (56Gbit) to work with Intel MPI - as they say is possible on the Azure HPC page?

Thanks

I am not any expert in Infiniband, but I always try and check the basics first. Try running this and report the results:


http://mvapich.cse.ohio-state.edu/benchmarks/

• osu_bw - Bandwidth Test
• osu_latency - Latency Test

[fertinaz]

Hello

As Faraz has already pointed out, performance bottleneck you encounter should be due to the network latency.

Anyone had any luck getting 4XFDR (56Gbit) to work with Intel MPI - as they say is possible on the Azure HPC page?

Infiniband switches are not directly available on Azure even if you use H-type instances, afaik. Is there a report where they achieve that performance? Instead, RDMA is used which should be fine enough. There are a few things you can check:

• Set fabrics to dapl: Assuming that you're using IntelMPI, set env-var to "ofa-v2-ib0"
• Increase debug level to see which fabrics are being used
• Run ifconfig: Make sure eth is utilized
• Check rdma conf file: Grep file "/etc/rdma/dat.conf" and see if ib0 is referenced to eth

Hope this helps

[jairoandres]

I think you should limit to 50k cells per processor.