[evcelica]

It seems like I have tried everything. I have the infiniband network working using IPoIB and the openfabrics 3.1 drivers.
OpenSM is running, ibping works, I can transfer files over the network no problem. But MPItest doesn't work.
The cards are 20Gbps Mellanox infinihost III.

I'm using platform MPI:
Every time I run the MPItest I get errors:
Cannot find active interface/port
can't initialize RDMA device
cannot initialize RDMA port

It works fine over Ethernet.
Any advice would be greatly appreciated.

Thanks in advance,
Erik

[evcelica]

I got it working, I just had to use the Mellanox 2.1.2 drivers for windows server 2008 R2, and it worked like a charm. 1750MB/sec bandwidth now.

[ghost82]

Hi Erik,
I'm interested in this topic.

Is your network made of only 2 machines without a switch right?

What is the model of your cards?

If you have time what were the main steps you did to make the ipoib working?

Why "only" 1750MB/sec?Should not be about 2500MB/s?

Thank you very much

Daniele

[evcelica]

The cards are Mellanox MHGS18-XTC rev A3, Right now I'm only using two with no switch, but in the future I will add a switch when I have more than 2 nodes.
IPoIB was simple to get working, but that is not what you want to use, you can only get about 250 MB/sec max with TCP protocol and IPoIB, and it uses a TON of CPU to do that. You want to use Infinibands native protocol instead.
I'm assuming there is some overhead which is why 2500MB/s wasn't obtained, but I'm not sure. I was working for a long time not getting anything to work except the IPoIB with TCP, so as soon as I got it to work at 1750 using the infiniband protocol, I was happy and didn't care about much else.

Install the cards into a PCIEx8 capable slot.
Install the drivers mentioned, or whatever are the most recent drivers according to which cards you have.
During install of the drivers, check the enable "winsock direct protocol" box. This is what allows windows to communicate through the native infiniband protocol, and not TCP.
Go to services in the task manager and make sure to start OPEN SM, set it to start automatically. This is the subnet manager which you have to use if you are not using a switch.
Test it out with MPItest. It should use the infiniband network automatically automatically if you use the computer names for the "machines" file.

[ghost82]

Thanks Erik for your detailed reply.
I was searching for 2 infiniband cards and in my list there are also a couple of cheap MHGS18-XTC: happy to know that they're working well with windows 7.
Before buying and asking to you I was searching on the net for drivers/softwares and it's not too clear: from the mellanox website I saw winof download only for win 2008/2012; then after reading your comment I discovered that the link for 2008 server R2 is for win 7.

Last 2 questions:
- is it ok 20 Gbs cards or 40 Gbs generation should be better? In other words is the bandwith enough for a "standard" simulation with 2 nodes?
- do you remember if after installing the mellanox drivers you had to manually copy some dlls in system32 directory (such as complib.dll, libibumad.dll, winmad.dll and some other)?

Again thank you very much!

Daniele

[evcelica]

I'm sure the newer 40gbps cards are technically better, but with only 2 nodes or probably many more, the 20gbps is not a really a bottleneck. I doubt you would see any difference at all between different cards with a few number of nodes.

I did not have to copy any .dll files.