[Pvfs2-users] no connection on infiniband dual port card
kschoche at gmail.com
Mon Jan 30 10:41:24 EST 2012
Hi All -
I dont think we ever got around to testing this with multiple ports
active on each HCA when we wrote it, so I believe I hard coded it to
just default to the first port... iirc we tried to bring up the 2nd
port at one point and found that there were some memory exhaustion
issues when using more than one port AND the default HCA
buffers/MTUs/etc on the cards that this was primarily tested on so it
went back to 1.
I wouldn't recommend changing this via a hard code for obvious
reasons, but at the same time it probably wouldn't take more than
20-30 lines of code to fix this up to take more than one port. I'll
try to take a look at it.
On Mon, Jan 30, 2012 at 9:21 AM, Phil Carns <carns at mcs.anl.gov> wrote:
> Hi Vlad,
> Let me prefact this by saying that I don't know much about InfiniBand, but I
> was curious about this and looked in the src/io/bmi/bmi_ib/openib.c source
> code, where I saw this comment:
> od->nic_port = IBV_PORT; /* maybe let this be configurable */
> IBV_PORT is hard coded to 1 with a #define at the top of the file. This is
> really just a guess on my part, but maybe you could try modifying that
> #define to 2 instead, recompile, and see if that works? As the comment
> indicates, this port value should probably be configurable but it doesn't
> look like it is right now.
> On 01/30/2012 09:44 AM, vlad wrote:
>> I`m evaluating now version 2.8.5 and I have been able to connect and
>> use hosts with single port infiniband cards (Mellanox Infinihost III, )
>> using the ib0 Interface, connection set to
>> Integrating of hosts with 2-port cards fails. On this hosts I have
>> plugged in port ib1, configured IPoverIB and I'm able to ping between the
>> infiniband hosts. I have also an OpenSM Subnet Manager running on each
>> When starting the servers on these hosts they complain about that they
>> cannot reach the other pvfs2-servers and I should check if my
>> on that particular host is running.
>> I suppose, the connection request is sent on ib0 instead of ib1.
>> My question is, if there are any parameters to tell the server to use port
>> 1 instead of port 0
>> Despite of all, everything runs very fine and we have very nice data
>> transfers (the 1-port 5 Nodes are writing with 415 MB per second).
>> I would appreciate any help.
>> Greeting from Salzburg/Austria/Europe
>> Vlad Popa
>> University of Salzburg
>> 5020 Salzburg
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
More information about the Pvfs2-users