[Pvfs2-users] Pvfs2 and multi-homing with Infiniband ports (IPoIB)

[Mark Van De Vyver]

Hi,
I'd appreciate the advice or insight anyone could provide on a pvfs2
configuration change I'm contemplating.

Specifically, I'd like to switch from using my current (congested)
Ethernet connections to using some Infiniband cards I have.
I have 'googled' the pvfs2 email list and one question about
multihoming didn't seem to have any result/thread.  I didn't see
anything in the pvfs/doc/ folder, unless I missed it.

I'd appreciate any comments or suggestions about the options I should
set in the configuration set out below.

The setup:
--------------
I have three IO servers each with a 2-port Infiniband card.  All ports
are connected to each other (no Infiniband switch).  Using IPoIB I can
ping all IB ports from all machines, and running: "osmtest -f a",
indicates pass.
Each PVFS IO server server is also a PVFS client.  One machine is the
metadata server.

The performance questions:
---------------------------------------
1) Is multi-homing in this setup likely to give benefits, or just headaches?
(a.k.a Is this a sane thing to contemplate?)

The configuration questions:
----------------------------------------
2) In pvfs2-conf.fs: should BMIModules refer to both tcp and ib to use IPoIB?
3) How should each IO server be "told" to listen on the two IP
addresses assigned to the 2 IB ports? Specifically:
  a) Does HostID in the server config file need to be changed?  If so,
  b) Does the Alias in the pvfs2-conf.fs get changed?
  c) Should TrustedPorts and TrustedNetwork values be set?
  d) Should the port set on each IPoIB/IP address be different or the
same (I recall seeing an email about this but can't seem to find it
again...)
4) Given I will be using IPoIB I assume I still set tcp as the PVFS
protocol nominated in the HostID and Alias?

The configuration files are below.

Regards
Mark

[pvfs2-server.conf-frontend]
StorageSpace /state/partition1/pvfs2-storage-space
HostID "tcp://frontend:3334"
LogFile /var/log/pvfs2

[pvfs2-server.conf-compute-0-1]
StorageSpace /state/partition1/pvfs2-storage-space
HostID "tcp://pvfs2-compute-0-1:3334"
LogFile /var/log/pvfs2


[pvfs2-server.conf-compute-0-0]
StorageSpace /state/partition1/pvfs2-storage-space
HostID "tcp://pvfs2-compute-0-0:3334"
LogFile /var/log/pvfs2

[pvfs2-conf.fs]
<Defaults>
	UnexpectedRequests 50
	LogFile /var/log/pvfs2
	EventLogging none
	LogStamp datetime
	BMIModules bmi_tcp
	FlowModules flowproto_multiqueue
	PerfUpdateInterval 1000
	ServerJobBMITimeoutSecs 30
	ServerJobFlowTimeoutSecs 30
	ClientJobBMITimeoutSecs 300
	ClientJobFlowTimeoutSecs 300
	ClientRetryLimit 5
	ClientRetryDelayMilliSecs 2000
</Defaults>

<Aliases>
	Alias frontend tcp://frontend:3334
	Alias pvfs2-compute-0-0 tcp://pvfs2-compute-0-0:3334
	Alias pvfs2-compute-0-1 tcp://pvfs2-compute-0-1:3334
</Aliases>

<Filesystem>
	Name pvfs2-fs
	ID 1054160401
	RootHandle 1048576
	<MetaHandleRanges>
		Range frontend 4-1431655767
	</MetaHandleRanges>
	<DataHandleRanges>
		Range pvfs2-compute-0-0 1431655768-2863311531
		Range pvfs2-compute-0-1 2863311532-4294967295
                Range frontend 4294967295-6294967295
	</DataHandleRanges>
	<StorageHints>
		TroveSyncMeta yes
		TroveSyncData no
	</StorageHints>
</Filesystem>