[Pvfs2-users] Re: PVFS2 over IB RDMA

[Chen, Helen Y]

I am having trouble installing pvfs2 2.8.0 to use IB RDMA under centos  5.3 with 2.6.18-128, and hope that someone on the mailing list can offer some help.  

1. The build:  I ran ./configure --with-openib=/usr --without-bmi-tcp --with-kernel=/usr/src/kernels/2.6.18-128.el5-x86_64.

2. I  configured the meta and I/O server on a single node (dn2).  And here is the content of my  /etc/pvfs2-fs.conf:

<Defaults>
        UnexpectedRequests 50
        EventLogging none
        EnableTracing no.
        LogStamp datetime
        BMIModules bmi_ib
        FlowModules flowproto_multiqueue
        PerfUpdateInterval 1000
        ServerJobBMITimeoutSecs 30
        ServerJobFlowTimeoutSecs 30
        ClientJobBMITimeoutSecs 300
        ClientJobFlowTimeoutSecs 300
        ClientRetryLimit 5
        ClientRetryDelayMilliSecs 2000
        PrecreateBatchSize 512
        PrecreateLowThreshold 256

        StorageSpace /pvfs2-storage-space
        LogFile /tmp/pvfs2-server.log
</Defaults>

<Aliases>
        Alias dn2-ib ib://dn2-ib:3335
</Aliases>

<Filesystem>
        Name pvfs2-fs
        ID 695113894
        RootHandle 1048576
        FileStuffing yes
        <MetaHandleRanges>
                Range dn2-ib 3-4611686018427387904
        </MetaHandleRanges>
        <DataHandleRanges>
                Range dn2-ib 4611686018427387905-9223372036854775806
        </DataHandleRanges>
        <StorageHints>
                TroveSyncMeta yes
                TroveSyncData no
                TroveMethod alt-aio
        </StorageHints>
</Filesystem>

3. I built an xfs filesystem on /dev/md0 and mounted it on /pvfs2.

4. I then started the server with "/usr/local/sbin/pvfs2-server /etc/pvfs2-fs.conf -f " for the 1st time and without "-f" subsequent times.

4. On a second node (dn3, the pvfs client) I created /etc/pvfs2tab with the following line:

ib://dn2-ib:3335/pvfs2 /mnt/pvfs2 pvfs2 defaults,noauto 0 0

5. Pvfs2-ping -m /mnt/pvfs2 failed.  Unfortunately I did'nt capture the erroe message in time before the nodes were grabbed by my colleague. I remembered vaguely that it complained about the exported pvfs2 filesystem not initialized. 

6. I verified that the IB RDMA network worked using rping.

Please let me know if I have done something wrong and whether you need more info to troubleshoot.  Your help will be appreciated.

hycsw at sandia.gov