[Pvfs2-users] pvfs performance over IB

[Scott Atchley]

Hi Rene,

First, what is your FlowBufferSize set to in your fs.conf file?

Second, I found that by playing with pvfs2-cp's switches, I could  
improve performance some. I typically use a FlowBufferSize of 4 MB.  
Pete would have to comment on what he uses with IB. Given that, I set  
-b $((N*1024*1024)) where N is 16, 20, 32, 40, 64, 100, and 128. You  
may also want to try using -s $((M*$FlowBufferSize)) where M is 1, 2,  
4, etc.

Scott

On Sep 6, 2007, at 10:52 AM, Salmon, Rene wrote:

> Hi,
>
>
> Searched through the list archives and it found some posts that  
> show IB
> performance around 400MB/s.  Also found the post that states in  
> order to
> get best performance over IB to configure with "--without-bmi-tcp".  I
> need tpc as well as IB so I this is not an option for me.
>
> This is a simple setup. Single pvfs server with a single pvfs client.
> DDR IB connect the two as well as 1Gb ethernet. On the server there is
> an xfs filesystem that can write at about 500MB/sec that is what the
> pvfs server will use.
>
>
> I would expect that my bottle neck would be the xfs filesystem at
> 500MB/sec.  When I run pvfs-cp on the client I only get around 260
> MB/sec bandwidth:
>
> hpcxe005(salmr0)37:/usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp
> -t /test/pvfs2/testfile1 /test/pvfs2/testfile5
> Wrote 655360000 bytes in 2.355496 seconds. 265.336905 MB/seconds
>
> Any ideas on how to get this to perform a bit better?
>
> client mount command:
> ---------------------
> mount -t pvfs2 ib://hpcxe007:3335/pvfs2-fs /test/pvfs2
>
>
> server pvfs2-fs.conf:
> ----------------------
> <Aliases>
>         Alias hpcxe007 ib://hpcxe007:3335,tcp://hpcxe007:3334
> </Aliases>
>
>
> pvfs2-server.conf-hpcxe007:
> ---------------------------
>
> StorageSpace /pvfs2-storage-space
> HostID "ib://hpcxe007:3335,tcp://hpcxe007:3334"
>
>
> Thanks
> Rene
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users