[Pvfs2-users] pvfs performance over IB

[Salmon, Rene]

Hi,


I got the latest pvfs from CVS and rebuilt both the server and the
client with the "--without-bmi-tcp" flag.  This took me from 260MB/sec
to about 320MB/sec with pvfs2-cp.  About a 60MB/sec boost.

adjusting the buffer size option to pvfs2-cp did help:

pvfs2-cp -b16777216 -t /tmp/testfile1 /test/pvfs2/foo12
Wrote 16777216000 bytes in 45.138288 seconds. 354.466257 MB/seconds

pvfs2-cp -b33554432 -t /tmp/testfile1 /test/pvfs2/foo13
Wrote 16777216000 bytes in 43.276829 seconds. 369.712855 MB/seconds

pvfs2-cp -b67108864 -t /tmp/testfile1 /test/pvfs2/foo14
Wrote 16777216000 bytes in 40.825193 seconds. 391.914866 MB/seconds

pvfs2-cp -b104857600 -t /tmp/testfile1 /test/pvfs2/foo17
Wrote 16777216000 bytes in 39.127303 seconds. 408.921615 MB/seconds

pvfs2-cp -b134217728 -t /tmp/testfile1 /test/pvfs2/foo15
Wrote 16777216000 bytes in 41.528605 seconds. 385.276606 MB/seconds


Best I can get is around 400MB/s for a one(client) to one (server) test
config.  This is really close to what my disk back end can handle.
Since there is only one I/O server using the -s option for striping did
not help any in this case.

Thanks
Rene



On Thu, 2007-09-06 at 19:16 +0100, Scott Atchley wrote:
> Hi Rene,
> 
> First, what is your FlowBufferSize set to in your fs.conf file?
> 
> Second, I found that by playing with pvfs2-cp's switches, I could 
> improve performance some. I typically use a FlowBufferSize of 4 MB. 
> Pete would have to comment on what he uses with IB. Given that, I set 
> -b $((N*1024*1024)) where N is 16, 20, 32, 40, 64, 100, and 128. You 
> may also want to try using -s $((M*$FlowBufferSize)) where M is 1, 2, 
> 4, etc.
> 
> Scott
>