[Pvfs2-users] Read performance is dismal from pvfs volumes

[Sam Lang]

On Jan 17, 2007, at 2:09 PM, Robert Latham wrote:

> On Wed, Jan 17, 2007 at 11:47:37AM -0800, Chris Halstead wrote:
>> Copy tests were done by simply timing 'cp' from the mounted PVFS  
>> volume to
>> local /dev/shm (to avoid filesystem latency).
>>
>> When I strace the pvfs server process, I see the server writing to  
>> the TCP
>> socket in chunks no larger than ~4k:
>>
>> writev(11, [{"\277\312\0\0\4\0\0\0\213^\0\0\0\0\0\0(\20\0\0\0\0\0 
>> \0", 24},
>> {"\274\v\0\0\2\0\0\0!\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0>"...,  
>> 4136}], 2) =
>> 4160
>
> What you are seeing there is that the cp(1) program uses the blocksize
> of the destination file system when deciding how big a buffer to use.
> About a year ago we got the coreutils maintainers to take into account
> the source and destination file systems, but that's only available in
> coreutils-6.0 and newer (so it might take a while longer to show up in
> distributions).
>
>> I've tried tuning kernel-level TCP settings and
>> TCPBufferSend/TCPBufferReceive in the pvfs config, but I never see  
>> a change
>> in the size of the writes to the TCP socket.  Is this normal?  Is  
>> there a
>> tuning parameter that I'm missing that will influence the size of the
>> writes?
>
> Two things to try: 1: you can upgrade your coreutils or 2: you can use
> pvfs2-cp, which will bypass the kernel VFS and use a 4 MB buffer size
> by default.

If you wanted to go through the VFS, you could also use dd and  
specify the blocksize you want.

-sam

>
> ==rob
>
> -- 
> Rob Latham
> Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
> Argonne National Lab, IL USA                 B29D F333 664A 4280 315B
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>