[Pvfs2-developers] pvfs2-cp profile

[Sam Lang]

On Jan 11, 2007, at 1:14 PM, Scott Atchley wrote:

> On Jan 11, 2007, at 11:31 AM, Sam Lang wrote:
>
>> On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:
>>
>>> Hi all,
>>>
>>> I used MPE to measure send, sendunexpected and recv on the client  
>>> during pvfs2-cp (from the client to the server) of a (just less  
>>> than) 256 MB file. This is using older 2.4 GHz Xeons with MX-2G  
>>> (2 Gb/s line rate). I am using FlowBufferSizeBytes of 1 MB. I am  
>>> using the default FlowBuffersPerFlow. The server did not generate  
>>> a log and I am looking into why.
>>
>> Hi Scott,
>>
>> Ignore yesterday's email, I was thinking that the events included  
>> both the client and server, but you're only using MPE on the client.
>
> No problem. I still do not know why I cannot get a log from the  
> server.
>
>>> This first image shows the entire copy of the file from client to  
>>> server. Recv are blue, send are red and sendunexpected are  
>>> orange. There 25 transfer of 10 MB each and the 26th transfer is  
>>> ~5.5 MB. Note that there are distinct gaps between transfers in  
>>> the range of 17-18 msecs.
>>>
>>> <pvfs2-cp-3.png>
>>
>> As I said, I was confused yesterday.  These gaps are entirely on  
>> the client.  The end of one blue column marks the receive of the  
>> write ack completing, and the start of the next column marks the  
>> beginning of the next request.  The only thing that pvfs2-cp is  
>> doing in the meantime, is reading the next 10MB of the local  
>> file.  I guess you're getting about 55 MB/s from that local read.
>
> That sounds reasonable (these are older, slower machines). I would  
> have assumed that pvfs2-cp would have two buffers so that it can  
> read into one while sending the other.

Hi Scott,

Attached patch modifies pvfs2-cp to use two buffers.  This should  
remove the 20 msec gaps you're seeing between requests.  Initial  
tests I ran look like they also make the timings with pvfs2-cp more  
consistent between runs.

You'll need to run ./config.status after patching.

-sam

-------------- next part --------------
A non-text attachment was scrubbed...
Name: async-pvfs2-cp.patch
Type: application/octet-stream
Size: 7598 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20070112/2d5ce587/async-pvfs2-cp.obj
-------------- next part --------------


>
>>> The second image shows one of the 25 bulk IO transfers. There is  
>>> a 11 msec delay from the last send completing to the recv  
>>> completing (the large blue box). On the very far left is the  
>>> initial sendunexpected that starts this operation.
>>>
>>> <pvfs2-cp-2.png>
>>
>> So this 11 msec delay could be caused by writing to disk, but if  
>> you don't have data sync enabled (which I don't think you do),  
>> then 11 msec for 1 MB is about 90MB/s to write data to buffer  
>> cache and return the write ack response.  We might be delayed here  
>> by having to wait for flow buffers being free'd up from one of the  
>> first 8 1MB messages.  Could you increase the FlowBuffersPerFlow  
>> option in the fs.conf to 16 or 20 and see what happens?
>>
>> -sam
>
> I will try this.
>
> I know that the default for FlowBuffersPerFlow is 8 per the  
> webpage, but I count 10 per flow in the log. I also find that when  
> using the -b flag, that it performs best with multiples of 10 buffers.
>
> Scott
>