[Pvfs2-developers] pvfs2-cp profile

[Sam Lang]

On Jan 11, 2007, at 1:14 PM, Scott Atchley wrote:

> On Jan 11, 2007, at 11:31 AM, Sam Lang wrote:
>
>> On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:
>>
>>> Hi all,
>>>
>>> I used MPE to measure send, sendunexpected and recv on the client  
>>> during pvfs2-cp (from the client to the server) of a (just less  
>>> than) 256 MB file. This is using older 2.4 GHz Xeons with MX-2G  
>>> (2 Gb/s line rate). I am using FlowBufferSizeBytes of 1 MB. I am  
>>> using the default FlowBuffersPerFlow. The server did not generate  
>>> a log and I am looking into why.
>>
>> Hi Scott,
>>
>> Ignore yesterday's email, I was thinking that the events included  
>> both the client and server, but you're only using MPE on the client.
>
> No problem. I still do not know why I cannot get a log from the  
> server.
>
>>> This first image shows the entire copy of the file from client to  
>>> server. Recv are blue, send are red and sendunexpected are  
>>> orange. There 25 transfer of 10 MB each and the 26th transfer is  
>>> ~5.5 MB. Note that there are distinct gaps between transfers in  
>>> the range of 17-18 msecs.
>>>
>>> <pvfs2-cp-3.png>
>>
>> As I said, I was confused yesterday.  These gaps are entirely on  
>> the client.  The end of one blue column marks the receive of the  
>> write ack completing, and the start of the next column marks the  
>> beginning of the next request.  The only thing that pvfs2-cp is  
>> doing in the meantime, is reading the next 10MB of the local  
>> file.  I guess you're getting about 55 MB/s from that local read.
>
> That sounds reasonable (these are older, slower machines). I would  
> have assumed that pvfs2-cp would have two buffers so that it can  
> read into one while sending the other.
>
>>> The second image shows one of the 25 bulk IO transfers. There is  
>>> a 11 msec delay from the last send completing to the recv  
>>> completing (the large blue box). On the very far left is the  
>>> initial sendunexpected that starts this operation.
>>>
>>> <pvfs2-cp-2.png>
>>
>> So this 11 msec delay could be caused by writing to disk, but if  
>> you don't have data sync enabled (which I don't think you do),  
>> then 11 msec for 1 MB is about 90MB/s to write data to buffer  
>> cache and return the write ack response.  We might be delayed here  
>> by having to wait for flow buffers being free'd up from one of the  
>> first 8 1MB messages.  Could you increase the FlowBuffersPerFlow  
>> option in the fs.conf to 16 or 20 and see what happens?
>>
>> -sam
>
> I will try this.
>
> I know that the default for FlowBuffersPerFlow is 8 per the  
> webpage, but I count 10 per flow in the log. I also find that when  
> using the -b flag, that it performs best with multiples of 10 buffers.

You're doing 10MB requests with 1MB flow buffers.  I think the 10  
flows per request are from that, so the number of flow buffers is  
still 8, it just has to reuse 2 of them.

-b specifies the request size doesn't it?  Using multiples of 10 for  
that value improves performance?  I really have no idea, but would  
have guessed that multiples of the flow buffer size (in your case  
1MB) would have been better.

-sam
>
> Scott
>