[Pvfs2-developers] pvfs2-cp profile

[Sam Lang]

That last patch I sent to the list was messed up.  This one should  
apply cleanly.

-sam

-------------- next part --------------
A non-text attachment was scrubbed...
Name: async-pvfs2-cp.patch
Type: application/octet-stream
Size: 7503 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20070112/d9902dc6/async-pvfs2-cp-0001.obj
-------------- next part --------------


On Jan 12, 2007, at 5:20 PM, Sam Lang wrote:

>
> On Jan 11, 2007, at 1:14 PM, Scott Atchley wrote:
>
>> On Jan 11, 2007, at 11:31 AM, Sam Lang wrote:
>>
>>> On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:
>>>
>>>> Hi all,
>>>>
>>>> I used MPE to measure send, sendunexpected and recv on the  
>>>> client during pvfs2-cp (from the client to the server) of a  
>>>> (just less than) 256 MB file. This is using older 2.4 GHz Xeons  
>>>> with MX-2G (2 Gb/s line rate). I am using FlowBufferSizeBytes of  
>>>> 1 MB. I am using the default FlowBuffersPerFlow. The server did  
>>>> not generate a log and I am looking into why.
>>>
>>> Hi Scott,
>>>
>>> Ignore yesterday's email, I was thinking that the events included  
>>> both the client and server, but you're only using MPE on the client.
>>
>> No problem. I still do not know why I cannot get a log from the  
>> server.
>>
>>>> This first image shows the entire copy of the file from client  
>>>> to server. Recv are blue, send are red and sendunexpected are  
>>>> orange. There 25 transfer of 10 MB each and the 26th transfer is  
>>>> ~5.5 MB. Note that there are distinct gaps between transfers in  
>>>> the range of 17-18 msecs.
>>>>
>>>> <pvfs2-cp-3.png>
>>>
>>> As I said, I was confused yesterday.  These gaps are entirely on  
>>> the client.  The end of one blue column marks the receive of the  
>>> write ack completing, and the start of the next column marks the  
>>> beginning of the next request.  The only thing that pvfs2-cp is  
>>> doing in the meantime, is reading the next 10MB of the local  
>>> file.  I guess you're getting about 55 MB/s from that local read.
>>
>> That sounds reasonable (these are older, slower machines). I would  
>> have assumed that pvfs2-cp would have two buffers so that it can  
>> read into one while sending the other.
>
> Hi Scott,
>
> Attached patch modifies pvfs2-cp to use two buffers.  This should  
> remove the 20 msec gaps you're seeing between requests.  Initial  
> tests I ran look like they also make the timings with pvfs2-cp more  
> consistent between runs.
>
> You'll need to run ./config.status after patching.
>
> -sam
>
> <async-pvfs2-cp.patch>
>
>
>>
>>>> The second image shows one of the 25 bulk IO transfers. There is  
>>>> a 11 msec delay from the last send completing to the recv  
>>>> completing (the large blue box). On the very far left is the  
>>>> initial sendunexpected that starts this operation.
>>>>
>>>> <pvfs2-cp-2.png>
>>>
>>> So this 11 msec delay could be caused by writing to disk, but if  
>>> you don't have data sync enabled (which I don't think you do),  
>>> then 11 msec for 1 MB is about 90MB/s to write data to buffer  
>>> cache and return the write ack response.  We might be delayed  
>>> here by having to wait for flow buffers being free'd up from one  
>>> of the first 8 1MB messages.  Could you increase the  
>>> FlowBuffersPerFlow option in the fs.conf to 16 or 20 and see what  
>>> happens?
>>>
>>> -sam
>>
>> I will try this.
>>
>> I know that the default for FlowBuffersPerFlow is 8 per the  
>> webpage, but I count 10 per flow in the log. I also find that when  
>> using the -b flag, that it performs best with multiples of 10  
>> buffers.
>>
>> Scott
>>
>
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers