[Pvfs2-developers] pvfs2-cp profile

[Scott Atchley]

On Jan 11, 2007, at 11:31 AM, Sam Lang wrote:

> On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:
>
>> Hi all,
>>
>> I used MPE to measure send, sendunexpected and recv on the client  
>> during pvfs2-cp (from the client to the server) of a (just less  
>> than) 256 MB file. This is using older 2.4 GHz Xeons with MX-2G (2  
>> Gb/s line rate). I am using FlowBufferSizeBytes of 1 MB. I am  
>> using the default FlowBuffersPerFlow. The server did not generate  
>> a log and I am looking into why.
>
> Hi Scott,
>
> Ignore yesterday's email, I was thinking that the events included  
> both the client and server, but you're only using MPE on the client.

No problem. I still do not know why I cannot get a log from the server.

>> This first image shows the entire copy of the file from client to  
>> server. Recv are blue, send are red and sendunexpected are orange.  
>> There 25 transfer of 10 MB each and the 26th transfer is ~5.5 MB.  
>> Note that there are distinct gaps between transfers in the range  
>> of 17-18 msecs.
>>
>> <pvfs2-cp-3.png>
>
> As I said, I was confused yesterday.  These gaps are entirely on  
> the client.  The end of one blue column marks the receive of the  
> write ack completing, and the start of the next column marks the  
> beginning of the next request.  The only thing that pvfs2-cp is  
> doing in the meantime, is reading the next 10MB of the local file.   
> I guess you're getting about 55 MB/s from that local read.

That sounds reasonable (these are older, slower machines). I would  
have assumed that pvfs2-cp would have two buffers so that it can read  
into one while sending the other.

>> The second image shows one of the 25 bulk IO transfers. There is a  
>> 11 msec delay from the last send completing to the recv completing  
>> (the large blue box). On the very far left is the initial  
>> sendunexpected that starts this operation.
>>
>> <pvfs2-cp-2.png>
>
> So this 11 msec delay could be caused by writing to disk, but if  
> you don't have data sync enabled (which I don't think you do), then  
> 11 msec for 1 MB is about 90MB/s to write data to buffer cache and  
> return the write ack response.  We might be delayed here by having  
> to wait for flow buffers being free'd up from one of the first 8  
> 1MB messages.  Could you increase the FlowBuffersPerFlow option in  
> the fs.conf to 16 or 20 and see what happens?
>
> -sam

I will try this.

I know that the default for FlowBuffersPerFlow is 8 per the webpage,  
but I count 10 per flow in the log. I also find that when using the - 
b flag, that it performs best with multiples of 10 buffers.

Scott