[Pvfs2-developers] pvfs2-cp profile

[Sam Lang]

On Jan 11, 2007, at 2:28 PM, Scott Atchley wrote:

> On Jan 11, 2007, at 2:47 PM, Sam Lang wrote:
>
>>>> So this 11 msec delay could be caused by writing to disk, but if  
>>>> you don't have data sync enabled (which I don't think you do),  
>>>> then 11 msec for 1 MB is about 90MB/s to write data to buffer  
>>>> cache and return the write ack response.  We might be delayed  
>>>> here by having to wait for flow buffers being free'd up from one  
>>>> of the first 8 1MB messages.  Could you increase the  
>>>> FlowBuffersPerFlow option in the fs.conf to 16 or 20 and see  
>>>> what happens?
>>>>
>>>> -sam
>>>
>>> I will try this.
>>>
>>> I know that the default for FlowBuffersPerFlow is 8 per the  
>>> webpage, but I count 10 per flow in the log. I also find that  
>>> when using the -b flag, that it performs best with multiples of  
>>> 10 buffers.
>>
>> You're doing 10MB requests with 1MB flow buffers.  I think the 10  
>> flows per request are from that, so the number of flow buffers is  
>> still 8, it just has to reuse 2 of them.
>>
>> -b specifies the request size doesn't it?  Using multiples of 10  
>> for that value improves performance?  I really have no idea, but  
>> would have guessed that multiples of the flow buffer size (in your  
>> case 1MB) would have been better.
>>
>> -sam
>
> With FlowBuffersPerFlow set to 20, I am roughly the same gaps as  
> before (~11 msec between last send completion and the recv  
> completion and ~18 msec between transfers).
>

Hmm..yeah changing that value never seems to do anything.  <shrug>

As Julian points out we should probably try to debug this delay on  
the server better without getting the disk involved.  Unfortunately,  
we don't have TAS in place in our current release.  You might try  
putting the server storage on tmpfs, Pete did this a month or so ago  
and was getting repeatable results.  He was seeing context switches  
costing 10s of microseconds, so that doesn't explain an 11 msec lag.

http://www.beowulf-underground.org/pipermail/pvfs2-developers/2006- 
December/002972.html

-sam

> Scott
>