[Pvfs2-developers] server flow post

[Sam Lang]

On Jan 12, 2007, at 12:32 PM, Scott Atchley wrote:

> On Jan 12, 2007, at 12:52 PM, Sam Lang wrote:
>
>> On Jan 12, 2007, at 9:03 AM, Pete Wyckoff wrote:
>>
>>> I think that all makes sense.  Agree that the need for preposting
>>> receives is to avoid big queues of waiting unexpected messages.
>>
>> Hi Scott,
>>
>> The attached patch posts the flow (and receives) before posting  
>> the send of the response ack.  I posted the response ack before  
>> the flow, because the first call to BMI_memalloc (with a request  
>> for 1MB in your case) happens in the flow post call, so that  
>> delays posting of the response ack.  I'm curious if this will  
>> actually give you better performance.
>>
>> I also fixed that assert failure you were getting.  Let me know if  
>> this works for you.
>>
>> Thanks,
>>
>> -sam
>>
>> <io-flow-post.patch>
>
> I applied the patch, recompiled and ran pvfs2-cp again. I see about  
> the same number of unexpected receives on the server as before:
>
It looks like there are more early expected messages than  
previously.  You were only seeing about 9 per tag before IIRC.

We don't post the next BMI read until the previous has finished on  
the server, so there's still the possibility of a race-condition,  
because the client could send the next message before the server  
actually posted the receive for it, but the time in the unexpected  
queue should be pretty short.

Do you see any performance differences with the patch?  Maybe  
increase the flow buffer sizes to something big (10MB or so) to see  
if allocation kills performance?

-sam

> [D 13:28:57.671374] PVFS2 Server version  
> 2.6.1pre1-2007-01-11-200834 starting.
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 8  match  
> 0xe000000100000008
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 9  match  
> 0xe000000100000009
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 10  match  
> 0xe00000010000000a
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 11  match  
> 0xe00000010000000b
> Early expected message  length 1048576  tag 12  match  
> 0xe00000010000000c
> Early expected message  length 1048576  tag 12  match  
> 0xe00000010000000c
> Early expected message  length 1048576  tag 12  match  
> 0xe00000010000000c
> Early expected message  length 1048576  tag 12  match  
> 0xe00000010000000c
> Early expected message  length 1048576  tag 12  match  
> 0xe00000010000000c
> Early expected message  length 1048576  tag 12  match  
> 0xe00000010000000c
> Early expected message  length 1048576  tag 12  match  
> 0xe00000010000000c
> Early expected message  length 1048576  tag 13  match  
> 0xe00000010000000d
> Early expected message  length 1048576  tag 13  match  
> 0xe00000010000000d
> Early expected message  length 1048576  tag 13  match  
> 0xe00000010000000d
> Early expected message  length 1048576  tag 13  match  
> 0xe00000010000000d
> Early expected message  length 538689  tag 13  match  
> 0xe00000010000000d
>
> Note that it starts at tag 8. Is the client sending the second set  
> of buffers before the server has posted them?
>
> Scott
>