[Pvfs2-developers] server flow post

[Sam Lang]

Scott showed in his MX tests that a number of the expected messages  
were getting received before BMI operations had actually been posted  
for them.  He sees this on both client and server, but I'm going to  
focus on the server (and writes) for now.  I think he's seeing this  
in part due to a race condition in the IO state machine on the  
server.  We send back the positive ack, wait for it to complete, and  
then post the flow, which in turn posts the first expected message  
for data.  Posting the flow does the allocation of the first flow  
buffer, which in some cases may take as long as the client to send  
the first expected message (esp. with larger flow buffers).  The race  
occurs if allocating the buffer takes longer than sending the first  
message.

The attached patch tries to address this issue, by posting the ack  
and the flow together in a single state, and then waiting for both to  
complete before sending the write ack.  Its a bit more complicated  
than the current IO state machine, because we have different states  
for reads and writes now, but I tried to move most of the common code  
into separate functions.

I'm not sure this will actually make a difference for expected  
messages though.  With tcp, I'm seeing the bmi call that posts the  
response complete immediately (returning 1), so its essentially the  
same as before.  Do the other bmi methods complete immediately for  
small messages as well?  I could post the flow first before posting  
the ack, which will remove the race condition guaranteed, but it  
might also be slower (which takes longer, buffering the expected  
message and copying to the flow buffers or allocating the flow  
buffers first)?

In any case, Scott it might be worth it to try this patch and see if  
we're able to post operations for expected messages before they're  
received.

-sam


-------------- next part --------------
A non-text attachment was scrubbed...
Name: io-flow-post.patch
Type: application/octet-stream
Size: 16858 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20070111/7a0aff33/io-flow-post.obj