[Pvfs2-developers] BMI questions

[Scott Atchley]

On Nov 30, 2006, at 4:31 PM, Sam Lang wrote:

> Right now all our operations (or transactions, as you call them)  
> start with an unexpected message from the client, and end with an  
> expected message from the server.  I don't know if that's a design  
> requirement of BMI though, or just an artifact of how we use it in  
> PVFS.  I _think_ the BMI interfaces were meant to allow expected  
> messages in either direction in any order, and its left up to the  
> upper layers to make sure they get posted right, but again, I would  
> have to defer to one of the BMI sages.

Hmmm. I assumed that for any operation, that there would be a back  
and forth between client and server ending with a expected send from  
server to the client:

Client          Server
    |     unex      |
    |-------------->|
    |               |
    |      ex       |
    |<--------------|
    |               |
    |      ex       |
    |-------------->|
    |               |
    |      ex       |
    |<--------------|
    |               |

with a minimum of unexpected client to server followed by an expected  
from server to client. If this is the case I might be able to do a  
simple flow control on the client using a reference count (increment  
on send to server S and decrement on receive from S).

Are you saying that a single operation may not ping pong back and  
forth but have multiple expected sends in a single direction?

Client          Server
    |     unex      |
    |-------------->|
    |               |
    |      ex       |
    |<--------------|
    |               |
    |      ex       |
    |-------------->|
    |               |
    |      ex       |
    |-------------->|
    |               |
    |      ex       |
    |-------------->|
    |               |
    |      ex       |
    |<--------------|
    |               |

If so, would each of the receives (and matching sends) use different  
tags? Also, this case presents a resource starvation risk. Since the  
BMI method does not know about the entire operation (how many sends/ 
receives), it is possible that it could start the operation but not  
be able to get the additional resources for the subsequent sends/ 
receives to complete it.

> Are you able to do some kind of pre-posting if you know there's  
> always an expected coming back?
>
> -sam

I assumed that BMI always posted a receive for an expected incoming  
send? Does it not? I would hope that BMI or a higher layer would pre- 
post the receive before calling the send function. If not, let me know.

Scott