[Pvfs2-developers] server flow post

[Scott Atchley]

On Jan 11, 2007, at 5:43 PM, Sam Lang wrote:

> I could post the flow first before posting the ack, which will  
> remove the race condition guaranteed, but it might also be slower  
> (which takes longer, buffering the expected message and copying to  
> the flow buffers or allocating the flow buffers first)?
>
> -sam

Sam,

I am not sure what you are asking here.

In TCP, the OS will receive and buffer the data. There is always a  
copy regardless if you pre-post the receive or not. Are you asking  
which is faster between memory copy and network transfer? If so, I  
would think that the memory copy is always faster. Given that, the  
current strategy (ack, then post the flow) makes the most sense.

In IB, I believe that A cannot write/put a large message to B until B  
has allocated memory and sent the memory address to A. This is why  
bmi_ib needs the RTS and CTS messages.

MX does this internally. When A posts a large send, MX sends a  
"scout" message, which is equivalent to the RTS, to B that includes  
the matching info and length. If B has posted a receive, than B  
replies with an ack and A can start sending data. If B has not posted  
a receive, then the scout message goes into the unexpected queue.  
When B does post a matching receive, it then has to scan the  
unexpected queue to see if it has already arrived. If so, it matches  
and sends an ack to start the data transfer.

By pre-posting the receives, we eliminate the scanning a potentially  
very long unexpected queue (I am thinking of the case of a storage  
server handling 10s or 100s of clients).

If you pre-post the receives, then in the IB case you could send all  
of that data in the ack to the initial sendunexpected and potentially  
eliminate the RTS and CTS messages as well.

Pete, I could possible be smoking something and this is not possible  
in IB at all. Any thoughts?

Sam, it may be that I am trying to optimize something that will not  
provide much benefit at all. Can you send a patch that simply posts  
the flow before the ack. I can test it on MX-10G and see if it  
impacts performance at all. If not, leave things as they are.

Scott