[Pvfs2-developers] BMI questions

[Scott Atchley]

On Nov 30, 2006, at 12:13 PM, Sam Lang wrote:

> On Nov 30, 2006, at 9:27 AM, Scott Atchley wrote:
>
>> Hi all,
>>
>> No one replied the the original post. The one I am most curious  
>> about is (3). Is FlowBufferSizeBytes available to the BMI  
>> implementations?
>
> Hi Scott,
>
> Sorry for not responding.  I've included responses to your  
> questions inline.

No problem. I have been side-tracked on a couple things as well.

>> I am thinking about how to handle BMI_mx_memalloc(). I might be  
>> able to assist the reg cache in MX if I do things a certain way.
>
> I mention this below, on the server we always call BMI_memalloc to  
> allocate buffers and post them to BMI_post_send/BMI_post_recv.  On  
> the client that's not always the case, the user buffer is usually  
> passed directly to BMI_post_send/BMI_post_recv.  Would it help you  
> to know in advance of the post (or all posts) what the maximum size  
> of a buffer for BMI_post_send/BMI_post_recv is going to be?  We  
> might be able to add a BMI info option that other methods could  
> ignore.  We do limit it, but its all above the BMI interface that  
> this is done.

Knowing the max size in advance helps iff the buffers are allocated  
using BMI_memalloc(). If not, it does not matter.

>>> 1. Is there an upper bound on how many transfer operations  
>>> between a pair of hosts at any one time (i.e. between a client  
>>> and host)?
>
> There is a max of unexpected requests (UnexpectedRequests option in  
> the config file, defaults to 50).  This limits the number of  
> unexpected messages that can be handled at once by the server, but  
> doesn't limit the number of IO operations (large expected messages).

I read this as the server will only service up to 50 unexpected  
request, but that the client (or clients) can send more. Is that  
correct?

>>> 2. If there is a limit in the above and it is greater than 1, is  
>>> that value the same for small (unexpected and some expected)  
>>> messages and large (expected) messages? Or are large messages  
>>> restricted to a lower value?
>
> So I don't think there's a limit at present for expected messages.   
> The unexpected limit is the one I mentioned above.

Ok.

>>> 3. In the thread entitled "libpvfs2 usage", SamL mentioned the  
>>> tunable called FlowBufferSizeBytes and that it is set to 256KB by  
>>> default. Is this value the upper limit on large messages (i.e.  
>>> 8KB < large messages <= FlowBufferSizeBytes assuming 8 KB is the  
>>> maximum unexpected size)?
>
> Yes, on the server BMI_post_recv and BMI_post_send won't get called  
> with values larger than that.  On the client, there's no strict  
> limit on the size of the messages posted.

Below, you seem to imply that client messages are broken into  
FlowBufferSizeBytes chunks. That would make sense if a server never  
allocated more than FlowBufferSizeBytes.

>>> Would the BMI method have any knowledge of what this value is  
>>> (i.e. can it use FlowBufferSizeBytes to allocate large buffers)?
>
> On the server it does.  BMI_memalloc is called from the flow with  
> exactly the 256K (FlowBufferSizeBytes) value.  The buffer returned  
> is the one passed to BMI_post_recv or BMI_post_send (although, the  
> size of the request may be less than 256K).  On the client, the  
> user buffer is used for calls to BMI_post_recv and BMI_post_send,  
> so you won't see BMI_memalloc calls first.  The client does break  
> up the entire client request into chunks of FlowBufferSizeBytes for  
> the BMI posts, but you won't know what that value is in advance of  
> the post operations.
>
> -sam

Ok. Given that the client does not use BMI_memalloc(), I will not try  
to do anything for BMI_memalloc() for now other than call malloc().  
Do not worry about exposing the FlowBufferSizeBytes value for now.

Thanks,

Scott