[Pvfs2-developers] BMI questions

[Sam Lang]

On Nov 30, 2006, at 1:16 PM, Scott Atchley wrote:

> On Nov 30, 2006, at 12:13 PM, Sam Lang wrote:
>
>> On Nov 30, 2006, at 9:27 AM, Scott Atchley wrote:
>>
>>> Hi all,
>>>
>>> No one replied the the original post. The one I am most curious  
>>> about is (3). Is FlowBufferSizeBytes available to the BMI  
>>> implementations?
>>
>> Hi Scott,
>>
>> Sorry for not responding.  I've included responses to your  
>> questions inline.
>
> No problem. I have been side-tracked on a couple things as well.
>
>>> I am thinking about how to handle BMI_mx_memalloc(). I might be  
>>> able to assist the reg cache in MX if I do things a certain way.
>>
>> I mention this below, on the server we always call BMI_memalloc to  
>> allocate buffers and post them to BMI_post_send/BMI_post_recv.  On  
>> the client that's not always the case, the user buffer is usually  
>> passed directly to BMI_post_send/BMI_post_recv.  Would it help you  
>> to know in advance of the post (or all posts) what the maximum  
>> size of a buffer for BMI_post_send/BMI_post_recv is going to be?   
>> We might be able to add a BMI info option that other methods could  
>> ignore.  We do limit it, but its all above the BMI interface that  
>> this is done.
>
> Knowing the max size in advance helps iff the buffers are allocated  
> using BMI_memalloc(). If not, it does not matter.

So an extra BMI info option would help on the server?

>
>>>> 1. Is there an upper bound on how many transfer operations  
>>>> between a pair of hosts at any one time (i.e. between a client  
>>>> and host)?
>>
>> There is a max of unexpected requests (UnexpectedRequests option  
>> in the config file, defaults to 50).  This limits the number of  
>> unexpected messages that can be handled at once by the server, but  
>> doesn't limit the number of IO operations (large expected messages).
>
> I read this as the server will only service up to 50 unexpected  
> request, but that the client (or clients) can send more. Is that  
> correct?

Yes that's correct.  There's no limit on unexpected messages a client  
can send to one particular, or any number of servers.

>
>>>> 2. If there is a limit in the above and it is greater than 1, is  
>>>> that value the same for small (unexpected and some expected)  
>>>> messages and large (expected) messages? Or are large messages  
>>>> restricted to a lower value?
>>
>> So I don't think there's a limit at present for expected  
>> messages.  The unexpected limit is the one I mentioned above.
>
> Ok.
>
>>>> 3. In the thread entitled "libpvfs2 usage", SamL mentioned the  
>>>> tunable called FlowBufferSizeBytes and that it is set to 256KB  
>>>> by default. Is this value the upper limit on large messages  
>>>> (i.e. 8KB < large messages <= FlowBufferSizeBytes assuming 8 KB  
>>>> is the maximum unexpected size)?
>>
>> Yes, on the server BMI_post_recv and BMI_post_send won't get  
>> called with values larger than that.  On the client, there's no  
>> strict limit on the size of the messages posted.
>
> Below, you seem to imply that client messages are broken into  
> FlowBufferSizeBytes chunks. That would make sense if a server never  
> allocated more than FlowBufferSizeBytes.
>
>>>> Would the BMI method have any knowledge of what this value is  
>>>> (i.e. can it use FlowBufferSizeBytes to allocate large buffers)?
>>
>> On the server it does.  BMI_memalloc is called from the flow with  
>> exactly the 256K (FlowBufferSizeBytes) value.  The buffer returned  
>> is the one passed to BMI_post_recv or BMI_post_send (although, the  
>> size of the request may be less than 256K).  On the client, the  
>> user buffer is used for calls to BMI_post_recv and BMI_post_send,  
>> so you won't see BMI_memalloc calls first.  The client does break  
>> up the entire client request into chunks of FlowBufferSizeBytes  
>> for the BMI posts, but you won't know what that value is in  
>> advance of the post operations.
>>
>> -sam
>
> Ok. Given that the client does not use BMI_memalloc(), I will not  
> try to do anything for BMI_memalloc() for now other than call malloc 
> (). Do not worry about exposing the FlowBufferSizeBytes value for now.

Ok.  Let us know if adding a BMI_set_info option would help.

-sam

>
> Thanks,
>
> Scott
>
>