[Pvfs2-developers] BMI questions

[Sam Lang]

On Nov 30, 2006, at 9:27 AM, Scott Atchley wrote:

> Hi all,
>
> No one replied the the original post. The one I am most curious  
> about is (3). Is FlowBufferSizeBytes available to the BMI  
> implementations?
>

Hi Scott,

Sorry for not responding.  I've included responses to your questions  
inline.

> I am thinking about how to handle BMI_mx_memalloc(). I might be  
> able to assist the reg cache in MX if I do things a certain way.
>

I mention this below, on the server we always call BMI_memalloc to  
allocate buffers and post them to BMI_post_send/BMI_post_recv.  On  
the client that's not always the case, the user buffer is usually  
passed directly to BMI_post_send/BMI_post_recv.  Would it help you to  
know in advance of the post (or all posts) what the maximum size of a  
buffer for BMI_post_send/BMI_post_recv is going to be?  We might be  
able to add a BMI info option that other methods could ignore.  We do  
limit it, but its all above the BMI interface that this is done.

> Scott
>
> On Oct 25, 2006, at 1:53 PM, Scott Atchley wrote:
>
>> Hi all,
>>
>> A few quick questions:
>>
>> 1. Is there an upper bound on how many transfer operations between  
>> a pair of hosts at any one time (i.e. between a client and host)?

There is a max of unexpected requests (UnexpectedRequests option in  
the config file, defaults to 50).  This limits the number of  
unexpected messages that can be handled at once by the server, but  
doesn't limit the number of IO operations (large expected messages).

>>
>> 2. If there is a limit in the above and it is greater than 1, is  
>> that value the same for small (unexpected and some expected)  
>> messages and large (expected) messages? Or are large messages  
>> restricted to a lower value?

So I don't think there's a limit at present for expected messages.   
The unexpected limit is the one I mentioned above.

>> 3. In the thread entitled "libpvfs2 usage", SamL mentioned the  
>> tunable called FlowBufferSizeBytes and that it is set to 256KB by  
>> default. Is this value the upper limit on large messages (i.e. 8KB  
>> < large messages <= FlowBufferSizeBytes assuming 8 KB is the  
>> maximum unexpected size)?

Yes, on the server BMI_post_recv and BMI_post_send won't get called  
with values larger than that.  On the client, there's no strict limit  
on the size of the messages posted.

>> Would the BMI method have any knowledge of what this value is  
>> (i.e. can it use FlowBufferSizeBytes to allocate large buffers)?

On the server it does.  BMI_memalloc is called from the flow with  
exactly the 256K (FlowBufferSizeBytes) value.  The buffer returned is  
the one passed to BMI_post_recv or BMI_post_send (although, the size  
of the request may be less than 256K).  On the client, the user  
buffer is used for calls to BMI_post_recv and BMI_post_send, so you  
won't see BMI_memalloc calls first.  The client does break up the  
entire client request into chunks of FlowBufferSizeBytes for the BMI  
posts, but you won't know what that value is in advance of the post  
operations.

-sam

>>
>> Thanks,
>>
>> Scott
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>