[Pvfs2-developers] BMI_testunexpected and free

[Scott Atchley]

On Aug 21, 2006, at 5:58 PM, Pete Wyckoff wrote:

> vilayann at mcs.anl.gov wrote on Mon, 21 Aug 2006 14:32 -0500:
>> Sam and I raised this question for the following reason (motivated  
>> during the discussion of
>>  BMI-MX's design with Scott)
>> Suppose a BMI method wants to keep around a pre-allocated pool of
>> unexpected buffers, it seems wasteful to copy them into a user  
>> buffer and
>> then free it again.
>> So we were kind of wondering if a test_unexpected() returns one of  
>> those
>> preallocated buffers (and mark them internally as being used) and  
>> then
>> the caller can then call BMI_memfree() which will return it to the  
>> pool of
>> unexpected buffers (by marking them as free again).
>>
>> But if this does not appeal to you guys, I suspect the  
>> unexpected_free()
>> callback function would also be better than free()..
>
> I think doing what Phil suggested with BMI_unexpected_free() is the
> best way to go.  This is the same thing you're describing except it
> makes explicit the separate functions of:  1) freeing
> network-registered memory (BMI_memfree), and 2) acknowledging an
> unexpected message has been serviced (BMI_unexpected_free).
>
> You'll just have to go through the code and audit all users of
> testunexpected() to make sure they call your new function instead of
> normal free(); it'll have to be mandatory from now on.  And add
> stubs in bmi_* that implement this by calling free().
>
> Don't expect to see huge performance gains by doing this, though, as
> unexpected messages tend to be pretty small and everything will sit
> in L2 anyawy.  But saving the malloc/memcpy/free is probably worth
> the expansion of the API, I'll agree.  It'll also clear up any
> confusion about the lifetime of returned buffers, and bring
> responsibility for flow control of unexpected messages up to the
> application where it belongs.  Once you toss in these changes, I'll
> switch bmi_ib over to be clever with unexpected buffers too, as you
> describe.
>
> 		-- Pete

Sam and Murali brought this up based on my questions about  
BMI_memalloc() and BMI_memfree(). Unlike GM, MX does not have a  
memory registration API. It handles registration within the library  
automatically. The user always passes in plain malloc'd buffers for  
both send and receive. Since the unexpected buffer for a MX BMI would  
simply be a malloc'd buffer, I did not think it made much sense to  
malloc another buffer, copy the data and pass that back to BMI.

Given that, I could live with BMI_memalloc() and BMI_memfree() that  
are simply malloc() and free().

Scott