[Pvfs2-developers] BMI_testunexpected and free

[Sam Lang]

On Aug 18, 2006, at 8:33 PM, Pete Wyckoff wrote:

> slang at mcs.anl.gov wrote on Fri, 18 Aug 2006 18:44 -0500:
>> The BMI_testeunexpected call returns a list of BMI_unexpected_info
>> structs.  Each of these has a buffer that's been allocated by the bmi
>> method internally.  It looks like that buffer gets freed in
>> server_state_machine_complete (pvfs2-server.c) with a call to free
>> (also gets freed on an error).  Two questions:
>>
>> 1.  Should BMI_memfree be used instead of free, since the allocation
>> is done in the specific bmi method?
>
> The unexpected message is received into network-registered memory,
> but then copied to hand back to the user.  This copied buf is just
> malloc()-ed, not BMI_memalloc()-ed, because it won't ever be used
> for BMI operations (okay, unlikely at least).  So free is right.
>
> Under the hood, the network receive buffer is recycled back into
> service before the upper layers look at it.  In IB, this is done at
> the end of a successful testunexpected to provide some flow control
> on unex messages.  In GM, the memcpy happens earlier, so that the
> network receive buffer gets put back into service as soon as the
> unex message comes off the network.

Hi Pete,

Thanks for the method details.  My concern is mostly with the  
interface semantics for future bmi method implementations, as well as  
other users of bmi (possibly outside of pvfs).  Could a BMI method  
potentially return unexpected message buffers not allocated with malloc?

Also, from a consistency perspective, I would prefer to have bmi  
methods return malloced buffers, and not expect the user to know to  
call free on that buffer.  If only that bmi may at some point be a  
separate library, allocating memory in one library and freeing it in  
another is usually considered bad practice.

-sam

>
>> 2.  Could the free of that buffer occur in
>> server_state_machine_start, right after the call to PINT_decode?  It
>> doesn't look like the buffer is used after that, and it might allow
>> the bmi method to reuse buffers better if it handles unexpected
>> messages that way.  At the least, it might reduce the server's
>> footprint a little under heavy load.
>
> Sure.  It's your buffer.  Do with it what you will, as long as you
> free it when you're done.  Might reduce a bit of RSS, but unex
> messages are small to start with and we don't expect too many of
> them on each connection.  Still, earlier is better; I guess right
> after the PINT_decode() should work.
>
> 		-- Pete
>