[Pvfs2-developers] fix BMI multiplexing of multiple methods

[Rob Ross]

Hey,

For the CALLBACK option, you would use that to have the individual  
methods filling in things at the "generic BMI" layer (for lack of the  
right terminology), but the overall user API would be the same?

I don't think that the CONTEXT option is appropriate. I don't want to  
expose the specifics of the underlying networks any more than we have  
already.

There should be relevant research in the MPI space related to the  
POLL_PLAN option.

Do we consider this to be a problem for both clients and servers, or  
is it really a server-specific issue? If this is something we think  
will solely (or mostly) a server thing, we could consider throwing a  
thread at the issue. One option might be to kick off a thread to wait  
on the TCP side of things, since the kernel is doing most of the work  
for us anyway, and put completed TCP events into the completion list  
asynchronously (for servers only)?

Rob

On Jan 7, 2009, at 4:06 PM, Sam Lang wrote:

>
> Hi All,
>
> Right now if multiple methods are enabled in BMI, we tend to get  
> poor performance from the "fast" network, because BMI_testcontext  
> iterates through all the active methods calling testcontext for each  
> one.  It tries to be smart about which methods get scheduled ;-) to  
> prevent starvation, but it treats all the methods fairly, which  
> tends to make tcp (the slow one) hog the time spent in testcontext.   
> I have a few ideas for this, so I'll go ahead and propose them and  
> let you all shoot them down or propose others.
>
> Option CALLBACK:  Instead of returning completion as a list in  
> testcontext, we allow a BMI context to be constructed with a  
> callback, and on completion of operations, the callback is called.   
> This allows each method to drive its own operations, and notify the  
> consumer of completion immediately.  There would still need to be a  
> testcontext call for methods that only service operations during  
> that call.  The changes might not be that significant, the  
> BMI_open_context call could just take an extra parameter that was  
> the callback function.  If the parameter is null, we just use the  
> completion list as before.
>
> Option CONTEXT:  Require separate contexts for separate methods.   
> This pushes the problem up to the application, probably not where it  
> belongs, since active methods are opaque from the BMI api.
>
> Option POLL_PLAN:  Modify the construct_poll_plan function in bmi  
> that already tries to be fair, so that its aware of the performance  
> discrepancy between methods.  Maybe it can just skip tcp every other  
> time for example.  This is probably the easiest, since it doesn't  
> require API changes and the like.
>
> -sam