[Pvfs2-developers] server threads and kernel timers

[Sam Lang]

On Dec 13, 2006, at 8:45 AM, Pete Wyckoff wrote:

> slang at mcs.anl.gov wrote on Mon, 11 Dec 2006 14:34 -0600:
>> This is some really nice analysis Pete.  One thing we might consider
>> for reducing context switches would be to re-use the coalescing ideas
>> we've tried to apply for meta data syncs to context switches as
>> well.  If we've got a lot of operations completing in a particular
>> module (trove for example), we can just signal once instead of for
>> each operation.  The sync coalescing code already does this, but you
>> would only see any benefits from that with a bunch of clients.
>
> Maybe that would be effective for a loaded server, but that won't
> help single-op latency much.
>
>> The locking and context switches are inherent to our design of server
>> module separation and queueing framework.  I think it would be hard
>> to replace the context switches without doing some serious redesign.
>
> Agreed.
>
>> We might be able to eliminate the trove thread though.  It doesn't do
>> anything but move items from the trove completion queue to the job
>> completion queue.  Since that thread waits on a condition variable
>> (and gets signalled by trove), and then signals the job completion
>> condition variable, we're essentially doing a double context switch
>> when we only need one.  Instead we could change the trove apis to
>> take a callback and user ptr, and have the callback add the completed
>> job to the completion queue directly.  The bits of flow that use job
>> callbacks with trove would have to be changed too, but I think the
>> flows would benefit from the bmi callback being called directly from
>> trove as well.  Does this seem reasonable?
>
> This is a good idea.  The job- or flow-supplied callback function
> can do whatever locking it needs to do protect its own resources.
> Before the callback is invoked you can make sure the completion is
> out of any trove lists.  If you are interested in doing this part,
> I'll be happy to do some more timing tests.

Yeah I'll try to take a whack at it.

-sam

>
> 		-- Pete
>