[Pvfs2-developers] threaded client-core and the device thread

[Sam Lang]

Hi All,

Dean and I are looking at trying to push the efficiency of requests  
from the kernel module up through the device to client-core.  I added  
the --threaded option to the client to allow the client-core to run  
with multiple threads (one each for bmi, dev, and main -- and also a  
remount thread, but lets ignore that for now), so the device thread  
should be able to keep pulling requests of the device without having  
to wait for bmi operations to complete.

I noticed a couple things with the device thread that I wanted to ask  
about.

PINT_dev_test_unexpected takes an incount of 5, so its only going to  
read at most 5 requests off the device for each call.  Once it  
returns, each of the unexpected requests is added to the completed  
jobs array and then we signal the jobs completed condition variable  
_for each request_.  It seems like this will be 5x the number of  
context switches between the device thread and the main thread that  
we need.

Also, we poll every time before reading another request off the  
device.  What about trying to read a number of requests off the  
device at once with one read (or possibly a readv so we can keep  
separate buffers per request).

Also, it looks like we do a malloc for each new request buffer, and  
then a free once we're done with it, and a memset of the info  
struct.  It seems like we could manage the buffers on the stack  
instead of the heap, and save on a few system calls there.

For both threaded and nonthreaded, with the workload that Dean is  
using, he found that the PINT_dev_test_unexpected always returned 5  
requests in the outcount.  So it looks like there are always requests  
sitting on the device, waiting to be read by client-core.  Are we  
just not able to process requests fast enough through BMI and the  
state machines, or is the cost of polling and signaling every time we  
read a request off the device slowing us down?  In other words, does  
it make sense to rework the code a little bit or will we just get  
bottlenecked elsewhere?

-sam