[Pvfs2-developers] threaded client-core and the device thread

[Murali Vilayannur]

Hi Dean,
Searching for a slot out of a 5 element array should not be that slow (I 
hope :))
The only other thing I can think of:  by breaking down buffers into 4 KB 
chunks (page size chunks that is)
for copying, we could be slowing down when there are many such 
buffers/producer threads.
Hmm.. oh well I am handwaving. I will dig a little bit later this week 
and see if I can come up with something concrete.
thanks,
Murali

> Yes, this is exactly what we are seeing.  I have never tried a single 
> buffer, but looking a bit into the kernel code I can see some areas 
> that are a little inefficient.
> 1) In wait_for_a_slot <ident?v=pvfs2;i=wait_for_a_slot>, while holding 
> a spinlock, a thread must linearly search through all buffers to find 
> an available buffer.  If one is found, then fine.  If all buffers are 
> full (which is probably the common case if doing large I/O), it sleeps 
> until woken up, at which point it starts all over again, rescanning 
> the entire list for a buffer.  Technically, we could have starvation 
> for a thread, who continually gets unlucky about finding an empty 
> buffer.  My guess would be that this code is small should happen 
> really fast no matter what, but who knows.....
>
> 2) With multiple buffers, the threads will be fighting over using kmap 
> to copy the data to the mmapped buffer.  From my understanding of the 
> kernel (which may be outdated), there are very few kmap spinlocks 
> available, effectively serializing the process of copying data into 
> the mmapped buffers.  As we increase the number of buffers, this 
> contention will increase and the time to copy the data for any single 
> buffer will increase.
>