[Pvfs2-developers] [PATCH] threaded kmod helper

[Sam Lang]

On Feb 21, 2008, at 1:58 PM, Phil Carns wrote:

> Sam Lang wrote:
>> On Feb 14, 2008, at 4:34 PM, Pete Wyckoff wrote:
>>> Add --enable option to build a threaded pvfs2-client-core, that is,
>>> an executable linked against pvfs2-threaded.so.
>>>
>>> Remove the "--threaded" argument from pvfs2-client.  Whatever  
>>> version
>>> of client-core that was configured is all that is available at  
>>> runtime.
>>> It will always be installed as pvfs2-client-core.
>>> ---
>>> Makefile.in                          |   15 +++++++--------
>>> configure                            |   17 +++++++++++++++--
>>> configure.in                         |   20 ++++++++++++++++++++
>>> src/apps/kernel/linux/module.mk.in   |    9 ++++++---
>>> src/apps/kernel/linux/pvfs2-client.c |   31 + 
>>> +-----------------------------
>>> 5 files changed, 50 insertions(+), 42 deletions(-)
>>>
>>>
>>> What does everybody think about this?  I had complained a long time
>>> ago about always getting a pvfs2-client-core and
>>> pvfs2-client-core-threaded during a kernapps build.  The latter
>>> brings in the entire libpvfs2-threaded.a and all the objects for
>>> that.
>>>
>>> I don't have a feel for who uses the threaded client core.  The only
>>> way to use it now is to start pvfs2-client with the "--threaded"
>>> option.  So I made the --enable option default to off.  Is there a
>>> reason to want both threaded and non-threaded versions installed?
>> I think the patch looks great.  My only concern is that this will  
>> make the threaded client daemon never get used.  My view is that we  
>> should probably have the threaded version as the default.  I think  
>> in the general case, the threaded version is going to perform  
>> better.  Its pretty hard to buy a system that doesn't have at least  
>> two cores today, and even on a uni-processor system, the  
>> performance of the threaded daemon shouldn't be much worse than the  
>> non-threaded.
>> I originally added the threaded client functionality to see if it  
>> would help performance, but it turned out to be buggy.  Maybe with  
>> Phil's recent thread-safety fixes to the system interfaces, we  
>> should revisit making the threaded daemon the default.
>
> FYI, I think the threaded client just got a boost in stability.   
> There was a subtle bug in the how the job thread mgr was dealing  
> with the device that made it possible for it to pull operations out  
> that could not be handled.  I think this could technically happen  
> even with the non-threaded client, but this hasn't been a big deal  
> because it could only be triggered with a) a lot of concurrent  
> operations b) and a pvfs2-client-core daemon running fast enough to  
> empty out the device queue (much easier w/ a dedicated thread).
>
> There was also in the last week or two a fix to how bmi_tcp kept  
> track of socket errors that I suspect was relevant to the threaded  
> client core as well, but I was tracking that down in the context of  
> some server stuff so I can't say for sure.
>
> I agree with everything in this email thread, just commenting on the  
> state of the threaded client in case anyone following was curious.
>
>> I'm not sure that the overhead of a single operation will be  
>> affected much, honestly.  We can't disable threads and locking  
>> entirely, since we need the remount thread running.  I think the  
>> patch I just committed to fix the segfault in HEAD will likely help  
>> with individual operation performance more than disabling threads  
>> would.  Just my 2c though.
>> Its hard to get a feeling for how much the threaded daemon would  
>> help without a thorough performance analysis of the two on a number  
>> of different systems.  Something I would like to do at some point,  
>> but its pretty low on the list.
>
> I don't have a good feel for the threaded / non threaded impacts  
> performance either without looking into it more.
>
> For the non-threaded version, that remount thread really bugs me.   
> It seems like there has to be another way to accomplish whatever its  
> doing.  Its a shame to have to pull in lpthread and turn on all of  
> the library locks when the client-core doesn't need either for 99%  
> of what it does.

Yeah I agree.  I'm pretty sure it could be done with another state  
machine like our unexpected_sm.
-sam

>
>
> -Phil
>