[Pvfs2-developers] [PATCH] threaded kmod helper

[Phil Carns]

Sam Lang wrote:
> 
> On Feb 14, 2008, at 4:34 PM, Pete Wyckoff wrote:
> 
>> Add --enable option to build a threaded pvfs2-client-core, that is,
>> an executable linked against pvfs2-threaded.so.
>>
>> Remove the "--threaded" argument from pvfs2-client.  Whatever version
>> of client-core that was configured is all that is available at runtime.
>> It will always be installed as pvfs2-client-core.
>> ---
>> Makefile.in                          |   15 +++++++--------
>> configure                            |   17 +++++++++++++++--
>> configure.in                         |   20 ++++++++++++++++++++
>> src/apps/kernel/linux/module.mk.in   |    9 ++++++---
>> src/apps/kernel/linux/pvfs2-client.c |   31 
>> ++-----------------------------
>> 5 files changed, 50 insertions(+), 42 deletions(-)
>>
>>
>> What does everybody think about this?  I had complained a long time
>> ago about always getting a pvfs2-client-core and
>> pvfs2-client-core-threaded during a kernapps build.  The latter
>> brings in the entire libpvfs2-threaded.a and all the objects for
>> that.
>>
>> I don't have a feel for who uses the threaded client core.  The only
>> way to use it now is to start pvfs2-client with the "--threaded"
>> option.  So I made the --enable option default to off.  Is there a
>> reason to want both threaded and non-threaded versions installed?
> 
> I think the patch looks great.  My only concern is that this will make 
> the threaded client daemon never get used.  My view is that we should 
> probably have the threaded version as the default.  I think in the 
> general case, the threaded version is going to perform better.  Its 
> pretty hard to buy a system that doesn't have at least two cores today, 
> and even on a uni-processor system, the performance of the threaded 
> daemon shouldn't be much worse than the non-threaded.
> 
> I originally added the threaded client functionality to see if it would 
> help performance, but it turned out to be buggy.  Maybe with Phil's 
> recent thread-safety fixes to the system interfaces, we should revisit 
> making the threaded daemon the default.

FYI, I think the threaded client just got a boost in stability.  There 
was a subtle bug in the how the job thread mgr was dealing with the 
device that made it possible for it to pull operations out that could 
not be handled.  I think this could technically happen even with the 
non-threaded client, but this hasn't been a big deal because it could 
only be triggered with a) a lot of concurrent operations b) and a 
pvfs2-client-core daemon running fast enough to empty out the device 
queue (much easier w/ a dedicated thread).

There was also in the last week or two a fix to how bmi_tcp kept track 
of socket errors that I suspect was relevant to the threaded client core 
as well, but I was tracking that down in the context of some server 
stuff so I can't say for sure.

I agree with everything in this email thread, just commenting on the 
state of the threaded client in case anyone following was curious.

> I'm not sure that the overhead of a single operation will be affected 
> much, honestly.  We can't disable threads and locking entirely, since we 
> need the remount thread running.  I think the patch I just committed to 
> fix the segfault in HEAD will likely help with individual operation 
> performance more than disabling threads would.  Just my 2c though.
>  Its hard to get a feeling for how much the threaded daemon would help 
> without a thorough performance analysis of the two on a number of 
> different systems.  Something I would like to do at some point, but its 
> pretty low on the list.

I don't have a good feel for the threaded / non threaded impacts 
performance either without looking into it more.

For the non-threaded version, that remount thread really bugs me.  It 
seems like there has to be another way to accomplish whatever its doing. 
  Its a shame to have to pull in lpthread and turn on all of the library 
locks when the client-core doesn't need either for 99% of what it does.

-Phil