[Pvfs2-developers] server crash on startup with millions of files

[Phil Carns]

It ended up taking a little work to get another environment to trigger 
this reliably, but I think I have something now.

I modified the iterate_handles() function a bit so that it keeps 
scanning over and over again indefinitely rather than letting the server 
start up.  This forces the code path in question without having to 
restart the servers.  Using this setup I'm able to trigger it on an 
empty 8 node file system, but I have to leave all of the servers running 
on it anywhere from a few minutes to half an hour before one of them 
crashes.  Oddly enough, with this environment it crashes faster on an 
emtpy file system than one with 500,000 files.

I repeated this test with the latest HEAD version from trunk, and that 
didn't seem to make any difference.

I'll try the mutex suggestion next.

-Phil

Sam Lang wrote:
> 
> On Feb 20, 2007, at 11:32 AM, Pete Wyckoff wrote:
> 
>> pcarns at wastedcycles.org wrote on Tue, 20 Feb 2007 07:29 -0500:
>>
>>> dbpf-dspace.c:1371
>>> assert(!dbpf_op_queue_empty(dbpf_completion_queue_array [context_id]));
>>>
>>> According to the stack trace, this test() call followed a
>>> trove_dspace_iterate_handles() call within the
>>> trove_check_handle_ranges() function.  This is part of the logic on
>>> startup that scans all of the handles in the storage space to  update 
>>> the
>>> list of available/used handles in trove-handle-mgmt.
>>
>>
>> Another thought for Sam, who knows this code better.
>>
>> (1) DBPF_COMPLETION_START modifies cur_op->op.state without holding  the
>> dbpf_completion_queue_array_mutex[cid] mutex.  Then it grabs the
>> mutex and puts the op on the completion array.
>>
>> (2) dbpf_dspace_test grabs that mutex, looks at op.state, then asserts
>> that the queue must not be empty.
>>
>> Perhaps (1) modifies the state but doesn't get around to putting it
>> on the completion array.  (Possibly because the lock is held in
>> (2).)
>>
> Good point Pete.  Given that this seems to be race Phil is seeing,  your 
> theory seems more likely.
> 
>> Maybe (1) should put the op on the array before modifying its state,
>> and hold the array mutex the whole time.  I'm not sure what kind of
>> locking rules are involved between the mutex in the op and the mutex
>> on the completion array, though.  Or what else might break with such
>> a change.
> 
> 
> I don't think there should be any problems with doing this.  It  
> probably doesn't matter when the op is added to the completion queue  
> (before or after its state gets changed), just that the completion  
> queue's mutex gets locked before either (at the top of  
> DBPF_COMPLETION_START).  I wonder if Phil could make this change and  
> run his tests again.
> 
>>
>>         -- Pete
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>>
>