[Pvfs2-developers] server crash on startup with millions of files

[Sam Lang]

On Feb 20, 2007, at 11:32 AM, Pete Wyckoff wrote:

> pcarns at wastedcycles.org wrote on Tue, 20 Feb 2007 07:29 -0500:
>> dbpf-dspace.c:1371
>> assert(!dbpf_op_queue_empty(dbpf_completion_queue_array 
>> [context_id]));
>>
>> According to the stack trace, this test() call followed a
>> trove_dspace_iterate_handles() call within the
>> trove_check_handle_ranges() function.  This is part of the logic on
>> startup that scans all of the handles in the storage space to  
>> update the
>> list of available/used handles in trove-handle-mgmt.
>
> Another thought for Sam, who knows this code better.
>
> (1) DBPF_COMPLETION_START modifies cur_op->op.state without holding  
> the
> dbpf_completion_queue_array_mutex[cid] mutex.  Then it grabs the
> mutex and puts the op on the completion array.
>
> (2) dbpf_dspace_test grabs that mutex, looks at op.state, then asserts
> that the queue must not be empty.
>
> Perhaps (1) modifies the state but doesn't get around to putting it
> on the completion array.  (Possibly because the lock is held in
> (2).)
>
Good point Pete.  Given that this seems to be race Phil is seeing,  
your theory seems more likely.

> Maybe (1) should put the op on the array before modifying its state,
> and hold the array mutex the whole time.  I'm not sure what kind of
> locking rules are involved between the mutex in the op and the mutex
> on the completion array, though.  Or what else might break with such
> a change.

I don't think there should be any problems with doing this.  It  
probably doesn't matter when the op is added to the completion queue  
(before or after its state gets changed), just that the completion  
queue's mutex gets locked before either (at the top of  
DBPF_COMPLETION_START).  I wonder if Phil could make this change and  
run his tests again.

>
> 		-- Pete
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>