[Pvfs2-developers] handle ledger

[Sam Lang]

On Jan 28, 2008, at 4:46 PM, Murali Vilayannur wrote:

> Sam,
> Isn't it possible that creates can now fail after exhausting 32  
> retries?

Yes.  The odds of that happening though are extremely small.  Even  
with 16 million files in the filesystem, the odds of hitting  
collisions 32 times in a row are 1 in 250 billion.  I could just  
increase the retries to 1000, which makes the odds of the create  
failing before the handle space runs out something ridiculous.

There's a general problem here that as the filesystem gets extremely  
large, the accuracy of an ENOSPC error is directly proportional to the  
amount of time the create operation will take.

>
> Can we not load the ledger in the background while the server sets  
> itself up
> and let it continue  processing requests normally?
> that way we can still lookup some handles from the ledgers for
> create's handle allocations.

In these cases where collisions are more likely (huge filesystems),  
I'm not sure we can assume that the entire set of handles can be  
loaded into an in-memory ledger.  It is extent based, so it would have  
a fairly small memory footprint if files are only being added and  
never removed.  But failing that assumption, we would have to be able  
to store and query the ledger out of core.

RobR has mentioned before that the ledger was initially intended to be  
serializable so that it could be written to disk.  This would at least  
save recreating it on server startup, but as the handle space became  
more fragmented, we'd still have to worry about its being a memory  
hog.  But you're right, if we're targeting workloads where files are  
created and never removed, keeping the ledger is probably the right  
move.

-sam

>
> thanks,
> Murali
>
>
> On 1/28/08, Sam Lang <slang at mcs.anl.gov> wrote:
>>
>> Attached patch disables the handle ledger.  For those not familiar,
>> the handle ledger is an in-memory structure that maintains allocated
>> handles for a given server.  I'm disabling it because reading the
>> entire database each time the server loads is extremely expensive for
>> large filesystems.   Instead of choosing a handle from the ledger,  
>> the
>> patch picks one randomly.  This means we have to deal with collisions
>> now, but because of our large handle space, they only occur every 100
>> billion times or so.
>>
>> I didn't blow away the handle allocation code entirely...I just
>> disabled the calls that we had been using to invoke the handle  
>> ledger,
>> and added some functionality that picks a random handle from a given
>> range.  In the dspace code, I modified the create function to  
>> continue
>> up to 32 times if a collision with an already existing handle occurs.
>>
>> -sam
>>
>>
>>
>>
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>>
>>
>>
>