[Pvfs2-developers] server crash on startup with millions of files

[Rob Ross]

Wow, I didn't think that this stuff would come up again soon :).

The current implementation (a) tracks what is free and (b) what is 
recently used, (c) lets the server choose the handle to return, and (d) 
keeps a global handle space (a handle is unique across all servers).

the point of (a) was to avoid having to hit storage to find a free 
handle. i agree that this is perhaps not that big a deal now that we 
have a better handle on how to efficiently use berkeley db.

(a) and (c) together mean that clients never have to retry to get a 
handle. i agree that in itself isn't all that valuable.

(b) ensures that clients caching metadata on some file don't end up 
accessing some newly created file's data, or deleting some new file's 
object, or some similar thing. this is an important part of allowing 
clients to cache file metadata (specifically datafile handles) without 
coordination.

(c) also allows us to precreate objects if we wanted to, although we 
don't do that right now. this would be less important if/when 
server-to-server communication is in place and we move file creation 
over to the server side.

(d) eventually allows us to move objects around without updating the 
file's metadata, assuming that we come up with a different mechanism for 
determining where a file resides. A bloom filter sort of approach might 
work, as an example. Or if server-to-server were working the servers 
could just figure out where things are with some aggregate comm.

walt's idea seems to allow us to map a collection of objects (a 
"segment") to a given server, then a client could pick values in that 
segment. my feeling is that this hamstrings our ability to move objects 
around, because we would then need to move entire segments around, as at 
the very least it could take a very long time to reach a consistent 
state again (think of many large objects needing to be moved; how do 
clients know where to contact?). this idea is a generalization of pete's 
idea to have a server id be part of the object handle; pete's approach 
makes it impossible to migrate without changing file metadata. more on 
this below.

pete's idea of speeding up creates by guessing at free handles, is ok, 
but the right way to speed up creates is to precreate. then the latency 
can be hidden in the mix of other operations. lustre already does this, 
and i believe it is every effective for them.

he is correct that randomly picking values would lead towards nasty data 
structures in the ledger. i'd be happy to see the "free" list part of 
the leder disappear if that helps. i do think that the "recently freed" 
list has to stay for the reason listed above, although it could be 
implemented differently perhaps -- maybe just leave an entry in the DB 
noting when the object was freed, and if it is referenced again after 
the appropriate time we consider it up for grabs? this has a nice 
side-effect of keeping the object "off limits" even if a server is 
restarted.

i don't understand why it is difficult to get a value in a particular 
range in the OSD work. can you clarify this pete? can't you just "guess" 
a value in the range until you get one?

one thing that we could discuss is the relative merit of migration using 
this sort of approach. maybe in fact this idea that i have that we want 
to keep a FS-wide object handle space is flawed, that changing file 
metadata can be addressed in a reasonable way that simplifies the 
overall system, allows for migration, and doesn't have a negative impact 
on our caching of metadata.

overall i think that changing how we reference objects, with the 
exception of perhaps redoing how we keep up with free/recently-freed 
objects, is something that should perhaps wait until we have 
server-to-server working. we're likely to want to make some changes at 
that point anyway, once the system has more control over the 
construction of files and directories. maybe we can discuss how we'd 
like things to work in that context and concentrate on getting there, 
rather than torquing things now and then perhaps messing with things again?

thanks everyone! it's fun to get to sit and think about this stuff, 
especially after many days of travel and meetings :).

regards,

rob

Walter B. Ligon III wrote:
> I don't understand this.  Is there a scheme whereby there is no mapping 
> of the handle ID to a server?  If not, then what we are talking about, I 
> think, is whether the server mapping is fixed or not.  The idea behind 
> the current scheme was to make the mapping of servers to handles 
> flexible.  That said, the specific implementation might could be better. 
>  For example, using 128 bits we could have a 64 bit segment tag and a 64 
> bit handle ID.  The segment tag would map the handle to a server via the 
> tables, and the ID would be unique within that segment.  This might 
> simplify some things without losing the flexibility we have.
> 
> As it is, the server can still randomly pick an ID, or a client could 
> randomly pick an ID, they just have to do it within a range, which isn't 
> particularly hard.  With this suggested modification we could 
> "eliminate" the range by giving all "handle ranges" a built-in extent of 
> 64 bits, which I think is the same as what you were suggesting.
> 
> If I'm not being clear, let me know and I'll try again.  Or, if I don't 
> understand the problem, let me know that.
> 
> Walt
> 
> Pete Wyckoff wrote:
>>
>> For create scalability, you may want the client to pick handle IDs
>> and offer those to the server, so that you can optimistically create
>> a metafile assuming there are no collisions on the server.  These
>> guessed handle IDs can be random though.  We did not implement this
>> as it would be quite expensive if implemented in terms of the
>> existing extent/extentlist/ledger data structures.
>>
>> In the OSD work, we have to do painful things to return a handle ID
>> in a particular range.  I would much rather have the server pick a
>> random ID and give it to the client.  Or for the client to try to
>> pick a particular ID and hope there is no collision at the server.
>>
>> So I'd like to discard the idea of pre-assigned per-server handle
>> ranges and augment our notion of PVFS_handle to include some sort of
>> "server identifier" as well as the 64-bit ID that is private to the
>> particular device on which the object sits.
>>
>> Various distributed FS implementations for wide-area use seem to be
>> happy with 128-bit handles and assume collisions will never happen.
>> This always struck me as wildly reckless, but maybe it is time to
>> accept the fact that these number spaces are really big.
>>
>>         -- Pete
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>