[Pvfs2-developers] Active Storage?

[Sam Lang]

On Apr 24, 2007, at 11:26 AM, David Brown wrote:

> I know there's been lots of research into investigating the
> possibilities of putting some form of processing on archives with
> massive amounts of disks (active-storage). I was wondering if anyone
> has approached the pvfs developers with ideas of integrating an
> active-storage like system to the servers to allow them to perform
> preprocessing on the data before it gets sent over the network?
>
> One could implement something very quick using fuse and have the
> servers work on top of the fuse file system. However, that would
> depend on how the raw data of the files are stored on each of the
> storage servers. Could someone give me a place to start looking as to
> where the raw data of the files are located? algorithms used for
> distributing files among the storage servers? block sizes available?

Hi David,

The file's stripe on an IO server (we call it a byte-stream or  
bstream) is located at:

/pvfs-storage/<hex-fs-id>/bstreams/<bucket-hash>/<hex-handle>.bstream

The storage of files is all done in the trove dbpf component, namely  
dbpf-bstream.c.

The default algorithm (we call them 'distributions') for distributing  
files is our simple-stripe.  There's other distributions you can  
specify on a per-file basis.  The implementations of them are in src/ 
io/description/.

Let us know if you have other questions.

Thanks,

-sam

>
> Any information would be helpful.
>
> Thanks,
> - David Brown
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>