[Pvfs2-users] Confusion in size

[Phil Carns]

Hi Abhinav,

The PVFS file system will pretty much be full if one of the 60 GB drives 
fills up.  You can continue to read from it, of course, but you won't be 
able to write more data.

Technically it is possible to specify an explicit layout for each file 
(ie, dictate exactly which servers to stripe across).  In theory that 
would let you continue to use the servers with more capacity, but it 
would be really awkward and the interface isn't available to most 
applications.

-Phil

Abhinav Chawade wrote:
> Thank you Phil and Rob.
>  
> I could see the file size properly with pvfs2-statfs and karma gui. Phil 
> you mentioned in detail how pvfs2 calculates free space. Now if one of 
> the drives becomes full(say one of the 60 GB drives) and there is space 
> on other servers, will pvfs2 distribute data among them evenly or the 
> pvfs2 volume will beunusable?
> 
> On Fri, Feb 6, 2009 at 11:16 AM, Phil Carns <carns at mcs.anl.gov 
> <mailto:carns at mcs.anl.gov>> wrote:
> 
>     Hi Abhinav,
> 
>     To summarize, in total it sounds like you have PVFS servers running
>     on 14 nodes.  Most of those nodes only have 60 GB hard drives, but
>     one of them has 640 GB of storage.
> 
>     PVFS doesn't have any mechanism to account for unbalanced storage on
>     each of the servers.  If any one of your servers runs out of space
>     then the PVFS volume is pretty much out of space as well.  So when
>     it calculates the total space on the file system (for df) it uses
>     this algorithm:
> 
>     <capacity of smallest server> * <number of servers>
> 
>     ... which gets you to the 850 GB value that df is reporting.
> 
>     As far as the 52% being allocated already, this probably means that
>     on at least some of your servers, the server's storage directory is
>     on the same partition as your root file system (or some other data).
>      That means that things external to PVFS are counting against the
>     capacity on some servers.  There is nothing wrong with that
>     necessarily.  Most likely one or more of your machines has used up
>     half of its 60 GB already and PVFS therefore only has maybe 30 GB *
>     14 available out of the 60 GB * 14 total.
> 
>     If you want more detail about how much space is available on each
>     individual server, you can use the pvfs2-statfs command.
> 
>     There isn't any way to set a limit on each server from the PVFS
>     perspective.  You could do it at the node level by using a separate
>     partition for the PVFS data, though.
> 
>     -Phil
> 
>     Abhinav Chawade wrote:
> 
>         I have a 14 node cluster with 1 storage server 1 head node and
>         12 computational node running centos 5.2
>         I deployed pvfs2 on all of them and head and storage node are
>         metadata servers. Each computational node has 60GB hard drive
>         and the storage server has 640 GB storage.
>         When i mount the file system and ping it, all servers are
>         running. I try to see how much free space is left using df
>         command. df output shows all in all 850 GB storage out of which
>         52% is allocated already. There isn't a single file in pvfs2
>         mounted partition. How can i check size of pvfs2 space i
>         created? Is there a way to set limit on storage on each server?
> 
>         Abhinav
> 
> 
>         ------------------------------------------------------------------------
> 
>         _______________________________________________
>         Pvfs2-users mailing list
>         Pvfs2-users at beowulf-underground.org
>         <mailto:Pvfs2-users at beowulf-underground.org>
>         http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> 
> 
>