[Pvfs2-developers] Re: [Pvfs2-users] OrangeFS on nodes with different storage space capacities.

Becky Ligon ligon at omnibond.com
Tue Feb 21 17:44:10 EST 2012


Andrew:

We have a distribution that should help you with your storage problem.
Here is a link to a wiki that describes how to use it:
http://www.orangefs.org/trac/orangefs/wiki/Distributions.  In your case, I
would add this distribution to your config file:

<Filesystem>
    <Distribution>
           Name varstrip_dist
            Param strips
            Value 0:1S; 1:1S; 2:2S; 3:1S
    </Distribution>
</Filesystem>

where 1S=64K and 2S=128K (for example).

You can set the distribution but, by default, the list of physical servers
chosen for a logical file is still round-robin.  So, for example, if file1
is given servers a1,a2,a3,a4, in that order,  then server a3 will be given
twice the data, because the distribution specifies that the third server in
the list should have twice the data.  The next file created may be given
a2,a3,a4,a1, in that order, so server a4 will be given twice the data.
Distributions are not directly tied to specific servers.

To modify this round-robin behavior, you will have to change one line of
code in src/client/sysint/sys-create.sm:

In function PVFS_isys_create:

*if*(layout <http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>)
    228  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#228>
   {
    229  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#229>
       /* make sure it is a supported layout */
    230  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#230>
       *switch*(layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>)
    231  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#231>
       {
    232  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#232>
           /* these are valid */
    233  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#233>
           *case* PVFS_SYS_LAYOUT_ROUND_ROBIN
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_ROUND_ROBIN&project=trunk>:
    234  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#234>
           *case* PVFS_SYS_LAYOUT_RANDOM
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_RANDOM&project=trunk>:
    235  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#235>
           *case* PVFS_SYS_LAYOUT_LIST
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_LIST&project=trunk>:
    236  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#236>
               *break*;
    237  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#237>
           /* anything else is not */
    238  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#238>
           *default*:
    239  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#239>
               *return*(-PVFS_EINVAL
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_EINVAL&project=trunk>);
    240  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#240>
       }
    241  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#241>
    242  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#242>
       sm_p <http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>
= layout <http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>;
    243  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#243>
       *if*(layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>
== PVFS_SYS_LAYOUT_LIST
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_LIST&project=trunk>)
    244  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#244>
       {
    245  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#245>
           sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>
= layout <http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>;
    246  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#246>
           sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>
=
    247  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#247>
               malloc
<http://devorange.clemson.edu:8080/source/s?defs=malloc&project=trunk>(layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>
* sizeof <http://devorange.clemson.edu:8080/source/s?defs=sizeof&project=trunk>(PVFS_BMI_addr_t
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_BMI_addr_t&project=trunk>));
*if*(layout <http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>)
    228  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#228>
   {
    229  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#229>
       /* make sure it is a supported layout */
    230  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#230>
       *switch*(layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>)
    231  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#231>
       {
    232  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#232>
           /* these are valid */
    233  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#233>
           *case* PVFS_SYS_LAYOUT_ROUND_ROBIN
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_ROUND_ROBIN&project=trunk>:
    234  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#234>
           *case* PVFS_SYS_LAYOUT_RANDOM
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_RANDOM&project=trunk>:
    235  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#235>
           *case* PVFS_SYS_LAYOUT_LIST
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_LIST&project=trunk>:
    236  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#236>
               *break*;
    237  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#237>
           /* anything else is not */
    238  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#238>
           *default*:
    239  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#239>
               *return*(-PVFS_EINVAL
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_EINVAL&project=trunk>);
    240  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#240>
       }
    241  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#241>
    242  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#242>
       sm_p <http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>
= layout <http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>;
    243  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#243>
       *if*(layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>
== PVFS_SYS_LAYOUT_LIST
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_LIST&project=trunk>)
    244  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#244>
       {
    245  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#245>
           sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>
= layout <http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>;
    246  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#246>
           sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>
=
    247  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#247>
               malloc
<http://devorange.clemson.edu:8080/source/s?defs=malloc&project=trunk>(layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>
* sizeof <http://devorange.clemson.edu:8080/source/s?defs=sizeof&project=trunk>(PVFS_BMI_addr_t
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_BMI_addr_t&project=trunk>));
    248  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#248>
           *if*(!sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>)
    249  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#249>
           {
    250  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#250>
               *return* -PVFS_ENOMEM
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_ENOMEM&project=trunk>;
    251  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#251>
           }
    252  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#252>
           memcpy
<http://devorange.clemson.edu:8080/source/s?defs=memcpy&project=trunk>(sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>,
    253  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#253>
                  layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>,
    254  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#254>
                  layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>
* sizeof <http://devorange.clemson.edu:8080/source/s?defs=sizeof&project=trunk>(PVFS_BMI_addr_t
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_BMI_addr_t&project=trunk>));
    255  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#255>
       }
    256  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#256>
   }
    257  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#257>
   *else*server 0 will be given 1S of data, server 1 will be given 1S
of data,
server 2 will be given 2S of data, and server 3 will be given 1S of
data.
    258  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#258>
   {
    259  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#259>
       sm_p <http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>
= PVFS_SYS_LAYOUT_ROUND_ROBIN
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_ROUND_ROBIN&project=trunk>;
    260  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#260>
   }
    248  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#248>
           *if*(!sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>)
    249  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#249>
           {
    250  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#250>
               *return* -PVFS_ENOMEM
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_ENOMEM&project=trunk>;
    251  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#251>
           }
    252  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#252>
           memcpy
<http://devorange.clemson.edu:8080/source/s?defs=memcpy&project=trunk>(sm_p
<http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>,
    253  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#253>
                  layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.servers
<http://devorange.clemson.edu:8080/source/s?defs=servers&project=trunk>,
    254  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#254>
                  layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>->server_list
<http://devorange.clemson.edu:8080/source/s?defs=server_list&project=trunk>.count
<http://devorange.clemson.edu:8080/source/s?defs=count&project=trunk>
* sizeof <http://devorange.clemson.edu:8080/source/s?defs=sizeof&project=trunk>(PVFS_BMI_addr_t
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_BMI_addr_t&project=trunk>));
    255  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#255>
       }
    256  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#256>
   }
    257  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#257>
   *else*


/***************** change the default layout in the line below to
PVFS_SYS_LAYOUT_NONE instead of PVFS_SYS_LAYOUT_ROUND_ROBIN
*******************/



    258  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#258>
   {
    259  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#259>
       sm_p <http://devorange.clemson.edu:8080/source/s?defs=sm_p&project=trunk>->u.create
<http://devorange.clemson.edu:8080/source/s?defs=create&project=trunk>.layout
<http://devorange.clemson.edu:8080/source/s?defs=layout&project=trunk>.algorithm
<http://devorange.clemson.edu:8080/source/s?defs=algorithm&project=trunk>
= PVFS_SYS_LAYOUT_ROUND_ROBIN
<http://devorange.clemson.edu:8080/source/s?defs=PVFS_SYS_LAYOUT_ROUND_ROBIN&project=trunk>;
    260  <http://devorange.clemson.edu:8080/source/xref/trunk/src/client/sysint/sys-create.sm#260>
   }


By specifiying PVFS_SYS_LAYOUT_NONE, the system will ALWAYS choose server
zero first, which corresponds to the first server that you defined in your
config file, instead of the next round-robin server, then, server one,
server two, and server three, again in order of the servers defined in your
config file.

Continuing with the above distribution example, whenever a file is created,
the order of the servers will always be a1,a2,a3,a4.  So, a3 will always be
given twice the data as the other 3.  Just be sure that a3 is the third
server specified in the config file.

Apparently, when this variable strip distribution was developed, all of the
user API's weren't updated to allow easy use.  Hopefully, we can change
that!

Let me know if you have any questions.  We haven't actually tried this and
would like to know if it is still working properly!

Thanks,
Becky



On Mon, Feb 20, 2012 at 10:16 PM, Becky Ligon <ligon at omnibond.com> wrote:

> Andrew:
>
> It is possible to use a different distribution (other than simple stripe)
> that will take advantage of servers with different space allocations.
> However, the code that performs this service hasn't been used in a very
> long time.  My co-worker is going to look into it and see if this
> distribution is still viable.  I'll get back to you when we know.
>
> Becky
>
>
> On Mon, Feb 20, 2012 at 4:30 PM, Andrew Savchenko <bircoph at gmail.com>wrote:
>
>> Hello,
>>
>> On Mon, 20 Feb 2012 13:47:45 -0500 Becky Ligon wrote:
>> > Andrew:
>> >
>> > Having two servers on the 2S machine should work fine, but it does
>> depend
>> > on the storage backend and how it is connected to the server.  If you
>> are
>> > using local storage, then you are correct in that having two servers
>> > *could* be a performance hit; however, if you are using some sort of
>> SAN,
>> > then that statement may not be valid.  In our own environment, we had 4
>> > servers running on one machine attached to a DDN device, and performance
>> > was good.  How is your 2S storage attached?
>>
>> Each S is just a local SATA hard drive, though with good hardware
>> cache. Each node has 8 cores and 8 GB RAM. 2S node is a master node,
>> it has double in disks and half in CPU/RAM: 4/4. On top of that each
>> node will be used for computing (MPI/PBS), master node will be used
>> only partially. I'm aware, using the same hosts for both storage and
>> computing will hurt overall performance, but with current hardware it
>> is the best I can do.
>>
>> Best regards,
>> Andrew Savchenko
>>
>
>
>
> --
> Becky Ligon
> OrangeFS Support and Development
> Omnibond Systems
> Anderson, South Carolina
>
>
>


-- 
Becky Ligon
OrangeFS Support and Development
Omnibond Systems
Anderson, South Carolina
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120221/f945aeaa/attachment-0001.htm


More information about the Pvfs2-developers mailing list