[Pvfs2-users] Problems with basic distribution (with xattrs)

Tue Jul 1 10:50:34 EDT 2008

The twod_stripe problem that you describe sounds like the one that Kyle 
Schochenmaier found and fixed in this mailing list thread:

http://www.beowulf-underground.org/pipermail/pvfs2-developers/2008-June/004069.html

You can check by running pvfs2-viewdist -f <filename> on the file that 
you create with these parameters.  If the num_groups shows up as 2 then 
you are hitting a bug that prevents the distribution from honoring that 
setting.

The attached patch should fix it if that's the case.

-Phil

Tony Kew wrote:
> Dear Phil,
> 
> On a fist pass it looks like the patch fixes the basic_dist, but breaks
> Murali's  varstrip_dist xattrs patch.  I'll try some more tests just to
> make sure I didn't fat finger something ...
> 
> I'm also beginning to suspect that there is problem with xattrs
> settings for twod_stripe.  Testing on a single node (which I must
> say makes no sense for a 2D stipe) & setting the strip to use one
> group (since there is only one node) the server logfile shows errors
> regarding using 2 groups with one server, so the xattr setting might
> not be getting honored (or you cannot use this distribution without
> at least two servers)
> 
> e.g.
> 
>    $ mkdir twod_stripe
>    $ setfattr -n "user.pvfs2.dist_name" -v "twod_stripe" twod_stripe
>    $ setfattr -n "user.pvfs2.dist_params" \
>    >  -v "strip_size:2048;num_groups:1;group_strip_factor:512" twod_stripe
>    $ cd twod_stripe/
>    $ dd if=/dev/urandom of=testfile bs=16k count=16
>    16+0 records in
>    16+0 records out
>    262144 bytes (262 kB) copied, 0.162147 seconds, 1.6 MB/s
>    $ ls -l
>    total 260
>    -rw-r--r-- 1 tonykew ccrstaff 262144 Jun 30 15:52 testfile
>    $
> 
> The fie is created (and is the correct size), but the server logfile
> shows many errors:
> 
>    [E 06/30 15:52] physical_to_logical_offset: Invalid 
> num_groups/server_ct options: gr:2 server:1
>    [E 06/30 15:52] next_mapped_offset: Invalid num_groups/server_ct 
> options: gr:2 server:1
>    [E 06/30 15:52] logical_to_physical_offset: Invalid 
> num_groups/server_ct options: gr:2 server:1
>    [E 06/30 15:52] next_mapped_offset: Invalid num_groups/server_ct 
> options: gr:2 server:1
>    [E 06/30 15:52] logical_to_physical_offset: Invalid 
> num_groups/server_ct options: gr:2 server:1
>    etc...
> 
> If I read this correctly, the number of groups "gr:2" is still 2 even 
> though I set it
> to 1 with the fsetattr command
> 
> 
> 
> Many Thanks for your help,
> Tony
> 
> Tony Kew
> SAN Administrator
> The Center for Computational Research
> New York State Center of Excellence
> in Bioinformatics & Life Sciences
> 701 Ellicott Street, Buffalo, NY 14203
> 
> CoE Office: (716) 881-8930           Fax: (716) 849-6656
> CSE Office: (716) 645-3797 x2174
>      Cell: (716) 560-0910          Home: (716) 874-2126
> 
> "I love deadlines, I love the whooshing noise they make as they go by."
>                                                          Douglas Adams
> 
> 
> 
> Phil Carns wrote:
>> Hi Tony,
>>
>> Thanks for the bug report.  I have not looked at the varstrip issue 
>> yet, but the attached patch (also available in cvs head) should at 
>> least fix the basic_dist problem.
>>
>> The problem was that basic_dist only works for files with 1 datafile, 
>> but the logic that lets it override the number of datafiles was 
>> happening before the directory hints were processed.
>>
>> Someone correct me if I'm wrong, but I reorganized the code a little 
>> chose the number of datafiles in this priority (after making sure the 
>> final distribution for the file is set first, via hints or otherwise):
>>
>> 1) count specified in attr's passed into sys_create
>>    - example: -n option to pvfs2-cp
>> 2) directory hints
>>    - example: "user.pvfs2.num_dfiles" extended attribute
>> 3) mount options
>>    - example: num_dfiles option in /etc/pvfs2tab
>> 4) system default
>>    - example: DefaultNumDfiles in server conf, or number of servers
>>
>> The distribution itself gets a chance to override the value at the end 
>> if (as in the case of basic_dist) it doesn't support an arbitrary 
>> number of datafiles.
>>
>> I'll look into the varstrip issue soon.
>>
>> -Phil
>>
>> Tony Kew wrote:
>>> Dear PVFS users list,
>>>
>>> Mounting a pvfs 7.2.1, creating directories and setting the distribution
>>> type with setfattr for each directory, only simple_stipe and twod_stripe
>>> would allow me to write a file in the given directory.
>>>
>>> Adding the varstrip_dist xattrs patch from Murali Vilayannur attached to
>>> this message thread:
>>>
>>>    http://osdir.com/ml/file-systems.pvfs2.users/2007-01/msg00050.html
>>>
>>> fixes the variable strip distibution, but the basic distribution still
>>> doesn't work for me.
>>>
>>> The patch adds
>>>
>>>    #include "pvfs2-dist-basic.h
>>>
>>> to src/client/sysint/sys-create.sm so I wonder if no one has tried 
>>> this...???
>>> Perhaps there is a little more code needed to make basic_dist work with
>>> xargs?
>>>
>>> The patch above is against 2.6, but works (and is needed for 
>>> varstrip_dist
>>> with xargs) with 7.2.1  & the latest cvs (best I can tell) - perhaps 
>>> this patch
>>> could be added to both?
>>>
>>> Let me know if there is useful debug info I can provide...
>>>
>>>
>>> Thanks Much
>>> Tony
>>>
>>> Tony Kew
>>> SAN Administrator
>>> The Center for Computational Research
>>> New York State Center of Excellence
>>> in Bioinformatics & Life Sciences
>>> 701 Ellicott Street, Buffalo, NY 14203
>>>
>>> CoE Office: (716) 881-8930           Fax: (716) 849-6656
>>> CSE Office: (716) 645-3797 x2174
>>>      Cell: (716) 560-0910          Home: (716) 874-2126
>>>
>>> "I love deadlines, I love the whooshing noise they make as they go by."
>>>                                                          Douglas Adams
>>>
>>> _______________________________________________
>>> Pvfs2-users mailing list
>>> Pvfs2-users at beowulf-underground.org
>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs-twod_stripe.patch
Type: text/x-diff
Size: 2225 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20080701/8caa0d84/pvfs-twod_stripe.bin