[Pvfs2-users] Problems with the twod_stripe distribution (with
xattrs)
Tony Kew
tonykew at ccr.buffalo.edu
Tue Jul 1 14:21:26 EDT 2008
Dear Kyle,
Thanks, that was the fix - which follows what is says in 6.6.2 of
the FAQ (so my bad!)
I got confused with the syntax for the varstrip_dist where the
node_number:strip_size pairs are separated by semicolons ';'
As I mentioned before it would be useful if the setfattr command
would fail (ideally with a useful error message) if the parameters
are incorrect, though I don't know how hard this would be to achieve?
Many thanks to you & Phil for helping me with this,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Kyle Schochenmaier wrote:
> It looks like you might have the setfattr syntax wrong?
>
> Here is an example that works on my systems with the previous patching:
> setfattr -n "user.pvfs2.dist_params" -v
> "strip_size:4194304,num_groups:3,group_strip_factor:32" /scratch/2d
>
> You need to specify num_groups >= num_servers, the groups are the 2D
> part, so no you cant really do a 2D stripe on a single server, but it
> should still work, the code is there to make it work exactly like
> simple_stripe if there is only one group.
>
> If there is an issue with the setfattr, the distribution will set the
> attributes to default (2), I will roll out a patch this afternoon to
> make the default num_groups = 1 so this dist will not break on a
> single-server.
>
> For now, try to set the num_groups = 1, and see if you are still
> having a problem with setfattr.
>
>
> Kyle
>
> On Tue, Jul 1, 2008 at 12:09 PM, Tony Kew <tonykew at ccr.buffalo.edu> wrote:
>
>> Dear Phil,
>>
>> I applied the patch, but it doesn't work for me when testing with a single
>> node (if this is a valid thing to do with a 2D stripe...)
>> Could you check that I got the setfattr syntax correct, I'd hate for the
>> problem to be that! (...user error...)
>>
>> Would it be possible for the fsetattr call to fail if the parameters are
>> invalid?
>> e.g. setting
>> $ setfattr -n "user.pvfs2.dist_params" -v "num_groups:3" _some_dir
>> would fail if you only have 2 or less server nodes, strip_size should always
>> be greater than zero etc...
>>
>>
>> $ mkdir twod_stripe/
>> $ setfattr -n "user.pvfs2.dist_name" -v "twod_stripe" twod_stripe
>> $ setfattr -n "user.pvfs2.dist_params" \
>>
>>> -v "strip_size:16384;num_groups:1;group_strip_factor:512" twod_stripe
>>>
>> $ cd twod_stripe/
>> $ dd if=/dev/urandom of=testfile bs=16k count=16
>> 16+0 records in
>> 16+0 records out
>> 262144 bytes (262 kB) copied, 0.148667 seconds, 1.8 MB/s
>> $ ls -l testfile
>> -rw-r--r-- 1 tonykew ccrstaff 262144 Jul 1 12:48 testfile
>> $ pvfs2-viewdist -f testfile
>> dist_name = twod_stripe
>> dist_params:
>> num_groups:2,strip_size:65536,factor:256
>>
>> Number of datafiles/servers = 1
>> Server 0 - tcp://ramones:3334, handle: 9223372036854775801
>> (7ffffffffffffff9.bstream)
>> $
>>
>> The server logfile contains (as before)
>>
>> [...]
>> [E 07/01 12:45] logical_to_physical_offset: Invalid num_groups/server_ct
>> options: gr:2 server:1
>> [E 07/01 12:45] next_mapped_offset: Invalid num_groups/server_ct options:
>> gr:2 server:1
>> [E 07/01 12:45] logical_to_physical_offset: Invalid num_groups/server_ct
>> options: gr:2 server:1
>> [E 07/01 12:45] next_mapped_offset: Invalid num_groups/server_ct options:
>> gr:2 server:1
>> [...]
>>
>> If I set num_groups to 3 (just to test):
>>
>> $ rm testfile
>> $ cd ..
>> $ setfattr -n "user.pvfs2.dist_params" -v
>> "strip_size:16384;num_groups:3;group_strip_factor:512" twod_stripe
>> $ echo $?
>> 0
>> $ cd twod_stripe/
>> $ dd if=/dev/urandom of=testfile bs=16k count=16
>> 16+0 records in
>> 16+0 records out
>> 262144 bytes (262 kB) copied, 0.144843 seconds, 1.8 MB/s
>> $ ls -l testfile
>> -rw-r--r-- 1 tonykew ccrstaff 262144 Jul 1 12:48 testfile
>> $ pvfs2-viewdist -f testfile
>> dist_name = twod_stripe
>> dist_params:
>> num_groups:2,strip_size:65536,factor:256
>>
>> Number of datafiles/servers = 1
>> Server 0 - tcp://ramones:3334, handle: 9223372036854775800
>> (7ffffffffffffff8.bstream)
>> $
>>
>> server log file also (still) shows 2 groups:
>> [E 07/01 12:48] logical_to_physical_offset: Invalid num_groups/server_ct
>> options: gr:2 server:1
>> [E 07/01 12:48] next_mapped_offset: Invalid num_groups/server_ct options:
>> gr:2 server:1
>> [E 07/01 12:48] logical_to_physical_offset: Invalid num_groups/server_ct
>> options: gr:2 server:1
>> [E 07/01 12:48] next_mapped_offset: Invalid num_groups/server_ct options:
>> gr:2 server:1
>>
>> Thanks,
>> Tony
>>
>> Tony Kew
>> SAN Administrator
>> The Center for Computational Research
>> New York State Center of Excellence
>> in Bioinformatics & Life Sciences
>> 701 Ellicott Street, Buffalo, NY 14203
>>
>> CoE Office: (716) 881-8930 Fax: (716) 849-6656
>> CSE Office: (716) 645-3797 x2174
>> Cell: (716) 560-0910 Home: (716) 874-2126
>>
>> "I love deadlines, I love the whooshing noise they make as they go by."
>> Douglas Adams
>>
>>
>>
>> Phil Carns wrote:
>>
>>> The twod_stripe problem that you describe sounds like the one that Kyle
>>> Schochenmaier found and fixed in this mailing list thread:
>>>
>>>
>>> http://www.beowulf-underground.org/pipermail/pvfs2-developers/2008-June/004069.html
>>>
>>> You can check by running pvfs2-viewdist -f <filename> on the file that you
>>> create with these parameters. If the num_groups shows up as 2 then you are
>>> hitting a bug that prevents the distribution from honoring that setting.
>>>
>>> The attached patch should fix it if that's the case.
>>>
>>> -Phil
>>>
>>> [...]
>>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-user
More information about the Pvfs2-users
mailing list