[Pvfs2-users] Problems with xattr settings with long arguments

Wed Oct 22 12:10:23 EDT 2008

Hi Tony,

Would you mind trying the attached patch?  It increases the maximum 
xattr value size to 8KB and also fixes a problem further down in the 
code regarding how to handle xattrs that large.  The latter part of the 
patch should solve the problem that you ran into in which you could only 
set a large xattr once.

thanks,
-Phil

Tony Kew wrote:
> Dear Phil,
> 
> I verified that this is not an issue with the underlying filesystem:
> 
> c23n12$ mount | grep \ /scratch\
> /dev/sda3 on /scratch type ext2 (rw)
> c23n12$ sudo umount /scratch
> c23n12$ sudo mount -o rw,user_xattr /dev/sda3 /scratch
> c23n12$ mount | grep \ /scratch\
> /dev/sda3 on /scratch type ext2 (rw,user_xattr)
> c23n12$ cd /scratch
> c23n12$ mkdir test
> c23n12$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
> c23n12$ getfattr test/
> # file: test
> user.pvfs2.dist_name
> 
> c23n12$ setfattr -n "user.pvfs2.dist_params" -v 
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K" 
> test
> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K" 
> 
> 
> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
> c23n12$ getfattr -n user.pvfs2.dist_params test
> # file: test
> user.pvfs2.dist_params="strips:0:16K"
> 
> c23n12$ setfattr -n "user.pvfs2.dist_params" -v 
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K" 
> test
> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K" 
> 
> 
> c23n12$ setfattr -n "user.pvfs2.dist_params" -v 
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K" 
> test
> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K" 
> 
> 
> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" 
> testc23n12$ getfattr -n user.pvfs2.dist_params test
> # file: test
> user.pvfs2.dist_params="strips:0:16K"
> 
> c23n12$ setfattr -n "user.pvfs2.dist_params" -v 
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K" 
> test
> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K" 
> 
> 
> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
> c23n12$ getfattr -n user.pvfs2.dist_params test
> # file: test
> user.pvfs2.dist_params="strips:0:16K"
> 
> c23n12$
> 
> I tried changing PVFS_MAX_XATTR_VALUELEN in include/pvfs2-types.h from
> 256 to 512 (only, rather than 16K) - still has problems , setting the 
> "user.pvfs2.dist_params"
> param with a attribute value >256 characters works the first time you 
> try it, but no value
> ( irrespective of how ling it is) can be set on the directory 
> subsequently...
> 
> 
> $ mount | grep /var/tmp/PVFSv2/mnt
> tcp://ramones:3334/pvfs2-fs-kew on /var/tmp/PVFSv2/mnt type pvfs2 (rw)
> $ cd /var/tmp/PVFSv2/mnt
> $ mkdir test
> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
> $ getfattr test
> # file: test
> user.pvfs2.dist_name
> 
> $ setfattr -n "user.pvfs2.dist_params" -v 
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K" 
> test
> $ getfattr test
> # file: test
> user.pvfs2.dist_name
> user.pvfs2.dist_params
> 
> $ getfattr -n user.pvfs2.dist_params test
> # file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K" 
> 
> 
> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
> $ getfattr -n user.pvfs2.dist_params test
> # file: test
> user.pvfs2.dist_params="strips:0:16K"
> 
> $ setfattr -n "user.pvfs2.dist_params" -v 
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K" 
> test
> $ getfattr -n user.pvfs2.dist_params test
> # file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K" 
> 
> 
> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" testsetfattr: 
> test: Invalid argument
> $
> 
> Thanks,
> Tony
> 
> Tony Kew
> SAN Administrator
> The Center for Computational Research
> New York State Center of Excellence
> in Bioinformatics & Life Sciences
> 701 Ellicott Street, Buffalo, NY 14203
> 
> CoE Office: (716) 881-8930           Fax: (716) 849-6656
> CSE Office: (716) 645-3797 x2174
>      Cell: (716) 560-0910          Home: (716) 874-2126
> 
> "I love deadlines, I love the whooshing noise they make as they go by."
>                                                          Douglas Adams
> 
> 
> 
> Phil Carns wrote:
>> Tony Kew wrote:
>>> Dear Phil,
>>>
>>> I can't test the setfattr command on a local filesystem as far as I 
>>> can tell:
>>>
>>> $ cd /var/tmp
>>> $ mkdir test
>>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>> setfattr: test: Operation not supported
>>> $
>>>
>>> ...or is there a test I can do on an ext3 filesystem dir?
>>>
>>> I changed the value in include/pvfs2-types.h to 16K (should be plenty
>>> big enough I guess...)  & recompiled - now setfattr works the first 
>>> time,
>>> but fails subsequently..
>>> Note: testing on a single RHEL5 node, if that makes a difference 
>>
>> <cut>
>>
>>> It is not easy for me to test this code on 64 nodes.
>>> I don't know if we can go further with this without me doing so?
>>
>> Well, it looks like there are still problems just setting the 
>> attributes, so there isn't really any need to try testing it on 64 
>> nodes until that much is fixed.
>>
>> Regarding how to test on ext3, you will need to follow Emmanuel 
>> Florac's suggestion about the mount option to use for ext3.  I just 
>> want to make sure there isn't a system limit on the attribute size 
>> before trying to bump it up in PVFS.
>>
>> 16K may be too large for the request messaging system.  If you 
>> experiment with that it might be safer to pick 8K or smaller.
>>
>> -Phil

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs-8KB-xattr.patch
Type: text/x-diff
Size: 2234 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20081022/fb985236/pvfs-8KB-xattr.bin