[Pvfs2-users] Problems with xattr settings with long arguments
Tony Kew
tonykew at ccr.buffalo.edu
Wed Oct 22 15:54:57 EDT 2008
Dear Phil,
The patch looks good - I can set a 64 node config now:
e.g.
ramones$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
ramones$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
ramones$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
ramones$
It may take a little while till I can install this on the cluster & test
PVFSv2
over 64 nodes, but at least the parameter can be set :-)
Thanks,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
> Hi Tony,
>
> Would you mind trying the attached patch? It increases the maximum
> xattr value size to 8KB and also fixes a problem further down in the
> code regarding how to handle xattrs that large. The latter part of
> the patch should solve the problem that you ran into in which you
> could only set a large xattr once.
>
> thanks,
> -Phil
>
> Tony Kew wrote:
>> Dear Phil,
>>
>> I verified that this is not an issue with the underlying filesystem:
>>
>> c23n12$ mount | grep \ /scratch\
>> /dev/sda3 on /scratch type ext2 (rw)
>> c23n12$ sudo umount /scratch
>> c23n12$ sudo mount -o rw,user_xattr /dev/sda3 /scratch
>> c23n12$ mount | grep \ /scratch\
>> /dev/sda3 on /scratch type ext2 (rw,user_xattr)
>> c23n12$ cd /scratch
>> c23n12$ mkdir test
>> c23n12$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>> c23n12$ getfattr test/
>> # file: test
>> user.pvfs2.dist_name
>>
>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>> test
>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>
>>
>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>> c23n12$ getfattr -n user.pvfs2.dist_params test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K"
>>
>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>> test
>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>
>>
>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>> test
>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>
>>
>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
>> testc23n12$ getfattr -n user.pvfs2.dist_params test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K"
>>
>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>> test
>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>>
>>
>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>> c23n12$ getfattr -n user.pvfs2.dist_params test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K"
>>
>> c23n12$
>>
>> I tried changing PVFS_MAX_XATTR_VALUELEN in include/pvfs2-types.h from
>> 256 to 512 (only, rather than 16K) - still has problems , setting the
>> "user.pvfs2.dist_params"
>> param with a attribute value >256 characters works the first time you
>> try it, but no value
>> ( irrespective of how ling it is) can be set on the directory
>> subsequently...
>>
>>
>> $ mount | grep /var/tmp/PVFSv2/mnt
>> tcp://ramones:3334/pvfs2-fs-kew on /var/tmp/PVFSv2/mnt type pvfs2 (rw)
>> $ cd /var/tmp/PVFSv2/mnt
>> $ mkdir test
>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>> $ getfattr test
>> # file: test
>> user.pvfs2.dist_name
>>
>> $ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>> test
>> $ getfattr test
>> # file: test
>> user.pvfs2.dist_name
>> user.pvfs2.dist_params
>>
>> $ getfattr -n user.pvfs2.dist_params test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>
>>
>> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>> $ getfattr -n user.pvfs2.dist_params test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K"
>>
>> $ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>> test
>> $ getfattr -n user.pvfs2.dist_params test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>
>>
>> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
>> testsetfattr: test: Invalid argument
>> $
>>
>> Thanks,
>> Tony
>>
>> Tony Kew
>> SAN Administrator
>> The Center for Computational Research
>> New York State Center of Excellence
>> in Bioinformatics & Life Sciences
>> 701 Ellicott Street, Buffalo, NY 14203
>>
>> CoE Office: (716) 881-8930 Fax: (716) 849-6656
>> CSE Office: (716) 645-3797 x2174
>> Cell: (716) 560-0910 Home: (716) 874-2126
>>
>> "I love deadlines, I love the whooshing noise they make as they go by."
>> Douglas Adams
>>
>>
>>
>> Phil Carns wrote:
>>> Tony Kew wrote:
>>>> Dear Phil,
>>>>
>>>> I can't test the setfattr command on a local filesystem as far as I
>>>> can tell:
>>>>
>>>> $ cd /var/tmp
>>>> $ mkdir test
>>>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>>> setfattr: test: Operation not supported
>>>> $
>>>>
>>>> ...or is there a test I can do on an ext3 filesystem dir?
>>>>
>>>> I changed the value in include/pvfs2-types.h to 16K (should be plenty
>>>> big enough I guess...) & recompiled - now setfattr works the first
>>>> time,
>>>> but fails subsequently..
>>>> Note: testing on a single RHEL5 node, if that makes a difference
>>>
>>> <cut>
>>>
>>>> It is not easy for me to test this code on 64 nodes.
>>>> I don't know if we can go further with this without me doing so?
>>>
>>> Well, it looks like there are still problems just setting the
>>> attributes, so there isn't really any need to try testing it on 64
>>> nodes until that much is fixed.
>>>
>>> Regarding how to test on ext3, you will need to follow Emmanuel
>>> Florac's suggestion about the mount option to use for ext3. I just
>>> want to make sure there isn't a system limit on the attribute size
>>> before trying to bump it up in PVFS.
>>>
>>> 16K may be too large for the request messaging system. If you
>>> experiment with that it might be safer to pick 8K or smaller.
>>>
>>> -Phil
>
More information about the Pvfs2-users
mailing list