[Pvfs2-users] Problems with xattr settings with long arguments
Tony Kew
tonykew at ccr.buffalo.edu
Thu Oct 9 14:53:12 EDT 2008
Dear Phil,
I can't test the setfattr command on a local filesystem as far as I can
tell:
$ cd /var/tmp
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
setfattr: test: Operation not supported
$
...or is there a test I can do on an ext3 filesystem dir?
I changed the value in include/pvfs2-types.h to 16K (should be plenty
big enough I guess...) & recompiled - now setfattr works the first time,
but fails subsequently..
Note: testing on a single RHEL5 node, if that makes a difference
$ mkdir test
$ getfattr test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr test
# file: test
user.pvfs2.dist_name
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
$ getfattr test
# file: test
user.pvfs2.dist_name
user.pvfs2.dist_params
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
setfattr: test: Invalid argument
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
setfattr: test: Invalid argument
$
I cannot write anything in the directory (though note, the configuration is
invalid for a single node, so that may be reasonable)
$ cd test/
$ dd if=/dev/urandom of=testfile bs=16k count=16
dd: opening `testfile': Invalid argument
$ cd ..
At this point the only way to fix the settings seems to be to remove
and recreate the directory
"dd" does work with a valid single node configuration, however:
$ rm -rf test/
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
$ getfattr test
# file: test
user.pvfs2.dist_name
user.pvfs2.dist_params
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K"
$ cd test
$ dd if=/dev/urandom of=testfile bs=16k count=16
16+0 records in
16+0 records out
262144 bytes (262 kB) copied, 0.162831 seconds, 1.6 MB/s
$ ls -l testfile
-rw-r--r-- 1 tonykew ccrstaff 262144 Oct 9 14:47 testfile
$ cd ..
$
It is not easy for me to test this code on 64 nodes.
I don't know if we can go further with this without me doing so?
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
> Hi Tony,
>
> Looking at the length of those strings, it looks like the problem
> starts when the xattr value is 256 characters. Digging around in the
> code it looks like we have a hard coded limit on this in PVFS (from
> pvfs2-types.h):
>
> #define PVFS_MAX_XATTR_VALUELEN 256
>
> I don't know if there is a good reason for this limit or if it is
> arbitrary. Could you test and see if you can set an extended
> attribute with that value on a local file system on your box?
>
> I don't think we have had anyone try to specify varstrip parameters
> for that many datafiles before :)
>
> thanks,
> -Phil
>
> Tony Kew wrote:
>> Dear All,
>>
>> I'm running pvfs 7.2.1 with Murali Vilayannur xarrts patch from this
>> message thread:
>> http://osdir.com/ml/file-systems.pvfs2.users/2007-01/msg00050.html
>> (which, if memory serves, Phil Carns put into CVS)
>>
>> I have a PVFS filesystem over 64 nodes, mounted on all those nodes:
>>
>> $ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt
>> [...]
>> (6) Verifying that fsid 1076995 is acceptable to all servers...
>>
>> Ok; all servers understand fs_id 1076995
>>
>> (7) Verifying that root handle is owned by one server...
>>
>> Root handle: 1048576
>> Ok; root handle is owned by exactly one server.
>>
>> =============================================================
>>
>> The PVFS2 filesystem at /scratch/1076995.bono.ccr.buffalo.edu/mnt
>> appears to be correctly configured.
>>
>> $
>>
>> All nodes 64 nodes are up:
>>
>> $ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt | sed -n
>> '/^(5)/,/^(6)/p' | sed -n '/data servers/,/^$/p' | sed 1d | sed '$d'
>> | wc -l
>> 64
>> $
>>
>> I'm trying to create a varstrip_dist dir with (effectively) a stripe
>> across all 64 nodes,
>> with 16K on each node, but the setfattr fails for 37 (or more) nodes:
>>
>> $ mount | grep pvfs2
>> tcp://c27n20:3334/pvfs2-fs-1076995.bono.ccr.buffalo.edu on
>> /scratch/1076995.bono.ccr.buffalo.edu/mnt type pvfs2 (rw)
>> $ cd /scratch/1076995.bono.ccr.buffalo.edu/mnt
>> $ mkdir test
>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>> $ getfattr test
>> # file: test
>> user.pvfs2.dist_name
>>
>> $ getfattr -n "user.pvfs2.dist_name" test
>> # file: test
>> user.pvfs2.dist_name="varstrip_dist"
>>
>> $ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>> test
>> $ getfattr -n "user.pvfs2.dist_params" test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>
>>
>> $ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>> test
>> setfattr: test: Invalid argument
>> $ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>> test
>> setfattr: test: Invalid argument
>> $
>>
>>
>> Sorry for the over long lines (I made sure its not a problem with
>> line wrapping).
>>
>> Since the error is from setfattr I wonder if this is a RedHat problem...
>> I'm running RedHat Enterprise Linux AS v.4 update 5 on these nodes, and
>> the attr package (which includes setfattr) is the latest version from
>> RH:
>>
>> $ rpm -qf `type -p setfattr`
>> attr-2.4.16-3.1.el4
>> $
>>
>>
>> Thanks Much,
>> Tony
>>
>> Tony Kew
>> SAN Administrator
>> The Center for Computational Research
>> New York State Center of Excellence
>> in Bioinformatics & Life Sciences
>> 701 Ellicott Street, Buffalo, NY 14203
>>
>> CoE Office: (716) 881-8930 Fax: (716) 849-6656
>> CSE Office: (716) 645-3797 x2174
>> Cell: (716) 560-0910 Home: (716) 874-2126
>>
>> "I love deadlines, I love the whooshing noise they make as they go by."
>> Douglas Adams
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
More information about the Pvfs2-users
mailing list