[Pvfs2-users] Problems with xattr settings with long arguments
Tony Kew
tonykew at ccr.buffalo.edu
Wed Oct 22 17:29:13 EDT 2008
Dear Phil,
The patch works under Red Hat Enterprise Linux 5.2, but not under
RHEL 4 update 5, which doesn't have DB_BUFFER_SMALL in
/usr/include/db4/db.h
From RHEL5.2 /usr/include/db4/db.h
/*
* DB (user visible) error return codes.
*
* !!!
* For source compatibility with DB 2.X deadlock return (EAGAIN), use the
* following:
* #include <errno.h>
* #define DB_LOCK_DEADLOCK EAGAIN
*
* !!!
* We don't want our error returns to conflict with other packages where
* possible, so pick a base error value that's hopefully not common. We
* document that we own the error name space from -30,800 to -30,999.
*/
/* DB (public) error return codes. */
#define DB_BUFFER_SMALL (-30999)/* User memory too small for
return. */
#define DB_DONOTINDEX (-30998)/* "Null" return from 2ndary
callbk. */
#define DB_KEYEMPTY (-30997)/* Key/data deleted or never
created. */
#define DB_KEYEXIST (-30996)/* The key/data pair already
exists. */
#define DB_LOCK_DEADLOCK (-30995)/* Deadlock. */
#define DB_LOCK_NOTGRANTED (-30994)/* Lock unavailable. */
#define DB_LOG_BUFFER_FULL (-30993)/* In-memory log buffer full. */
#define DB_NOSERVER (-30992)/* Server panic return. */
#define DB_NOSERVER_HOME (-30991)/* Bad home sent to server. */
#define DB_NOSERVER_ID (-30990)/* Bad ID sent to server. */
#define DB_NOTFOUND (-30989)/* Key/data pair not found (EOF). */
#define DB_OLD_VERSION (-30988)/* Out-of-date version. */
#define DB_PAGE_NOTFOUND (-30987)/* Requested page not found. */
#define DB_REP_DUPMASTER (-30986)/* There are two masters. */
#define DB_REP_HANDLE_DEAD (-30985)/* Rolled back a commit. */
#define DB_REP_HOLDELECTION (-30984)/* Time to hold an election. */
#define DB_REP_ISPERM (-30983)/* Cached not written perm
written.*/
#define DB_REP_NEWMASTER (-30982)/* We have learned of a new
master. */
#define DB_REP_NEWSITE (-30981)/* New site entered system. */
#define DB_REP_NOTPERM (-30980)/* Permanent log record not
written. */
#define DB_REP_STARTUPDONE (-30979)/* Client startup complete. */
#define DB_REP_UNAVAIL (-30978)/* Site cannot currently be
reached. */
#define DB_RUNRECOVERY (-30977)/* Panic return. */
#define DB_SECONDARY_BAD (-30976)/* Secondary index corrupt. */
#define DB_VERIFY_BAD (-30975)/* Verify failed; bad format. */
#define DB_VERSION_MISMATCH (-30974)/* Environment version mismatch. */
from RHEL 4 update 5 /usr/include/db4/db.h
/*
* DB (user visible) error return codes.
*
* !!!
* For source compatibility with DB 2.X deadlock return (EAGAIN), use the
* following:
* #include <errno.h>
* #define DB_LOCK_DEADLOCK EAGAIN
*
* !!!
* We don't want our error returns to conflict with other packages where
* possible, so pick a base error value that's hopefully not common. We
* document that we own the error name space from -30,800 to -30,999.
*/
/* DB (public) error return codes. */
#define DB_DONOTINDEX (-30999)/* "Null" return from 2ndary
callbk. */
#define DB_FILEOPEN (-30998)/* Rename/remove while file is
open. */
#define DB_KEYEMPTY (-30997)/* Key/data deleted or never
created. */
#define DB_KEYEXIST (-30996)/* The key/data pair already
exists. */
#define DB_LOCK_DEADLOCK (-30995)/* Deadlock. */
#define DB_LOCK_NOTGRANTED (-30994)/* Lock unavailable. */
#define DB_NOSERVER (-30993)/* Server panic return. */
#define DB_NOSERVER_HOME (-30992)/* Bad home sent to server. */
#define DB_NOSERVER_ID (-30991)/* Bad ID sent to server. */
#define DB_NOTFOUND (-30990)/* Key/data pair not found (EOF). */
#define DB_OLD_VERSION (-30989)/* Out-of-date version. */
#define DB_PAGE_NOTFOUND (-30988)/* Requested page not found. */
#define DB_REP_DUPMASTER (-30987)/* There are two masters. */
#define DB_REP_HANDLE_DEAD (-30986)/* Rolled back a commit. */
#define DB_REP_HOLDELECTION (-30985)/* Time to hold an election. */
#define DB_REP_ISPERM (-30984)/* Cached not written perm
written.*/
#define DB_REP_NEWMASTER (-30983)/* We have learned of a new
master. */
#define DB_REP_NEWSITE (-30982)/* New site entered system. */
#define DB_REP_NOTPERM (-30981)/* Permanent log record not
written. */
#define DB_REP_OUTDATED (-30980)/* Site is too far behind master. */
#define DB_REP_UNAVAIL (-30979)/* Site cannot currently be
reached. */
#define DB_RUNRECOVERY (-30978)/* Panic return. */
#define DB_SECONDARY_BAD (-30977)/* Secondary index corrupt. */
#define DB_VERIFY_BAD (-30976)/* Verify failed; bad format. */
Ideas?
Thanks Much,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Tony Kew wrote:
> Dear Phil,
>
> The patch looks good - I can set a 64 node config now:
> e.g.
>
> ramones$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
> ramones$ setfattr -n "user.pvfs2.dist_params" -v
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
> test
> ramones$ getfattr -n "user.pvfs2.dist_params" test
> # file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>
>
> ramones$
>
> It may take a little while till I can install this on the cluster &
> test PVFSv2
> over 64 nodes, but at least the parameter can be set :-)
>
> Thanks,
> Tony
>
> Tony Kew
> SAN Administrator
> The Center for Computational Research
> New York State Center of Excellence
> in Bioinformatics & Life Sciences
> 701 Ellicott Street, Buffalo, NY 14203
>
> CoE Office: (716) 881-8930 Fax: (716) 849-6656
> CSE Office: (716) 645-3797 x2174
> Cell: (716) 560-0910 Home: (716) 874-2126
>
> "I love deadlines, I love the whooshing noise they make as they go by."
> Douglas Adams
>
>
>
> Phil Carns wrote:
>> Hi Tony,
>>
>> Would you mind trying the attached patch? It increases the maximum
>> xattr value size to 8KB and also fixes a problem further down in the
>> code regarding how to handle xattrs that large. The latter part of
>> the patch should solve the problem that you ran into in which you
>> could only set a large xattr once.
>>
>> thanks,
>> -Phil
>>
>> Tony Kew wrote:
>>> Dear Phil,
>>>
>>> I verified that this is not an issue with the underlying filesystem:
>>>
>>> c23n12$ mount | grep \ /scratch\
>>> /dev/sda3 on /scratch type ext2 (rw)
>>> c23n12$ sudo umount /scratch
>>> c23n12$ sudo mount -o rw,user_xattr /dev/sda3 /scratch
>>> c23n12$ mount | grep \ /scratch\
>>> /dev/sda3 on /scratch type ext2 (rw,user_xattr)
>>> c23n12$ cd /scratch
>>> c23n12$ mkdir test
>>> c23n12$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>> c23n12$ getfattr test/
>>> # file: test
>>> user.pvfs2.dist_name
>>>
>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>> test
>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>>
>>>
>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>>> c23n12$ getfattr -n user.pvfs2.dist_params test
>>> # file: test
>>> user.pvfs2.dist_params="strips:0:16K"
>>>
>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>> test
>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>
>>>
>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>> test
>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>
>>>
>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
>>> testc23n12$ getfattr -n user.pvfs2.dist_params test
>>> # file: test
>>> user.pvfs2.dist_params="strips:0:16K"
>>>
>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>>> test
>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>>>
>>>
>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>>> c23n12$ getfattr -n user.pvfs2.dist_params test
>>> # file: test
>>> user.pvfs2.dist_params="strips:0:16K"
>>>
>>> c23n12$
>>>
>>> I tried changing PVFS_MAX_XATTR_VALUELEN in include/pvfs2-types.h from
>>> 256 to 512 (only, rather than 16K) - still has problems , setting
>>> the "user.pvfs2.dist_params"
>>> param with a attribute value >256 characters works the first time
>>> you try it, but no value
>>> ( irrespective of how ling it is) can be set on the directory
>>> subsequently...
>>>
>>>
>>> $ mount | grep /var/tmp/PVFSv2/mnt
>>> tcp://ramones:3334/pvfs2-fs-kew on /var/tmp/PVFSv2/mnt type pvfs2 (rw)
>>> $ cd /var/tmp/PVFSv2/mnt
>>> $ mkdir test
>>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>> $ getfattr test
>>> # file: test
>>> user.pvfs2.dist_name
>>>
>>> $ setfattr -n "user.pvfs2.dist_params" -v
>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>> test
>>> $ getfattr test
>>> # file: test
>>> user.pvfs2.dist_name
>>> user.pvfs2.dist_params
>>>
>>> $ getfattr -n user.pvfs2.dist_params test
>>> # file: test
>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>>
>>>
>>> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>>> $ getfattr -n user.pvfs2.dist_params test
>>> # file: test
>>> user.pvfs2.dist_params="strips:0:16K"
>>>
>>> $ setfattr -n "user.pvfs2.dist_params" -v
>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>> test
>>> $ getfattr -n user.pvfs2.dist_params test
>>> # file: test
>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>
>>>
>>> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
>>> testsetfattr: test: Invalid argument
>>> $
>>>
>>> Thanks,
>>> Tony
>>>
>>> Tony Kew
>>> SAN Administrator
>>> The Center for Computational Research
>>> New York State Center of Excellence
>>> in Bioinformatics & Life Sciences
>>> 701 Ellicott Street, Buffalo, NY 14203
>>>
>>> CoE Office: (716) 881-8930 Fax: (716) 849-6656
>>> CSE Office: (716) 645-3797 x2174
>>> Cell: (716) 560-0910 Home: (716) 874-2126
>>>
>>> "I love deadlines, I love the whooshing noise they make as they go by."
>>> Douglas Adams
>>>
>>>
>>>
>>> Phil Carns wrote:
>>>> Tony Kew wrote:
>>>>> Dear Phil,
>>>>>
>>>>> I can't test the setfattr command on a local filesystem as far as
>>>>> I can tell:
>>>>>
>>>>> $ cd /var/tmp
>>>>> $ mkdir test
>>>>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>>>> setfattr: test: Operation not supported
>>>>> $
>>>>>
>>>>> ...or is there a test I can do on an ext3 filesystem dir?
>>>>>
>>>>> I changed the value in include/pvfs2-types.h to 16K (should be plenty
>>>>> big enough I guess...) & recompiled - now setfattr works the
>>>>> first time,
>>>>> but fails subsequently..
>>>>> Note: testing on a single RHEL5 node, if that makes a difference
>>>>
>>>> <cut>
>>>>
>>>>> It is not easy for me to test this code on 64 nodes.
>>>>> I don't know if we can go further with this without me doing so?
>>>>
>>>> Well, it looks like there are still problems just setting the
>>>> attributes, so there isn't really any need to try testing it on 64
>>>> nodes until that much is fixed.
>>>>
>>>> Regarding how to test on ext3, you will need to follow Emmanuel
>>>> Florac's suggestion about the mount option to use for ext3. I just
>>>> want to make sure there isn't a system limit on the attribute size
>>>> before trying to bump it up in PVFS.
>>>>
>>>> 16K may be too large for the request messaging system. If you
>>>> experiment with that it might be safer to pick 8K or smaller.
>>>>
>>>> -Phil
>>
>
More information about the Pvfs2-users
mailing list