[Pvfs2-users] Problems with xattr settings with long arguments
Phil Carns
carns at mcs.anl.gov
Wed Oct 22 17:46:53 EDT 2008
Whoops, thanks for catching that. This additional patch should fix it.
thanks,
-Phil
Tony Kew wrote:
> Dear Phil,
>
> The patch works under Red Hat Enterprise Linux 5.2, but not under
> RHEL 4 update 5, which doesn't have DB_BUFFER_SMALL in
> /usr/include/db4/db.h
>
> From RHEL5.2 /usr/include/db4/db.h
>
> /*
> * DB (user visible) error return codes.
> *
> * !!!
> * For source compatibility with DB 2.X deadlock return (EAGAIN), use the
> * following:
> * #include <errno.h>
> * #define DB_LOCK_DEADLOCK EAGAIN
> *
> * !!!
> * We don't want our error returns to conflict with other packages where
> * possible, so pick a base error value that's hopefully not common. We
> * document that we own the error name space from -30,800 to -30,999.
> */
> /* DB (public) error return codes. */
> #define DB_BUFFER_SMALL (-30999)/* User memory too small for
> return. */
> #define DB_DONOTINDEX (-30998)/* "Null" return from 2ndary
> callbk. */
> #define DB_KEYEMPTY (-30997)/* Key/data deleted or never
> created. */
> #define DB_KEYEXIST (-30996)/* The key/data pair already
> exists. */
> #define DB_LOCK_DEADLOCK (-30995)/* Deadlock. */
> #define DB_LOCK_NOTGRANTED (-30994)/* Lock unavailable. */
> #define DB_LOG_BUFFER_FULL (-30993)/* In-memory log buffer full. */
> #define DB_NOSERVER (-30992)/* Server panic return. */
> #define DB_NOSERVER_HOME (-30991)/* Bad home sent to server. */
> #define DB_NOSERVER_ID (-30990)/* Bad ID sent to server. */
> #define DB_NOTFOUND (-30989)/* Key/data pair not found
> (EOF). */
> #define DB_OLD_VERSION (-30988)/* Out-of-date version. */
> #define DB_PAGE_NOTFOUND (-30987)/* Requested page not found. */
> #define DB_REP_DUPMASTER (-30986)/* There are two masters. */
> #define DB_REP_HANDLE_DEAD (-30985)/* Rolled back a commit. */
> #define DB_REP_HOLDELECTION (-30984)/* Time to hold an election. */
> #define DB_REP_ISPERM (-30983)/* Cached not written perm
> written.*/
> #define DB_REP_NEWMASTER (-30982)/* We have learned of a new
> master. */
> #define DB_REP_NEWSITE (-30981)/* New site entered system. */
> #define DB_REP_NOTPERM (-30980)/* Permanent log record not
> written. */
> #define DB_REP_STARTUPDONE (-30979)/* Client startup complete. */
> #define DB_REP_UNAVAIL (-30978)/* Site cannot currently be
> reached. */
> #define DB_RUNRECOVERY (-30977)/* Panic return. */
> #define DB_SECONDARY_BAD (-30976)/* Secondary index corrupt. */
> #define DB_VERIFY_BAD (-30975)/* Verify failed; bad format. */
> #define DB_VERSION_MISMATCH (-30974)/* Environment version mismatch. */
>
>
> from RHEL 4 update 5 /usr/include/db4/db.h
>
> /*
> * DB (user visible) error return codes.
> *
> * !!!
> * For source compatibility with DB 2.X deadlock return (EAGAIN), use the
> * following:
> * #include <errno.h>
> * #define DB_LOCK_DEADLOCK EAGAIN
> *
> * !!!
> * We don't want our error returns to conflict with other packages where
> * possible, so pick a base error value that's hopefully not common. We
> * document that we own the error name space from -30,800 to -30,999.
> */
> /* DB (public) error return codes. */
> #define DB_DONOTINDEX (-30999)/* "Null" return from 2ndary
> callbk. */
> #define DB_FILEOPEN (-30998)/* Rename/remove while file is
> open. */
> #define DB_KEYEMPTY (-30997)/* Key/data deleted or never
> created. */
> #define DB_KEYEXIST (-30996)/* The key/data pair already
> exists. */
> #define DB_LOCK_DEADLOCK (-30995)/* Deadlock. */
> #define DB_LOCK_NOTGRANTED (-30994)/* Lock unavailable. */
> #define DB_NOSERVER (-30993)/* Server panic return. */
> #define DB_NOSERVER_HOME (-30992)/* Bad home sent to server. */
> #define DB_NOSERVER_ID (-30991)/* Bad ID sent to server. */
> #define DB_NOTFOUND (-30990)/* Key/data pair not found
> (EOF). */
> #define DB_OLD_VERSION (-30989)/* Out-of-date version. */
> #define DB_PAGE_NOTFOUND (-30988)/* Requested page not found. */
> #define DB_REP_DUPMASTER (-30987)/* There are two masters. */
> #define DB_REP_HANDLE_DEAD (-30986)/* Rolled back a commit. */
> #define DB_REP_HOLDELECTION (-30985)/* Time to hold an election. */
> #define DB_REP_ISPERM (-30984)/* Cached not written perm
> written.*/
> #define DB_REP_NEWMASTER (-30983)/* We have learned of a new
> master. */
> #define DB_REP_NEWSITE (-30982)/* New site entered system. */
> #define DB_REP_NOTPERM (-30981)/* Permanent log record not
> written. */
> #define DB_REP_OUTDATED (-30980)/* Site is too far behind
> master. */
> #define DB_REP_UNAVAIL (-30979)/* Site cannot currently be
> reached. */
> #define DB_RUNRECOVERY (-30978)/* Panic return. */
> #define DB_SECONDARY_BAD (-30977)/* Secondary index corrupt. */
> #define DB_VERIFY_BAD (-30976)/* Verify failed; bad format. */
>
>
> Ideas?
>
> Thanks Much,
> Tony
>
>
> Tony Kew
> SAN Administrator
> The Center for Computational Research
> New York State Center of Excellence
> in Bioinformatics & Life Sciences
> 701 Ellicott Street, Buffalo, NY 14203
>
> CoE Office: (716) 881-8930 Fax: (716) 849-6656
> CSE Office: (716) 645-3797 x2174
> Cell: (716) 560-0910 Home: (716) 874-2126
>
> "I love deadlines, I love the whooshing noise they make as they go by."
> Douglas Adams
>
>
>
> Tony Kew wrote:
>> Dear Phil,
>>
>> The patch looks good - I can set a 64 node config now:
>> e.g.
>>
>> ramones$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>> ramones$ setfattr -n "user.pvfs2.dist_params" -v
>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>> test
>> ramones$ getfattr -n "user.pvfs2.dist_params" test
>> # file: test
>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>>
>>
>> ramones$
>>
>> It may take a little while till I can install this on the cluster &
>> test PVFSv2
>> over 64 nodes, but at least the parameter can be set :-)
>>
>> Thanks,
>> Tony
>>
>> Tony Kew
>> SAN Administrator
>> The Center for Computational Research
>> New York State Center of Excellence
>> in Bioinformatics & Life Sciences
>> 701 Ellicott Street, Buffalo, NY 14203
>>
>> CoE Office: (716) 881-8930 Fax: (716) 849-6656
>> CSE Office: (716) 645-3797 x2174
>> Cell: (716) 560-0910 Home: (716) 874-2126
>>
>> "I love deadlines, I love the whooshing noise they make as they go by."
>> Douglas Adams
>>
>>
>>
>> Phil Carns wrote:
>>> Hi Tony,
>>>
>>> Would you mind trying the attached patch? It increases the maximum
>>> xattr value size to 8KB and also fixes a problem further down in the
>>> code regarding how to handle xattrs that large. The latter part of
>>> the patch should solve the problem that you ran into in which you
>>> could only set a large xattr once.
>>>
>>> thanks,
>>> -Phil
>>>
>>> Tony Kew wrote:
>>>> Dear Phil,
>>>>
>>>> I verified that this is not an issue with the underlying filesystem:
>>>>
>>>> c23n12$ mount | grep \ /scratch\
>>>> /dev/sda3 on /scratch type ext2 (rw)
>>>> c23n12$ sudo umount /scratch
>>>> c23n12$ sudo mount -o rw,user_xattr /dev/sda3 /scratch
>>>> c23n12$ mount | grep \ /scratch\
>>>> /dev/sda3 on /scratch type ext2 (rw,user_xattr)
>>>> c23n12$ cd /scratch
>>>> c23n12$ mkdir test
>>>> c23n12$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>>> c23n12$ getfattr test/
>>>> # file: test
>>>> user.pvfs2.dist_name
>>>>
>>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>>> test
>>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>>>
>>>>
>>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>>>> c23n12$ getfattr -n user.pvfs2.dist_params test
>>>> # file: test
>>>> user.pvfs2.dist_params="strips:0:16K"
>>>>
>>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>> test
>>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>>
>>>>
>>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>> test
>>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>>
>>>>
>>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
>>>> testc23n12$ getfattr -n user.pvfs2.dist_params test
>>>> # file: test
>>>> user.pvfs2.dist_params="strips:0:16K"
>>>>
>>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v
>>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>>>> test
>>>> c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
>>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
>>>>
>>>>
>>>> c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>>>> c23n12$ getfattr -n user.pvfs2.dist_params test
>>>> # file: test
>>>> user.pvfs2.dist_params="strips:0:16K"
>>>>
>>>> c23n12$
>>>>
>>>> I tried changing PVFS_MAX_XATTR_VALUELEN in include/pvfs2-types.h from
>>>> 256 to 512 (only, rather than 16K) - still has problems , setting
>>>> the "user.pvfs2.dist_params"
>>>> param with a attribute value >256 characters works the first time
>>>> you try it, but no value
>>>> ( irrespective of how ling it is) can be set on the directory
>>>> subsequently...
>>>>
>>>>
>>>> $ mount | grep /var/tmp/PVFSv2/mnt
>>>> tcp://ramones:3334/pvfs2-fs-kew on /var/tmp/PVFSv2/mnt type pvfs2 (rw)
>>>> $ cd /var/tmp/PVFSv2/mnt
>>>> $ mkdir test
>>>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>>> $ getfattr test
>>>> # file: test
>>>> user.pvfs2.dist_name
>>>>
>>>> $ setfattr -n "user.pvfs2.dist_params" -v
>>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>>> test
>>>> $ getfattr test
>>>> # file: test
>>>> user.pvfs2.dist_name
>>>> user.pvfs2.dist_params
>>>>
>>>> $ getfattr -n user.pvfs2.dist_params test
>>>> # file: test
>>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>>>>
>>>>
>>>> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
>>>> $ getfattr -n user.pvfs2.dist_params test
>>>> # file: test
>>>> user.pvfs2.dist_params="strips:0:16K"
>>>>
>>>> $ setfattr -n "user.pvfs2.dist_params" -v
>>>> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>> test
>>>> $ getfattr -n user.pvfs2.dist_params test
>>>> # file: test
>>>> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
>>>>
>>>>
>>>> $ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
>>>> testsetfattr: test: Invalid argument
>>>> $
>>>>
>>>> Thanks,
>>>> Tony
>>>>
>>>> Tony Kew
>>>> SAN Administrator
>>>> The Center for Computational Research
>>>> New York State Center of Excellence
>>>> in Bioinformatics & Life Sciences
>>>> 701 Ellicott Street, Buffalo, NY 14203
>>>>
>>>> CoE Office: (716) 881-8930 Fax: (716) 849-6656
>>>> CSE Office: (716) 645-3797 x2174
>>>> Cell: (716) 560-0910 Home: (716) 874-2126
>>>>
>>>> "I love deadlines, I love the whooshing noise they make as they go by."
>>>> Douglas Adams
>>>>
>>>>
>>>>
>>>> Phil Carns wrote:
>>>>> Tony Kew wrote:
>>>>>> Dear Phil,
>>>>>>
>>>>>> I can't test the setfattr command on a local filesystem as far as
>>>>>> I can tell:
>>>>>>
>>>>>> $ cd /var/tmp
>>>>>> $ mkdir test
>>>>>> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
>>>>>> setfattr: test: Operation not supported
>>>>>> $
>>>>>>
>>>>>> ...or is there a test I can do on an ext3 filesystem dir?
>>>>>>
>>>>>> I changed the value in include/pvfs2-types.h to 16K (should be plenty
>>>>>> big enough I guess...) & recompiled - now setfattr works the
>>>>>> first time,
>>>>>> but fails subsequently..
>>>>>> Note: testing on a single RHEL5 node, if that makes a difference
>>>>>
>>>>> <cut>
>>>>>
>>>>>> It is not easy for me to test this code on 64 nodes.
>>>>>> I don't know if we can go further with this without me doing so?
>>>>>
>>>>> Well, it looks like there are still problems just setting the
>>>>> attributes, so there isn't really any need to try testing it on 64
>>>>> nodes until that much is fixed.
>>>>>
>>>>> Regarding how to test on ext3, you will need to follow Emmanuel
>>>>> Florac's suggestion about the mount option to use for ext3. I just
>>>>> want to make sure there isn't a system limit on the attribute size
>>>>> before trying to bump it up in PVFS.
>>>>>
>>>>> 16K may be too large for the request messaging system. If you
>>>>> experiment with that it might be safer to pick 8K or smaller.
>>>>>
>>>>> -Phil
>>>
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs-8KB-xattr-fix.patch
Type: text/x-diff
Size: 1046 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20081022/3bee7104/pvfs-8KB-xattr-fix-0001.bin
More information about the Pvfs2-users
mailing list