[Pvfs2-users] Problems with xattr settings with long arguments
Phil Carns
carns at mcs.anl.gov
Thu Oct 9 12:10:25 EDT 2008
Hi Tony,
Looking at the length of those strings, it looks like the problem starts
when the xattr value is 256 characters. Digging around in the code it
looks like we have a hard coded limit on this in PVFS (from pvfs2-types.h):
#define PVFS_MAX_XATTR_VALUELEN 256
I don't know if there is a good reason for this limit or if it is
arbitrary. Could you test and see if you can set an extended attribute
with that value on a local file system on your box?
I don't think we have had anyone try to specify varstrip parameters for
that many datafiles before :)
thanks,
-Phil
Tony Kew wrote:
> Dear All,
>
> I'm running pvfs 7.2.1 with Murali Vilayannur xarrts patch from this
> message thread:
> http://osdir.com/ml/file-systems.pvfs2.users/2007-01/msg00050.html
> (which, if memory serves, Phil Carns put into CVS)
>
> I have a PVFS filesystem over 64 nodes, mounted on all those nodes:
>
> $ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt
>
> (1) Parsing tab file...
>
> (2) Initializing system interface...
>
> (3) Initializing each file system found in tab file: /etc/mtab...
>
> PVFS2 servers: tcp://c27n20:3334
> Storage name: pvfs2-fs-1076995.bono.ccr.buffalo.edu
> Local mount point: /scratch/1076995.bono.ccr.buffalo.edu/mnt
> /scratch/1076995.bono.ccr.buffalo.edu/mnt: Ok
>
> (4) Searching for /scratch/1076995.bono.ccr.buffalo.edu/mnt in pvfstab...
>
> PVFS2 servers: tcp://c27n20:3334
> Storage name: pvfs2-fs-1076995.bono.ccr.buffalo.edu
> Local mount point: /scratch/1076995.bono.ccr.buffalo.edu/mnt
>
> meta servers:
> tcp://c25n17:3334
> tcp://c25n18:3334
> tcp://c25n19:3334
> tcp://c25n20:3334
> tcp://c25n21:3334
> tcp://c25n22:3334
> tcp://c25n23:3334
> tcp://c25n24:3334
> tcp://c25n25:3334
> tcp://c25n26:3334
> tcp://c25n27:3334
> tcp://c25n28:3334
> tcp://c26n01:3334
> tcp://c26n02:3334
> tcp://c26n03:3334
> tcp://c26n04:3334
> tcp://c26n05:3334
> tcp://c26n06:3334
> tcp://c26n07:3334
> tcp://c26n08:3334
> tcp://c26n09:3334
> tcp://c26n10:3334
> tcp://c26n11:3334
> tcp://c26n12:3334
> tcp://c26n13:3334
> tcp://c26n14:3334
> tcp://c26n15:3334
> tcp://c26n16:3334
> tcp://c26n17:3334
> tcp://c26n18:3334
> tcp://c26n19:3334
> tcp://c26n20:3334
> tcp://c26n21:3334
> tcp://c26n22:3334
> tcp://c26n23:3334
> tcp://c26n24:3334
> tcp://c26n25:3334
> tcp://c26n26:3334
> tcp://c26n27:3334
> tcp://c26n28:3334
> tcp://c26n29:3334
> tcp://c26n30:3334
> tcp://c26n31:3334
> tcp://c26n32:3334
> tcp://c27n01:3334
> tcp://c27n02:3334
> tcp://c27n03:3334
> tcp://c27n04:3334
> tcp://c27n05:3334
> tcp://c27n06:3334
> tcp://c27n07:3334
> tcp://c27n08:3334
> tcp://c27n09:3334
> tcp://c27n10:3334
> tcp://c27n11:3334
> tcp://c27n12:3334
> tcp://c27n13:3334
> tcp://c27n14:3334
> tcp://c27n15:3334
> tcp://c27n16:3334
> tcp://c27n17:3334
> tcp://c27n18:3334
> tcp://c27n19:3334
> tcp://c27n20:3334
>
> data servers:
> tcp://c25n17:3334
> tcp://c25n18:3334
> tcp://c25n19:3334
> tcp://c25n20:3334
> tcp://c25n21:3334
> tcp://c25n22:3334
> tcp://c25n23:3334
> tcp://c25n24:3334
> tcp://c25n25:3334
> tcp://c25n26:3334
> tcp://c25n27:3334
> tcp://c25n28:3334
> tcp://c26n01:3334
> tcp://c26n02:3334
> tcp://c26n03:3334
> tcp://c26n04:3334
> tcp://c26n05:3334
> tcp://c26n06:3334
> tcp://c26n07:3334
> tcp://c26n08:3334
> tcp://c26n09:3334
> tcp://c26n10:3334
> tcp://c26n11:3334
> tcp://c26n12:3334
> tcp://c26n13:3334
> tcp://c26n14:3334
> tcp://c26n15:3334
> tcp://c26n16:3334
> tcp://c26n17:3334
> tcp://c26n18:3334
> tcp://c26n19:3334
> tcp://c26n20:3334
> tcp://c26n21:3334
> tcp://c26n22:3334
> tcp://c26n23:3334
> tcp://c26n24:3334
> tcp://c26n25:3334
> tcp://c26n26:3334
> tcp://c26n27:3334
> tcp://c26n28:3334
> tcp://c26n29:3334
> tcp://c26n30:3334
> tcp://c26n31:3334
> tcp://c26n32:3334
> tcp://c27n01:3334
> tcp://c27n02:3334
> tcp://c27n03:3334
> tcp://c27n04:3334
> tcp://c27n05:3334
> tcp://c27n06:3334
> tcp://c27n07:3334
> tcp://c27n08:3334
> tcp://c27n09:3334
> tcp://c27n10:3334
> tcp://c27n11:3334
> tcp://c27n12:3334
> tcp://c27n13:3334
> tcp://c27n14:3334
> tcp://c27n15:3334
> tcp://c27n16:3334
> tcp://c27n17:3334
> tcp://c27n18:3334
> tcp://c27n19:3334
> tcp://c27n20:3334
>
> (5) Verifying that all servers are responding...
>
> meta servers:
> tcp://c25n17:3334 Ok
> tcp://c25n18:3334 Ok
> tcp://c25n19:3334 Ok
> tcp://c25n20:3334 Ok
> tcp://c25n21:3334 Ok
> tcp://c25n22:3334 Ok
> tcp://c25n23:3334 Ok
> tcp://c25n24:3334 Ok
> tcp://c25n25:3334 Ok
> tcp://c25n26:3334 Ok
> tcp://c25n27:3334 Ok
> tcp://c25n28:3334 Ok
> tcp://c26n01:3334 Ok
> tcp://c26n02:3334 Ok
> tcp://c26n03:3334 Ok
> tcp://c26n04:3334 Ok
> tcp://c26n05:3334 Ok
> tcp://c26n06:3334 Ok
> tcp://c26n07:3334 Ok
> tcp://c26n08:3334 Ok
> tcp://c26n09:3334 Ok
> tcp://c26n10:3334 Ok
> tcp://c26n11:3334 Ok
> tcp://c26n12:3334 Ok
> tcp://c26n13:3334 Ok
> tcp://c26n14:3334 Ok
> tcp://c26n15:3334 Ok
> tcp://c26n16:3334 Ok
> tcp://c26n17:3334 Ok
> tcp://c26n18:3334 Ok
> tcp://c26n19:3334 Ok
> tcp://c26n20:3334 Ok
> tcp://c26n21:3334 Ok
> tcp://c26n22:3334 Ok
> tcp://c26n23:3334 Ok
> tcp://c26n24:3334 Ok
> tcp://c26n25:3334 Ok
> tcp://c26n26:3334 Ok
> tcp://c26n27:3334 Ok
> tcp://c26n28:3334 Ok
> tcp://c26n29:3334 Ok
> tcp://c26n30:3334 Ok
> tcp://c26n31:3334 Ok
> tcp://c26n32:3334 Ok
> tcp://c27n01:3334 Ok
> tcp://c27n02:3334 Ok
> tcp://c27n03:3334 Ok
> tcp://c27n04:3334 Ok
> tcp://c27n05:3334 Ok
> tcp://c27n06:3334 Ok
> tcp://c27n07:3334 Ok
> tcp://c27n08:3334 Ok
> tcp://c27n09:3334 Ok
> tcp://c27n10:3334 Ok
> tcp://c27n11:3334 Ok
> tcp://c27n12:3334 Ok
> tcp://c27n13:3334 Ok
> tcp://c27n14:3334 Ok
> tcp://c27n15:3334 Ok
> tcp://c27n16:3334 Ok
> tcp://c27n17:3334 Ok
> tcp://c27n18:3334 Ok
> tcp://c27n19:3334 Ok
> tcp://c27n20:3334 Ok
>
> data servers:
> tcp://c25n17:3334 Ok
> tcp://c25n18:3334 Ok
> tcp://c25n19:3334 Ok
> tcp://c25n20:3334 Ok
> tcp://c25n21:3334 Ok
> tcp://c25n22:3334 Ok
> tcp://c25n23:3334 Ok
> tcp://c25n24:3334 Ok
> tcp://c25n25:3334 Ok
> tcp://c25n26:3334 Ok
> tcp://c25n27:3334 Ok
> tcp://c25n28:3334 Ok
> tcp://c26n01:3334 Ok
> tcp://c26n02:3334 Ok
> tcp://c26n03:3334 Ok
> tcp://c26n04:3334 Ok
> tcp://c26n05:3334 Ok
> tcp://c26n06:3334 Ok
> tcp://c26n07:3334 Ok
> tcp://c26n08:3334 Ok
> tcp://c26n09:3334 Ok
> tcp://c26n10:3334 Ok
> tcp://c26n11:3334 Ok
> tcp://c26n12:3334 Ok
> tcp://c26n13:3334 Ok
> tcp://c26n14:3334 Ok
> tcp://c26n15:3334 Ok
> tcp://c26n16:3334 Ok
> tcp://c26n17:3334 Ok
> tcp://c26n18:3334 Ok
> tcp://c26n19:3334 Ok
> tcp://c26n20:3334 Ok
> tcp://c26n21:3334 Ok
> tcp://c26n22:3334 Ok
> tcp://c26n23:3334 Ok
> tcp://c26n24:3334 Ok
> tcp://c26n25:3334 Ok
> tcp://c26n26:3334 Ok
> tcp://c26n27:3334 Ok
> tcp://c26n28:3334 Ok
> tcp://c26n29:3334 Ok
> tcp://c26n30:3334 Ok
> tcp://c26n31:3334 Ok
> tcp://c26n32:3334 Ok
> tcp://c27n01:3334 Ok
> tcp://c27n02:3334 Ok
> tcp://c27n03:3334 Ok
> tcp://c27n04:3334 Ok
> tcp://c27n05:3334 Ok
> tcp://c27n06:3334 Ok
> tcp://c27n07:3334 Ok
> tcp://c27n08:3334 Ok
> tcp://c27n09:3334 Ok
> tcp://c27n10:3334 Ok
> tcp://c27n11:3334 Ok
> tcp://c27n12:3334 Ok
> tcp://c27n13:3334 Ok
> tcp://c27n14:3334 Ok
> tcp://c27n15:3334 Ok
> tcp://c27n16:3334 Ok
> tcp://c27n17:3334 Ok
> tcp://c27n18:3334 Ok
> tcp://c27n19:3334 Ok
> tcp://c27n20:3334 Ok
>
> (6) Verifying that fsid 1076995 is acceptable to all servers...
>
> Ok; all servers understand fs_id 1076995
>
> (7) Verifying that root handle is owned by one server...
>
> Root handle: 1048576
> Ok; root handle is owned by exactly one server.
>
> =============================================================
>
> The PVFS2 filesystem at /scratch/1076995.bono.ccr.buffalo.edu/mnt
> appears to be correctly configured.
>
> $
>
> All nodes 64 nodes are up:
>
> $ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt | sed -n
> '/^(5)/,/^(6)/p' | sed -n '/data servers/,/^$/p' | sed 1d | sed '$d' |
> wc -l
> 64
> $
>
> I'm trying to create a varstrip_dist dir with (effectively) a stripe
> across all 64 nodes,
> with 16K on each node, but the setfattr fails for 37 (or more) nodes:
>
> $ mount | grep pvfs2
> tcp://c27n20:3334/pvfs2-fs-1076995.bono.ccr.buffalo.edu on
> /scratch/1076995.bono.ccr.buffalo.edu/mnt type pvfs2 (rw)
> $ cd /scratch/1076995.bono.ccr.buffalo.edu/mnt
> $ mkdir test
> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
> $ getfattr test
> # file: test
> user.pvfs2.dist_name
>
> $ getfattr -n "user.pvfs2.dist_name" test
> # file: test
> user.pvfs2.dist_name="varstrip_dist"
>
> $ setfattr -n "user.pvfs2.dist_params" -v
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
> test
> $ getfattr -n "user.pvfs2.dist_params" test
> # file: test
> user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
>
>
> $ setfattr -n "user.pvfs2.dist_params" -v
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
> test
> setfattr: test: Invalid argument
> $ setfattr -n "user.pvfs2.dist_params" -v
> "strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
> test
> setfattr: test: Invalid argument
> $
>
>
> Sorry for the over long lines (I made sure its not a problem with line
> wrapping).
>
> Since the error is from setfattr I wonder if this is a RedHat problem...
> I'm running RedHat Enterprise Linux AS v.4 update 5 on these nodes, and
> the attr package (which includes setfattr) is the latest version from RH:
>
> $ rpm -qf `type -p setfattr`
> attr-2.4.16-3.1.el4
> $
>
>
> Thanks Much,
> Tony
>
> Tony Kew
> SAN Administrator
> The Center for Computational Research
> New York State Center of Excellence
> in Bioinformatics & Life Sciences
> 701 Ellicott Street, Buffalo, NY 14203
>
> CoE Office: (716) 881-8930 Fax: (716) 849-6656
> CSE Office: (716) 645-3797 x2174
> Cell: (716) 560-0910 Home: (716) 874-2126
>
> "I love deadlines, I love the whooshing noise they make as they go by."
> Douglas Adams
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
More information about the Pvfs2-users
mailing list