[PVFS-users] pvfs becoming unavailable/deathly slow

[Brian K. Jones]

Hi all, 

I have a relatively small cluster of 16 compute nodes and 18 pvfs nodes
(eventually I hope to add another 60 compute nodes). They are all
running ROCKS 3.3.0 (based on Redhat EL 3), and I'm using PVFS as
supplied with that distribution, and I have things configured such that
the data directories use two disks set up in a RAID 0 array. Things
worked wonderfully during testing, but now people are trying to do real
work using the standard Linux system calls to perform the IO, and pvfs
is becoming so slow that it looks to the users like it has completely
crashed. I have, of course, pointed them to "man -k pvfs" which shows
them the pvfs-specific calls ;-)

In the meantime, though, one user has successfully written hundreds of
~2-~10MB files to his pvfs directory, and just running "ls" in that
directory hangs for a LONG, LONG time before producing anything. Running
strace on this results in TONS of messages like these:

================================================================================
lstat64("/mnt/pvfs/jonesy/hrdavid/eyes2-sweep-0-trans-100-rot-171-sub-0_xf.ply", {st_mode=S_IFREG|0644, st_size=3205314, ...}) = 0
getxattr("/mnt/pvfs/jonesy/hrdavid/eyes2-sweep-0-trans-100-rot-171-sub-0_xf.ply", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported)
lstat64("/mnt/pvfs/jonesy/hrdavid/eyes2-sweep-1-trans-220-rot-171-sub-0_xf.ply", {st_mode=S_IFREG|0644, st_size=15043579, ...}) = 0
getxattr("/mnt/pvfs/jonesy/hrdavid/eyes2-sweep-1-trans-220-rot-171-sub-0_xf.ply", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported)
lstat64("/mnt/pvfs/jonesy/hrdavid/eyes2-sweep-2-trans-340-rot-171-sub-0_xf.ply", {st_mode=S_IFREG|0644, st_size=3177225, ...}) = 0
getxattr("/mnt/pvfs/jonesy/hrdavid/eyes2-sweep-2-trans-340-rot-171-sub-0_xf.ply", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported)
lstat64("/mnt/pvfs/jonesy/hrdavid/eyes3-sweep-0-trans--07-rot-83-sub-0_xf.ply", {st_mode=S_IFREG|0644, st_size=12578337, ...}) = 0
getxattr("/mnt/pvfs/jonesy/hrdavid/eyes3-sweep-0-trans--07-rot-83-sub-0_xf.ply", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported)
lstat64("/mnt/pvfs/jonesy/hrdavid/eyes4-sweep-0-trans-245-rot-206-sub-0_xf.ply", {st_mode=S_IFREG|0644, st_size=13500528, ...}) = 0
getxattr("/mnt/pvfs/jonesy/hrdavid/eyes4-sweep-0-trans-245-rot-206-sub-0_xf.ply", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported)
lstat64("/mnt/pvfs/jonesy/hrdavid/eyes6-sweep-0-trans--07-rot-63-sub-0_xf.ply", {st_mode=S_IFREG|0644, st_size=11878266, ...}) = 0
getxattr("/mnt/pvfs/jonesy/hrdavid/eyes6-sweep-0-trans--07-rot-63-sub-0_xf.ply", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported)
============================================================================


Any clues on how to get rid of these messages? If it weren't for these
attempts to resolve things that apparently aren't supported by the
filesystem, standard linux "ls" would be fine. 

BTW - I know there is a pvfs-ls command, but this is a bit less
transparent to the user. 

Thanks, 
brian.