[Pvfs2-developers] File inconsistencies
Brian R. Smith
brs at rc.usf.edu
Fri Jan 20 12:05:20 EST 2006
Just to let you know, we've observed this same behavior on another cluster,
with uniform 32-bit Intel Xeon processors, all running 2.6.14-1.1653_FC4smp.
Again, thank you again for an help and please let me know if you want any more
On Thursday 19 January 2006 23:41, Brian R. Smith wrote:
> Hi all,
> I'll probably be told to take this up in the user's list, but I've had bad
> experiences with 'user' lists. I figured this time, I'd go straight to the
> guys in-the-know.
> I wasn't quite sure how to title this post given the nature of my problem.
> My cluster is made up of some machines which are i386-based, running
> 2.6.13-1.1532_FC4 and 2.6.14-1.1653_FC4smp, and some machines which are
> x86_64-based, running 126.96.36.199smp. I/O and Metaservers are running
> Here is what I have found:
> Given this example file /scratch/xxx/file where /scratch is our PVFS2-1.3.0
> volume and an identical file stored on ext3 /home/student/x/xxx/file, I get
> different results from the stat command:
> [root at host ~]$ stat /scratch/xxx/file
> File: `/scratch/xxx/file'
> Size: 150263 Blocks: 296 IO Block: 4194304 regular file
> Device: 13h/19d Inode: 1046003 Links: 1
> Access: (0644/-rw-r--r--) Uid: (1288406/xxx) Gid: ( 100/ users)
> Access: 2006-01-10 16:48:35.000000000 -0500
> Modify: 2006-01-10 16:48:35.000000000 -0500
> Change: 2006-01-10 16:48:35.000000000 -0500
> [root at host ~]$ stat /home/student/x/xxx/file
> File: `/home/student/x/xxx/file'
> Size: 150263 Blocks: 304 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 63047415 Links: 1
> Access: (0644/-rw-r--r--) Uid: (1229806/xxx) Gid: ( 100/ users)
> Access: 2006-01-19 21:24:41.000000000 -0500
> Modify: 2006-01-19 21:18:02.000000000 -0500
> Change: 2006-01-19 21:18:02.000000000 -0500
> even though 'diff' shows no differences between the two files.
> >From this, I get:
> /scratch/xxx/file: Block size = 296 - File size = 147456
> /home/student/x/xxx/file: Block size = 304 - File size = 150263
> Blocks (stat) Sizes (ls -l)
> 296/304 ~= 147456/150263
> where (296/304 = 0.973584) and (147456/150263 = 0.981319) showing that the
> ratios are roughly equivalent which lead me to investigate the stat, cp,
> and cat commands. Copying the file from PVFS2 causes the file to be
> truncated upon being written to disk.
> The 'cp' command must call xstat() or lxstat() to get the number of blocks
> from a particular file, assumedly using that information to know "how much"
> to copy. On the otherhand
> cat /scratch/xxx/file > /home/student/x/xxx/file
> seems to work perfectly, calling fxstat(). Running 'strings' on 'cp' and
> 'cat' seems to point out a difference in the system calls as 'cat' has a
> single reference to fxstat() while 'cp' references xstat(), fxstat(),
> lxstat(). The stat command only references lxstat() and xstat() so I
> assume that fxstat() must be "safe" for this file at least since 'cat' and
> 'diff' work, but 'cp' and 'stat' are consistently wrong with respect to
> this particular file.
> Now, the point of all this is that I am curious as to why PVFS2 is
> recording/reporting false Block sizes in/from the 'inode table' (not sure
> if PVFS2 even has an inode table or if its emulated). Is this a bug, an
> error on my part or has this already been fixed and I probably should have
> asked for a ChangeLog instead?
> In summary:
> 1) Some files copy cleanly from PVFS2 to other file systems while others do
> 2) The difference in block size reported by stat is proportional to the
> difference in file size reported by 'ls' even though 'cat'ing the file or
> viewing the file while ON PVFS2 shows that all the contents are accounted
> 3) Using the 'cat' command seems to avoid the problem (possibly since it
> avoids making a call to lxstat() or xstat()) but I don't want to push this
> as a fix for our users (nor would any of you, I'd assume).
> 4) My PVFS2 I/O and Metaservers are running on AthlonXP's while the client
> in question is running on Opterons.
> 5) Files "appear" different using stat when in fact they are the same.
> Copying with 'cp' yields truncated files being written. The difference in
> size of the original file from the copy is proportional to the
> discrepancies observed with 'stat'.
> Thanks for any help.
> -Brian Smith
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
Brian R. Smith
HPC Systems Administrator, Research Computing
University of South Florida
4202 E Fowler Ave. LIB 608
Tampa, FL. 33620
More information about the Pvfs2-developers