[Pvfs2-developers] File inconsistencies
Brian R. Smith
brs at rc.usf.edu
Mon Jan 23 12:47:26 EST 2006
Nevermind. 1.3.2 fixes the problem. Should have read the ChangeLog more
On Friday 20 January 2006 12:05, Brian R. Smith wrote:
> Hello again,
> Just to let you know, we've observed this same behavior on another cluster,
> with uniform 32-bit Intel Xeon processors, all running
> Again, thank you again for an help and please let me know if you want any
> more information.
> On Thursday 19 January 2006 23:41, Brian R. Smith wrote:
> > Hi all,
> > I'll probably be told to take this up in the user's list, but I've had
> > bad experiences with 'user' lists. I figured this time, I'd go straight
> > to the guys in-the-know.
> > I wasn't quite sure how to title this post given the nature of my
> > problem. My cluster is made up of some machines which are i386-based,
> > running 2.6.13-1.1532_FC4 and 2.6.14-1.1653_FC4smp, and some machines
> > which are x86_64-based, running 22.214.171.124smp. I/O and Metaservers are
> > running 2.6.13-1.1532_FC4.
> > Here is what I have found:
> > Given this example file /scratch/xxx/file where /scratch is our
> > PVFS2-1.3.0 volume and an identical file stored on ext3
> > /home/student/x/xxx/file, I get different results from the stat command:
> > [root at host ~]$ stat /scratch/xxx/file
> > File: `/scratch/xxx/file'
> > Size: 150263 Blocks: 296 IO Block: 4194304 regular file
> > Device: 13h/19d Inode: 1046003 Links: 1
> > Access: (0644/-rw-r--r--) Uid: (1288406/xxx) Gid: ( 100/ users)
> > Access: 2006-01-10 16:48:35.000000000 -0500
> > Modify: 2006-01-10 16:48:35.000000000 -0500
> > Change: 2006-01-10 16:48:35.000000000 -0500
> > [root at host ~]$ stat /home/student/x/xxx/file
> > File: `/home/student/x/xxx/file'
> > Size: 150263 Blocks: 304 IO Block: 4096 regular file
> > Device: 811h/2065d Inode: 63047415 Links: 1
> > Access: (0644/-rw-r--r--) Uid: (1229806/xxx) Gid: ( 100/ users)
> > Access: 2006-01-19 21:24:41.000000000 -0500
> > Modify: 2006-01-19 21:18:02.000000000 -0500
> > Change: 2006-01-19 21:18:02.000000000 -0500
> > even though 'diff' shows no differences between the two files.
> > >From this, I get:
> > /scratch/xxx/file: Block size = 296 - File size = 147456
> > /home/student/x/xxx/file: Block size = 304 - File size = 150263
> > Blocks (stat) Sizes (ls -l)
> > 296/304 ~= 147456/150263
> > where (296/304 = 0.973584) and (147456/150263 = 0.981319) showing that
> > the ratios are roughly equivalent which lead me to investigate the stat,
> > cp, and cat commands. Copying the file from PVFS2 causes the file to be
> > truncated upon being written to disk.
> > The 'cp' command must call xstat() or lxstat() to get the number of
> > blocks from a particular file, assumedly using that information to know
> > "how much" to copy. On the otherhand
> > cat /scratch/xxx/file > /home/student/x/xxx/file
> > seems to work perfectly, calling fxstat(). Running 'strings' on 'cp' and
> > 'cat' seems to point out a difference in the system calls as 'cat' has a
> > single reference to fxstat() while 'cp' references xstat(), fxstat(),
> > lxstat(). The stat command only references lxstat() and xstat() so I
> > assume that fxstat() must be "safe" for this file at least since 'cat'
> > and 'diff' work, but 'cp' and 'stat' are consistently wrong with respect
> > to this particular file.
> > Now, the point of all this is that I am curious as to why PVFS2 is
> > recording/reporting false Block sizes in/from the 'inode table' (not sure
> > if PVFS2 even has an inode table or if its emulated). Is this a bug, an
> > error on my part or has this already been fixed and I probably should
> > have asked for a ChangeLog instead?
> > In summary:
> > 1) Some files copy cleanly from PVFS2 to other file systems while others
> > do not.
> > 2) The difference in block size reported by stat is proportional to the
> > difference in file size reported by 'ls' even though 'cat'ing the file or
> > viewing the file while ON PVFS2 shows that all the contents are accounted
> > for.
> > 3) Using the 'cat' command seems to avoid the problem (possibly since it
> > avoids making a call to lxstat() or xstat()) but I don't want to push
> > this as a fix for our users (nor would any of you, I'd assume).
> > 4) My PVFS2 I/O and Metaservers are running on AthlonXP's while the
> > client in question is running on Opterons.
> > 5) Files "appear" different using stat when in fact they are the same.
> > Copying with 'cp' yields truncated files being written. The difference
> > in size of the original file from the copy is proportional to the
> > discrepancies observed with 'stat'.
> > Thanks for any help.
> > -Brian Smith
> > _______________________________________________
> > Pvfs2-developers mailing list
> > Pvfs2-developers at beowulf-underground.org
> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Brian R. Smith
HPC Systems Administrator, Research Computing
University of South Florida
4202 E Fowler Ave. LIB 608
Tampa, FL. 33620
More information about the Pvfs2-developers