[Pvfs2-developers] File inconsistencies

Brian R. Smith brs at rc.usf.edu
Mon Jan 23 12:47:26 EST 2006


Nevermind. 1.3.2 fixes the problem.  Should have read the ChangeLog more 
carefully.

-Brian

On Friday 20 January 2006 12:05, Brian R. Smith wrote:
> Hello again,
>
> Just to let you know, we've observed this same behavior on another cluster,
> with uniform 32-bit Intel Xeon processors, all running
> 2.6.14-1.1653_FC4smp.
>
> Again, thank you again for an help and please let me know if you want any
> more information.
>
> -Brian
>
> On Thursday 19 January 2006 23:41, Brian R. Smith wrote:
> > Hi all,
> >
> > I'll probably be told to take this up in the user's list, but I've had
> > bad experiences with 'user' lists.  I figured this time, I'd go straight
> > to the guys in-the-know.
> >
> > I wasn't quite sure how to title this post given the nature of my
> > problem. My cluster is made up of some machines which are i386-based,
> > running 2.6.13-1.1532_FC4 and 2.6.14-1.1653_FC4smp, and some machines
> > which are x86_64-based, running 2.6.11.11smp.  I/O and Metaservers are
> > running 2.6.13-1.1532_FC4.
> >
> > Here is what I have found:
> >
> > Given this example file /scratch/xxx/file where /scratch is our
> > PVFS2-1.3.0 volume and an identical file stored on ext3
> > /home/student/x/xxx/file, I get different results from the stat command:
> >
> > [root at host ~]$ stat /scratch/xxx/file
> > File: `/scratch/xxx/file'
> >   Size: 150263          Blocks: 296        IO Block: 4194304 regular file
> > Device: 13h/19d Inode: 1046003     Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (1288406/xxx)   Gid: (  100/   users)
> > Access: 2006-01-10 16:48:35.000000000 -0500
> > Modify: 2006-01-10 16:48:35.000000000 -0500
> > Change: 2006-01-10 16:48:35.000000000 -0500
> >
> > [root at host ~]$ stat /home/student/x/xxx/file
> >   File: `/home/student/x/xxx/file'
> >   Size: 150263          Blocks: 304        IO Block: 4096   regular file
> > Device: 811h/2065d      Inode: 63047415    Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (1229806/xxx)   Gid: (  100/   users)
> > Access: 2006-01-19 21:24:41.000000000 -0500
> > Modify: 2006-01-19 21:18:02.000000000 -0500
> > Change: 2006-01-19 21:18:02.000000000 -0500
> >
> > even though 'diff' shows no differences between the two files.
> >
> > >From this, I get:
> >
> > /scratch/xxx/file: Block size = 296 - File size = 147456
> > /home/student/x/xxx/file: Block size = 304 - File size = 150263
> >
> > Blocks (stat)        Sizes (ls -l)
> > 296/304         ~= 147456/150263
> >
> > where (296/304 = 0.973584) and (147456/150263 = 0.981319) showing that
> > the ratios are roughly equivalent which lead me to investigate the stat,
> > cp, and cat commands.  Copying the file from PVFS2 causes the file to be
> > truncated upon being written to disk.
> >
> > The 'cp' command must call xstat() or lxstat() to get the number of
> > blocks from a particular file, assumedly using that information to know
> > "how much" to copy.  On the otherhand
> >
> > cat /scratch/xxx/file > /home/student/x/xxx/file
> >
> > seems to work perfectly, calling fxstat().  Running 'strings' on 'cp' and
> > 'cat' seems to point out a difference in the system calls as 'cat' has a
> > single reference to fxstat() while 'cp' references xstat(), fxstat(),
> > lxstat().  The stat command only references lxstat() and xstat() so I
> > assume that fxstat() must be "safe" for this file at least since 'cat'
> > and 'diff' work, but 'cp' and 'stat' are consistently wrong with respect
> > to this particular file.
> >
> > Now, the point of all this is that I am curious as to why PVFS2 is
> > recording/reporting false Block sizes in/from the 'inode table' (not sure
> > if PVFS2 even has an inode table or if its emulated).  Is this a bug, an
> > error on my part or has this already been fixed and I probably should
> > have asked for a ChangeLog instead?
> >
> > In summary:
> >
> > 1) Some files copy cleanly from PVFS2 to other file systems while others
> > do not.
> >
> > 2) The difference in block size reported by stat is proportional to the
> > difference in file size reported by 'ls' even though 'cat'ing the file or
> > viewing the file while ON PVFS2 shows that all the contents are accounted
> > for.
> >
> > 3) Using the 'cat' command seems to avoid the problem (possibly since it
> > avoids making a call to lxstat() or xstat()) but I don't want to push
> > this as a fix for our users (nor would any of you, I'd assume).
> >
> > 4) My PVFS2 I/O and Metaservers are running on AthlonXP's while the
> > client in question is running on Opterons.
> >
> > 5) Files "appear" different using stat when in fact they are the same.
> > Copying with 'cp' yields truncated files being written.  The difference
> > in size of the original file from the copy is proportional to the
> > discrepancies observed with 'stat'.
> >
> > Thanks for any help.
> >
> >
> > -Brian Smith
> >
> >
> >
> >
> > _______________________________________________
> > Pvfs2-developers mailing list
> > Pvfs2-developers at beowulf-underground.org
> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers

-- 
Brian R. Smith
HPC Systems Administrator, Research Computing
University of South Florida
4202 E Fowler Ave. LIB 608
Tampa, FL. 33620
1(813)974-1467
http://rc.usf.edu


More information about the Pvfs2-developers mailing list