[Pvfs2-developers] Review: readcaching code

[Murali Vilayannur]

Hi Sam,

> Ah ok.  That makes sense.  A bad value for an extended attribute
> shouldn't make the common attributes get garbled though.  Is there a
> range check we can do either on the server when the xattr is set or
> in the kernel module when the hint is checked?
Hmm.. yeah thats true. I wonder how/why it got garbled. It has been a
while since I looked at this code. I cannot recall ..:(
>
> Also, should we have checks in set-eattr.sm and del-eattr.sm that
> prevent the immutable keyval from being unset?  Once a user sets it
> they shouldn't be able to unset it without deleting the file.
yep; true
I was worried about how an admin will ever reclaim space if he/she
cannot unset that bit
and hence delete that file.
thanks,
Murali
>
> -sam
>
> On Aug 17, 2007, at 10:09 PM, Murali Vilayannur wrote:
>
> > Sam,
> > The problem is not in the system call (fsetxattr) but the arguments
> > to it..
> > user.pvfs2.meta_hint is the key and val is actually a uint64 which is
> > a bitwise OR
> > of PVFS_IMMUTABLE_FL, other pvfs flags.
> > modify_val() in pvfs2-xattr.c will give an example of this usage.
> > Sorry, it is a little convoluted ..:(
> > but I couldn't/didn't want to do more string parsing on server side.
> > Feel free to change that if you think it is needlessly convoluted.
> > thanks,
> > Murali
> >
> > PS: let me know how the caching patches work out :)
> > I havent had too much time to play with it since Feb though.
> > Hope it works :)
> >
> >
> > On 8/17/07, Sam Lang <slang at mcs.anl.gov> wrote:
> >>
> >> Hi Murali,
> >>
> >> I wrote a little program to test the performance of the read-caching
> >> immutable file stuff.  With the  attached program, I get a EINVAL
> >> error on the read of the file after the immutable attribute has been
> >> set (using fsetxattr).  Also, ls -la gives me really strange results
> >> for the files that I've set that immutable attribute on.  In the
> >> below listing, tmpfile1 and tmpfile10 didn't have the immutable
> >> attribute set.  It looks like the problem is with the fsetxattr
> >> system call.  The setfattr util does the same thing.  When I set the
> >> xattr with pvfs2-xattr though, I don't see the corruption in listing
> >> the file.  I'll try to investigate what fsetxattr is doing, but are
> >> you aware of any problems with using the system call?
> >>
> >> -sam
> >>
> >> root at bb22:/tmp/pvfsmnt# ls -la
> >> total 10260
> >> drwxrwxrwt 1 slang mpi      4096 2007-08-17 16:35 .
> >> drwxrwxrwt 5 root  root     4096 2007-08-17 15:47 ..
> >> drwxrwxrwx 1 slang mpi      4096 2007-08-17 15:47 lost+found
> >> -rw-r--r-- 1 root  root        0 2007-08-17 16:24 tmpfile1
> >> -rw-r--r-- 1 root  root 10485760 2007-08-17 16:34 tmpfile10
> >> ?--------- ? ?     ?           ?                ? tmpfile11
> >> ?--------- ? ?     ?           ?                ? tmpfile2
> >> ?--------- ? ?     ?           ?                ? tmpfile3
> >> ?--------- ? ?     ?           ?                ? tmpfile4
> >> ?--------- ? ?     ?           ?                ? tmpfile5
> >> ?--------- ? ?     ?           ?                ? tmpfile6
> >> ?--------- ? ?     ?           ?                ? tmpfile7
> >> ?--------- ? ?     ?           ?                ? tmpfile9
> >> root at bb22:/tmp/pvfsmnt#
> >>
> >>
> >>
> >> On Feb 20, 2007, at 1:06 AM, Murali Vilayannur wrote:
> >>
> >>> Hi all,
> >>> Finally, I got some time to whip up the read-caching patches for
> >>> non-mutable files into a semblance of shape and stability.
> >>> With this patch, I am able to get I/Os to a file (marked immutable)
> >>> serviced from the page-cache. One can tag a file as immutable by
> >>> running,
> >>> ./src/apps/admin/pvfs2-xattr -s -k user.pvfs2.meta_hint -v
> >>> "+immutable" /path/to/pvfs2-file
> >>> To verify if a file is indeed tagged immutable,
> >>> ./src/apps/admin/pvfs2-xattr -t -k user.pvfs2.meta_hint /path/to/
> >>> pvfs2-file
> >>> (or)
> >>> ./src/apps/admin/pvfs2-stat /path/to/pvfs2/file
> >>>
> >>> I have also added some preliminary statistics exported via
> >>> /proc/sys/pvfs2/stats/
> >>> that can be used as a placeholder for more interesting statistics
> >>> later on.
> >>> Currently, it only shows # of reads, writes, hits in thepage-cache
> >>> and misses.
> >>>
> >>> For some reason now, cache hits do not happen across a file close.
> >>> Within a file open-close session, all reads get serviced from the
> >>> cache though. Very weird.
> >>> My hunch is that file pages are somehow getting removed from the
> >>> radix
> >>> tree of the address space due to some page-ref counting issues. I
> >>> will
> >>> dig into this later this week.
> >>>
> >>> In any case, this code should not cause any regression of older code
> >>> paths (hopefully!) and should not impose any performance
> >>> penalties for
> >>> workloads making use of the page-cache because of the way we
> >>> aggregate
> >>> cache miss I/Os to the server.
> >>> It was really nice to be able to make use of the iox()
> >>> infrastructure
> >>> that was already in place to service non-contigous file and memory
> >>> I/O.
> >>> More details of the implementation is described in the thread below.
> >>> http://www.beowulf-underground.org/pipermail/pvfs2-developers/2006-
> >>> November/002847.html
> >>> Hopefully, I have addressed most of Pete's comments.
> >>> More comments and testing welcome!
> >>> thanks,
> >>> Murali
> >>> <read-cache-5.patch>
> >>> _______________________________________________
> >>> Pvfs2-developers mailing list
> >>> Pvfs2-developers at beowulf-underground.org
> >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> >>
> >>
> >>
> >>
> >
>
>