[Pvfs2-developers] Review: readcaching code

Sun Aug 19 03:13:45 EDT 2007

On Aug 18, 2007, at 12:19 AM, Murali Vilayannur wrote:

> Hi Sam,
>
>> Ah ok.  That makes sense.  A bad value for an extended attribute
>> shouldn't make the common attributes get garbled though.  Is there a
>> range check we can do either on the server when the xattr is set or
>> in the kernel module when the hint is checked?
> Hmm.. yeah thats true. I wonder how/why it got garbled. It has been a
> while since I looked at this code. I cannot recall ..:(
>>
>> Also, should we have checks in set-eattr.sm and del-eattr.sm that
>> prevent the immutable keyval from being unset?  Once a user sets it
>> they shouldn't be able to unset it without deleting the file.
> yep; true
> I was worried about how an admin will ever reclaim space if he/she
> cannot unset that bit
> and hence delete that file.

Maybe I'm missing something but I think deleting the file should be  
allowed, in fact it should be the only way to remove the immutable  
attribute.

-sam

> thanks,
> Murali
>>
>> -sam
>>
>> On Aug 17, 2007, at 10:09 PM, Murali Vilayannur wrote:
>>
>>> Sam,
>>> The problem is not in the system call (fsetxattr) but the arguments
>>> to it..
>>> user.pvfs2.meta_hint is the key and val is actually a uint64  
>>> which is
>>> a bitwise OR
>>> of PVFS_IMMUTABLE_FL, other pvfs flags.
>>> modify_val() in pvfs2-xattr.c will give an example of this usage.
>>> Sorry, it is a little convoluted ..:(
>>> but I couldn't/didn't want to do more string parsing on server side.
>>> Feel free to change that if you think it is needlessly convoluted.
>>> thanks,
>>> Murali
>>>
>>> PS: let me know how the caching patches work out :)
>>> I havent had too much time to play with it since Feb though.
>>> Hope it works :)
>>>
>>>
>>> On 8/17/07, Sam Lang <slang at mcs.anl.gov> wrote:
>>>>
>>>> Hi Murali,
>>>>
>>>> I wrote a little program to test the performance of the read- 
>>>> caching
>>>> immutable file stuff.  With the  attached program, I get a EINVAL
>>>> error on the read of the file after the immutable attribute has  
>>>> been
>>>> set (using fsetxattr).  Also, ls -la gives me really strange  
>>>> results
>>>> for the files that I've set that immutable attribute on.  In the
>>>> below listing, tmpfile1 and tmpfile10 didn't have the immutable
>>>> attribute set.  It looks like the problem is with the fsetxattr
>>>> system call.  The setfattr util does the same thing.  When I set  
>>>> the
>>>> xattr with pvfs2-xattr though, I don't see the corruption in  
>>>> listing
>>>> the file.  I'll try to investigate what fsetxattr is doing, but are
>>>> you aware of any problems with using the system call?
>>>>
>>>> -sam
>>>>
>>>> root at bb22:/tmp/pvfsmnt# ls -la
>>>> total 10260
>>>> drwxrwxrwt 1 slang mpi      4096 2007-08-17 16:35 .
>>>> drwxrwxrwt 5 root  root     4096 2007-08-17 15:47 ..
>>>> drwxrwxrwx 1 slang mpi      4096 2007-08-17 15:47 lost+found
>>>> -rw-r--r-- 1 root  root        0 2007-08-17 16:24 tmpfile1
>>>> -rw-r--r-- 1 root  root 10485760 2007-08-17 16:34 tmpfile10
>>>> ?--------- ? ?     ?           ?                ? tmpfile11
>>>> ?--------- ? ?     ?           ?                ? tmpfile2
>>>> ?--------- ? ?     ?           ?                ? tmpfile3
>>>> ?--------- ? ?     ?           ?                ? tmpfile4
>>>> ?--------- ? ?     ?           ?                ? tmpfile5
>>>> ?--------- ? ?     ?           ?                ? tmpfile6
>>>> ?--------- ? ?     ?           ?                ? tmpfile7
>>>> ?--------- ? ?     ?           ?                ? tmpfile9
>>>> root at bb22:/tmp/pvfsmnt#
>>>>
>>>>
>>>>
>>>> On Feb 20, 2007, at 1:06 AM, Murali Vilayannur wrote:
>>>>
>>>>> Hi all,
>>>>> Finally, I got some time to whip up the read-caching patches for
>>>>> non-mutable files into a semblance of shape and stability.
>>>>> With this patch, I am able to get I/Os to a file (marked  
>>>>> immutable)
>>>>> serviced from the page-cache. One can tag a file as immutable by
>>>>> running,
>>>>> ./src/apps/admin/pvfs2-xattr -s -k user.pvfs2.meta_hint -v
>>>>> "+immutable" /path/to/pvfs2-file
>>>>> To verify if a file is indeed tagged immutable,
>>>>> ./src/apps/admin/pvfs2-xattr -t -k user.pvfs2.meta_hint /path/to/
>>>>> pvfs2-file
>>>>> (or)
>>>>> ./src/apps/admin/pvfs2-stat /path/to/pvfs2/file
>>>>>
>>>>> I have also added some preliminary statistics exported via
>>>>> /proc/sys/pvfs2/stats/
>>>>> that can be used as a placeholder for more interesting statistics
>>>>> later on.
>>>>> Currently, it only shows # of reads, writes, hits in thepage-cache
>>>>> and misses.
>>>>>
>>>>> For some reason now, cache hits do not happen across a file close.
>>>>> Within a file open-close session, all reads get serviced from the
>>>>> cache though. Very weird.
>>>>> My hunch is that file pages are somehow getting removed from the
>>>>> radix
>>>>> tree of the address space due to some page-ref counting issues. I
>>>>> will
>>>>> dig into this later this week.
>>>>>
>>>>> In any case, this code should not cause any regression of older  
>>>>> code
>>>>> paths (hopefully!) and should not impose any performance
>>>>> penalties for
>>>>> workloads making use of the page-cache because of the way we
>>>>> aggregate
>>>>> cache miss I/Os to the server.
>>>>> It was really nice to be able to make use of the iox()
>>>>> infrastructure
>>>>> that was already in place to service non-contigous file and memory
>>>>> I/O.
>>>>> More details of the implementation is described in the thread  
>>>>> below.
>>>>> http://www.beowulf-underground.org/pipermail/pvfs2-developers/ 
>>>>> 2006-
>>>>> November/002847.html
>>>>> Hopefully, I have addressed most of Pete's comments.
>>>>> More comments and testing welcome!
>>>>> thanks,
>>>>> Murali
>>>>> <read-cache-5.patch>
>>>>> _______________________________________________
>>>>> Pvfs2-developers mailing list
>>>>> Pvfs2-developers at beowulf-underground.org
>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2- 
>>>>> developers
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>