[Pvfs2-developers] Re: [Pvfs2-users] FUSE for PVFS

[Sam Lang]

On Jun 12, 2007, at 7:09 PM, Pete Wyckoff wrote:

> john.chandy at uconn.edu wrote on Tue, 12 Jun 2007 19:20 -0400:
>> If there is any interest in running non-linux clients on PVFS, we
>> have a preliminary release of pvfs2fuse which is a FUSE
>> implementation of a PVFS filesystem.  It allows any OS that supports
>> FUSE to mount a PVFS filesystem.  Presently, this includes Linux,
>> FreeBSD, and MacOSX.
>>
>> The code is a set of patches to pvfs-2.6.3 and is available at  
>> http://
>> snsl.engr.uconn.edu/downloads/pvfs2fuse.html.
>>
>> I've done limited testing on Linux 2.6.9 and MacOSX 10.4.9 but it
>> should work on other versions and FreeBSD as well.
>>
>> After doing some testing with a 6-node Linux cluster, pvfs2fuse on
>> Linux performance is comparable to the PVFS built-in client.  Iozone
>> shows the PVFS client topping out at ~100 MB/s and pvfs2fuse topping
>> out at ~75 MBs for large files but for small files and small record
>> sizes pvfs2fuse does about 30% better.  We have yet to do any
>> analysis of pvfs2fuse performance to see where the bottlenecks are.

Hi John,

Out of curiosity, what size files are considered small and large?

>
> This is awesome work.  Fascinating that it takes only 1200 lines to
> do a FUSE-based implementation compared to the 15k lines in the
> current kernel module.

I agree this is great.  From a lines of code perspective they seem  
comparable.  Fuse itself is about 15K lines of code as well.

>
> I would especially encourage a performance comparison.  Looking at
> latency, small- and large-block throughputs, etc. would be good work
> for a conference submission if you are interested in such things.
>
> Is it true, for instance, that FUSE generates only page-sized reads
> or writes?  How does that impact performance?  Are context switches
> more or less of a problem with the FUSE version (compared to stock
> pvfs kernel module)?
>
> FUSE is a purely synchronous implementation, apparently.  What
> happens if two processes (e.g., ls foo & ls bar)
>
> Some comments on the patch.
>
> Your configure has more changes in it than are reflected in the
> configure.in.  We want the stuff in configure for this patch; not
> sure why it is not in configure.in.
>
> The *-apple-* patch to configure.in should come in a different patch
> if it is a useful thing.  Ditto with the -L spacing changes in
> Makefile.in.  And the #if __APPLE__ multiple places elsewhere.  And
> PINT_FSTAB_TYPE fix.  Maybe you could break this up into "apple
> fixes" and "fuse patch"?

Along these lines, if its possible to make fixes to the code that  
aren't darwin/apple specific, we tend to prefer that.  For example,  
instead of a separate aiocb struct just for darwin, we could add a  
check to the configure.in for __error_code and __return_value fields  
in that struct, and if they don't exist, just #ifndef them out where  
they're normally used.

Thanks for the patch.  Modulo some minor changes I think this will be  
a worthwhile commit.  If you're interested in reporting nightly test  
builds on darwin to our tinderbox build status page so that we can  
keep track of commits that break on that platform, we could help you  
set that up.

See: http://www.pvfs.org/pvfs2-nightly-builds/tinderbox2/PVFS2/ 
status.html

-sam

>
> Any chance you could fix some of the 10 FIXMEs in pvfs2fuse.c?  Are
> they important?
>
> There is an errant usleep() in some getattr code.  Assuming that can
> be safely removed.  :)
>
> I don't understand how readdir works:
>     incount = 32
>     do {
> 	PVFS_sys_readdir(&outcount)
> 	put returned responses into fuse response
>     } while (outcount == incount)
> What happens for a directory that has 31 or 33 files?
>
> I can never keep track of the style in pvfs.  But I think for ifs
> and fors you want:
>
>     if ((x && foo) || (x && bar))
>     {
>
> noting the spaces around the various parens, and after the keyword.
>
> Does sendfile work?  :)
>
> 		-- Pete
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>