[Pvfs2-users] Alternative pvfs2 and Direct-NFS ? or am I mad, it's almost to good to be true

[Robert Latham]

On Tue, May 27, 2008 at 05:27:32PM +0200, belcampo wrote:
> I've found a another workaround which works remarkebly well.
> On a pvfs2-client which also is meta and io/server I mount  
> pvfs2-filesystem on /pvfs2 which I then export by nfs3 and localy mount  
> at /pvfs2nfs. rsize and wsize left out, see strace output. I accidently  
> discovered this by doing a simple mount -t nfs .. .. which was really  
> fast. When I modified fstab with .... rsize=8192,wsize=8192 which I  
> normally do for nfs3, performance was much less, same ffmpeg commands  
> resulted in a drop from fps=3129 to fps=1079, and fps=2329 to fps=610
>
> mplayer /pvfs2nfs/somefile.mpg works terrific
>
> How bad/stupid is this workaround ?

Hi Henk

This is a perfectly valid workaround.  It's one we recommend for
workload like yours: mplayer is a serial, read-only  application, so
there's no worry about NFS caching causing unexpected results for
another process.  

As you've seen, the bigger the rsize/wsize, the better.  We expect
this as well. 

> ffmpeg -i /pvfs2nfs/some.mpg .. local/nonpvfs2/filesystem       fps=3129
> ffmpeg -i /pvfs2nfs/some.mpg .. /pvfs2/filesystem               fps=396
> ffmpeg -i /pvfs2nfs/some.mpg .. /pvfs2nfs/somenew.mpg           fps=2329
> ffmpeg -i local/nonpvfs2/filesystem .. /pvfs2nfs/filesystem     fps=1911
>
> 1st is read from pvfs2 through nfs and write directly to local xfs
> 2nd is read from pvfs2 through nfs and write directly to pvfs2
> 3rd is read from pvfs2 through nfs and write through nfs to pvfs2
> 4th is read from local xfs and write to pvfs2 through nfs
>
> 1st strace IO value for read 32768 and write 2048
> 2nd strace IO value for read 2048 and write 2048
> 3rd strace IO value for read 32768 and write 2048
> 4th strace IO value for read 32768 and write 2048

This is where NFS caching is really helping PVFS: those 2k writes are
really bad for a parallel file system, but with NFS caching the writes
are batched up into 32k sized operations.  Much nicer.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B