[PVFS2-users] NFS and PVFS2

[Rob Ross]

So to summarize, the rsize/wsize values are not affecting the size of 
transfers between the NFSv4 server and the pvfs2-client, which is always 
small.  The best solution would be to find a way to get the NFSv4 server 
to use larger blocks when accessing PVFS2.

Regards,

Rob

On Thu, 7 Oct 2004, Neill Miller wrote:

> On Wed, 6 Oct 2004, Rob Ross wrote:
> 
> > It would be helpful to get a log of the interactions between the NFSv4 
> > server and the pvfs2-client-core; are those 32K operations, or 4K 
> > operations, for example.  Also, when you say "server" you're saying that 
> > the PVFS2 server and NFSv4 server are on the same machine, right?  It's a 
> > single processor box?
> 
> I've been looking at what's going on here and it looks like the worst case 
> is true.  For reference, I have not changed the block size of NFS, so if 
> 4K is the default, that's what I'm using for this.
> 
> ====
> maceva pvfs2 # df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> <snip>
> tcp://localhost:3334/pvfs2-fs
>                       14635008  13836288    798720  95% /tmp/mnt
> localhost:/tmp/mnt    14635008  13836288    798720  95% /tmp/nfs
> 
> 
> [ FIRST: a copy from /dev/zero to the mounted pvfs2 volume ]
> maceva linux-2.6 # time dd if=/dev/zero of=/tmp/mnt/pvfs2-file bs=4MB 
> count=1
> 1+0 records in
> 1+0 records out
> 4000000 bytes transferred in 0.622527 seconds (6425423 bytes/sec)
> 
> real    0m0.676s
> user    0m0.002s
> sys     0m0.019s
> 
> [ SECOND: a copy from /dev/zero to the mounted nfs volume over pvfs2 ]
> maceva linux-2.6 # time dd if=/dev/zero of=/tmp/nfs/nfs-file bs=4MB 
> count=1
> 1+0 records in
> 1+0 records out
> 4000000 bytes transferred in 15.722940 seconds (254405 bytes/sec)
> 
> real    0m15.820s
> user    0m0.002s
> sys     0m0.031s
> ====
> 
> I've verified that at least for my nfs over pvfs2 configuration (i.e. 
> nothing fancy), each pvfs2-client-core I/O request is in fact 4K or 
> smaller (there seem to be a number of 124 byte operations that complete a 
> previous 3972 byte operation -- which of course is expensive in the pvfs2 
> world).
> 
> Dean, I suspect that even if you're using 32K block sizes, any kind of 
> bulk I/O is going to be much slower over NFS/PVFS2 rather than directly to 
> PVFS2.
> 
> That said, iozone benchmarks running on NFS over PVFS2 still knock the 
> socks off standalone PVFS2, as it must be very smart about what needs to 
> actually be written to disk.  (In both cases, a number of ~4K blocks or 
> smaller are written through the pvfs2-client-core).
> 
> Does this information help any?
> 
> -Neill.
> 
>