[PVFS2-users] More NFS and PVFS2
dhildebz at eecs.umich.edu
Mon Oct 25 13:51:09 EDT 2004
Here is a snippet of my conversation with Neil Brown, the Linux NFS server
maintainer, regarding issues with larger block sizes in NFS. It seems
they currently have no plans as I think they are still rather busy with
NFSv4. My guess is that someone would actually have to implement larger
block sizes themselves (through a grant/contract/etc) and then submit it
to the Linux maintainers. As the group that is implementing NFSv4 on
Linux, we only implement the spec plus *special* things that companies pay
for (security flavours, etc). I am on the periphery of all this trying to
figure out a thesis.
"Currently nfsd will wait until an entire request has arrived on a TCP
connection before reading any of it. I don't think you will want to
do this for 4MB requests - the network stack won't want to buffer that
much data. Instead you should arrange the "read" the data while it is
still arriving, which means tying up an nfsd thread while the data is
arriving. For this you would need to make sure you have at least as
many threads as there are open TCP connections.
Also, nfsd currently allocated a full-sized buffer for every thread.
Allocting 4MB of memory, which will mostly be unused, for every thread
could be considered wasteful. You should address that issue."
On Mon, 25 Oct 2004, Rob Ross wrote:
> On Mon, 25 Oct 2004, Dean Hildebrand wrote:
> > > They told me that their NFS-on-GPFS
> > > servers can handle at most thirty or so clients apiece.
> > I've done a fair amount of work with NFSv4 and GPFS and there are many
> > issues that come into play. It has nothing to do with the Linux NFS
> > implementation. As for performance, a single NFS client can totally
> > exhaust a single GPFS server if they are the same type of machine. Or
> > possibly, the GPFS server is much more powerful than the client and so can
> > handle more users simultaneously reading and writing. Or the storage net
> > is faster than the client's net. Stating a number like 30 has more to do
> > with their user's file access characteristics that with GPFS itself.
> > Increasing beyond 30 must increase the chance that multiple clients are
> > simultaneously reading and writing.
> > In general, NFS and distributed/parallel file systems do not work well
> > together.
> Thanks for the discussion Dean! It probably doesn't help that the NFS
> server is accessing 4K at a time :). Do we know if there is any hope of
> adjusting the server so that it accesses larger "blocks" of data?
> PVFS2-users mailing list
> PVFS2-users at beowulf-underground.org
More information about the PVFS2-users