[PVFS2-users] syncing in PVFS2

Dean Hildebrand dhildebz at eecs.umich.edu
Mon Oct 11 13:56:44 EDT 2004


On Mon, 11 Oct 2004 neillm at mcs.anl.gov wrote:

> On Mon, Oct 11, 2004 at 12:06:36PM -0400, Dean Hildebrand wrote:
> > Thanks Neill.  NFS does a fsync so things should work the same.  O_DIRECT
> > is an open option, but I'm not using it.
>
> Right, but is it implied on the server with the 'sync' mode option
> (even regardless of the O_SYNC open option)?  Those are the kinds of
> details that I don't know about NFS.  I'm not even sure what O_DIRECT
> means exactly as an open mode for an NFS file.  Does it actually offer
> a semantic similar to ours?
O_DIRECT removes any client caching for a particular file.  It may
even slow down the system because read/writes are now synchronous
operations.  NFS depends on the exportfs options.  If a file system is
exported async with exportfs, then the nfs COMMIT operation does not
perform a fync, otherwise it does.  Currently a commit request is sent for
every write (every 32K).

> > In my tests, running iozone on the server disk gives 13MB/s writes.  When
> > storing the PVFS2 file system on the same disk and then accessing from a
> > remote PVFS2 client I get 44MB/s.  Am I missing something?  I don't
> > remember if the Berkley DB is involved in data or just metadata so I don't
> > know if it could affect things here.
>
> Sorry, but I get confused recalling your setup.  The first test was an
> NFS client (on top of PVFS2) running on the same machine as the NFS
> server?  The second was an NFS client (on top of PVFS2) running on a
> remote machine accessing the server?  I'm not sure I understand enough
> about your setup to say anything about your results.  A few things to
> check though are CPU usage?  Is something on the server slowing
> everything down when running the test on it (as opposed to the remote
> machine setup)?
Sorry for the confusion.  Completely forget my old setup.  New setup is 2
machines running PVFS2.  (client and server)  The PVFS2 data resides in a
ext3 file system.  Iozone using -Raec options (syncing).  Writing a 128MB
file.  No other running processes on machines.

Test1: iozone on ext3 file system (on server machine) -> 13MB/s writes
Test2: iozone on PVFS2 client                         -> 44MB/s writes

How is it possible for PVFS2 to perform better than the file system in
which it resides?  It is beating the disk bandwidth.  Something is not
syncing properly.

Dean


More information about the PVFS2-users mailing list