[PVFS2-users] syncing in PVFS2

[neillm at mcs.anl.gov]

On Mon, Oct 11, 2004 at 12:06:36PM -0400, Dean Hildebrand wrote:
> Thanks Neill.  NFS does a fsync so things should work the same.  O_DIRECT
> is an open option, but I'm not using it.

Right, but is it implied on the server with the 'sync' mode option
(even regardless of the O_SYNC open option)?  Those are the kinds of
details that I don't know about NFS.  I'm not even sure what O_DIRECT
means exactly as an open mode for an NFS file.  Does it actually offer
a semantic similar to ours?

> In my tests, running iozone on the server disk gives 13MB/s writes.  When
> storing the PVFS2 file system on the same disk and then accessing from a
> remote PVFS2 client I get 44MB/s.  Am I missing something?  I don't
> remember if the Berkley DB is involved in data or just metadata so I don't
> know if it could affect things here.

Sorry, but I get confused recalling your setup.  The first test was an
NFS client (on top of PVFS2) running on the same machine as the NFS
server?  The second was an NFS client (on top of PVFS2) running on a
remote machine accessing the server?  I'm not sure I understand enough
about your setup to say anything about your results.  A few things to
check though are CPU usage?  Is something on the server slowing
everything down when running the test on it (as opposed to the remote
machine setup)?

The Berkerley DB is used only for metadata (attributes, etc), btw.

> Also, as an aside, what is the new default behavior without the
> TroveSyncMode option?  I notice that the pvfs2_fysnc function in file.c is
> empty, so I guess syncs are independant of user requests.

In pvfs2-0.6.2, TroveSyncMode was replaced with two separate flags,
TroveSyncMeta, and TroveSyncData which can be set to either 'yes' or
'no' and are always 'yes' by default.  In pvfs2-0.6.5, I've removed
any sync calls to readonly operations (including both data and
metadata).  So there are a few differences from the old TroveSyncMode
behaviour (which called sync after every meta and data operation,
regardless of what it was.  This is potentially very expensive
depending on the underlying file system).

Currently, from the VFS standpoint in PVFS2, fsync does nothing
because data is sync'd on the server by default on successful write
completion.  While I could have added an upcall to PVFS2's sys-flush
in the VFS' fsync call, It wouldn't gain anything at the moment and
would only slow down normal operation.  If/when optional client side
caching is available, fsync will have to do real work to flush those
caches.

-Neill.