[Pvfs2-users] PVFS2 Performance

[Sam Lang]

Nasr Y.M.J.O. wrote:
> 
> Thanks Dean. Well, my /etc/exports file has this entry:
> 
> /export 10.0.0.0/255.0.0.0(rw,async)
> 
> and the pvfs2-fs.conf file has these entries:
> 
> <StorageHints>
>             TroveSyncMeta  Yes
>             TroveSyncData   No
> </StorageHints>
> 
> So both nfs and pvfs2 are set to async. But the meta is set to sync, it 
> may be the reason for this. Hmm... Still not clear! Maybe I need to set 
> Meta to 'no' even though its not advisable.

The TroveSyncMeta option is for metadata, so changing that won't affect 
your IO performance.  I can't say why the nfs numbers you get don't 
change much from sync to async, could the nfs server be syncing on close 
even though you've set async in your exports file?  As for pvfs, the 
performance drop for writes with sync is expected, since the writes 
actually hit the disk with sync enabled, but just stay in disk buffers 
with sync disabled.

I wouldn't expect read performance to change with sync enabled/disabled, 
since the reads don't require any syncing anyway.  You're likely doing 
reads directly from the disk buffer though, since you just did a write 
of the data and its still sitting in there.  If you want to see what the 
performance is for reading from disk, you'll have to run tests that read 
from random parts of a much larger file, or with large enough block 
sizes that disk buffers aren't a factor.

-sam

> 
> Thanks
> nasr
> 
>> From: Dean Hildebrand <dhildebz at eecs.umich.edu>
>> To: "Nasr Y.M.J.O." <nasr974 at hotmail.com>
>> CC: vilayann at mcs.anl.gov, pvfs2-users at beowulf-underground.org
>> Subject: Re: [Pvfs2-users] PVFS2 Performance
>> Date: Wed, 20 Sep 2006 13:03:29 -0400
>>
>>
>>> NFS:
>>> -----
>>>
>>> Access size per process = 67108864 bytes, ntimes = 5
>>> Write bandwidth without file sync = 70.134351 Mbytes/sec
>>> Write bandwidth including file sync = 71.789752 Mbytes/sec
>>>
>>> PVFS2
>>> --------
>>> Access size per process = 67108864 bytes, ntimes = 5
>>> Write bandwidth without file sync = 329.618503 Mbytes/sec
>>> Write bandwidth including file sync = 121.779536 Mbytes/sec
>>>
>>> I did the test after tuning TCP using netpipe benchmark. I can see 
>>> there is a big gap between write and read with sync in pvfs2 as 
>>> compared to NFS (121, 320 compared to 71, 80). Why?
>> I'll take a stab at this.  In order for NFS to maintain open-to-close 
>> semantics, all data is flushed to stable storage at file close.  
>> Issuing a fsync just before file close does not do anything extra.  So 
>> your NFS experiments are always doing the same thing.  I would guess 
>> PVFS2 does not sync data to stable storage unless it is specifically 
>> requested.
>>
>> If you want to try removing this behavior from NFS, simply change your 
>> /etc/exports file to export data asynchronously.  Once this is done, 
>> your server may or may not flush data to stable storage.  Here is the 
>> relevant part from the exports man page:
>>       async  This  option  allows  the NFS server to violate the NFS 
>> protocol
>>              and reply to requests before any changes made  by  that  
>> request
>>              have been committed to stable storage (e.g. disc drive).
>>
>>              Using  this option usually improves performance, but at 
>> the cost
>>              that an unclean server restart (i.e. a crash) can cause 
>> data  to
>>              be lost or corrupted.
>>
>>              In  releases  of nfs-utils upto and including 1.0.0, this 
>> option
>>              was the default.  In this  and  future  releases,  sync  
>> is  the
>>              default,  and  async  must  be explicit requested if 
>> needed.  To
>>              help make system adminstrators aware of this change,  
>> âexportfsâ
>>              will issue a warning if neither sync nor async is specified.
>>
>>
>>
>> Dean
>>>
>>> Many thanks,
>>> nasr
>>>
>>> --------------------------------------------------------------------------------------------------------------------------------------- 
>>>
>>>
>>> io.c program error
>>>
>>>
>>> # /opt/mpich1/intel/bin/mpicc io.c -g -Wall -o io
>>>
>>> io.c(49): error: identifier "O_LARGEFILE" is undefined
>>>  int     amode         = O_RDWR | O_CREAT | O_LARGEFILE;
>>>                                                                ^
>>>
>>> io.c(53): remark #1419: external declaration in primary source file
>>>  int parse_args(int argc, char **argv);
>>>      ^
>>>
>>> io.c(54): remark #1419: external declaration in primary source file
>>>  double Wtime(void);
>>>            ^
>>>
>>> io.c(57): remark #1419: external declaration in primary source file
>>>  extern char *optarg;
>>>                    ^
>>>
>>> io.c(58): remark #1419: external declaration in primary source file
>>>  extern int optind, opterr;
>>>                ^
>>>
>>> io.c(58): remark #1419: external declaration in primary source file
>>>  extern int optind, opterr;
>>>                           ^
>>>
>>> compilation aborted for io.c (code 2)
>>>
>>> ------------------------------------------------------------------------------------------------------------------------------------------- 
>>>
>>>
>>>
>>> _________________________________________________________________
>>> Find love online with MSN Personals. 
>>> http://match.msn.com.my/match/mt.cfm?pg=channel
>>>
>>> _______________________________________________
>>> Pvfs2-users mailing list
>>> Pvfs2-users at beowulf-underground.org
>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>> -- 
>> Dean Hildebrand
>> Ph.D. Candidate
>> University of Michigan
>>
> 
> _________________________________________________________________
> Find love online with MSN Personals. 
> http://match.msn.com.my/match/mt.cfm?pg=channel
> 
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>