[Pvfs2-developers] TroveSyncData settings

[Kyle Schochenmaier]

Phil Carns wrote:
> We recently ran some tests trying different sync settings in PVFS2.  
> We ran into one pleasant surprise, although probably it is already 
> obvious to others.  Here is the setup:
>
> 12 clients
> 4 servers
> read/write test application, 100 MB operations, large files
> fibre channel SAN storage
>
> The test application is essentially the same as was used in the 
> posting regarding kernel buffer sizes, although with different 
> parameters in this environment.
>
> At any rate, to get to the point:
>
> with TroveSyncData=no (default settings): 173 MB/s
> with TroveSyncData=yes: 194 MB/s
>
> I think the issue is that if syncdata is turned off, then the buffer 
> cache tends to get very full before it starts writing.  This bursty 
> behavior isn't doing the SAN any favors- it has a big cache on the 
> back end and probably performs better with sustained writes that don't 
> put so much sudden peak traffic on the HBA card.
>
> There are probably more sophisticated variations of this kind of 
> tuning around (/proc vm settings, using direct io, etc.) but this is 
> an easy config file change to get an extra 12% throughput.
>
> This setting is a little more unpredictable for local scsi disks- some 
> combinations of application and node go faster but some go slower. 
> Overall it seems better for our environment to just leave data syncing 
> on for both SAN and local disk, but your mileage may vary.
>
> This is different from results that we have seen in the past (maybe a 
> year ago or so) for local disk- it used to be a big penalty to sync 
> every data operation.  I'm not sure what exactly happened to change 
> this (new dbpf design?  alt-aio?  better kernels?) but I'm not 
> complaining :)
>
One thing that we noticed while testing for storage challenge was that 
(and everyone correct me if I'm wrong here) enabling the data-sync 
causes a flush/sync to occur after every sizeof(FlowBuffer) bytes had 
been written.  I can imagine how this would help a SAN, but I'm 
perplexed how it helps localdisk, what buffer size are you playing with? 

We found that unless we were using HUGE (~size of cache on storage 
controller) flowbuffers that this caused way too many syncs/seeks on the 
disks and hurt performance quite a bit, maybe even as bad as 50% 
performance because things were not being optimized for our disk 
subsystems and we were issuing many small ops instead of fewer large ones.

Granted I havent been able to get 2.6.0 building properly yet to test 
the latest out, but this was definitely the case for us on the 2.5 releases.

+=Kyle
> -Phil
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>
> !DSPAM:456db371123088992556831!
>


-- 
Kyle Schochenmaier
kschoche at scl.ameslab.gov
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory