[Pvfs2-developers] TroveSyncData settings

[Phil Carns]

> One thing that we noticed while testing for storage challenge was that 
> (and everyone correct me if I'm wrong here) enabling the data-sync 
> causes a flush/sync to occur after every sizeof(FlowBuffer) bytes had 
> been written.  I can imagine how this would help a SAN, but I'm 
> perplexed how it helps localdisk, what buffer size are you playing with?
> We found that unless we were using HUGE (~size of cache on storage 
> controller) flowbuffers that this caused way too many syncs/seeks on the 
> disks and hurt performance quite a bit, maybe even as bad as 50% 
> performance because things were not being optimized for our disk 
> subsystems and we were issuing many small ops instead of fewer large ones.
> 
> Granted I havent been able to get 2.6.0 building properly yet to test 
> the latest out, but this was definitely the case for us on the 2.5 
> releases.

You are definitely right about the data sync option causing a flush/sync 
on every sizeof(FLowBuffer).  I don't really have a good explanation for 
why this doesn't seem to burn us anymore on local disk.  Our settings 
are standard, except for:

- 512KB flow buffer size
- alt aio method
- 512KB tcp buffers (with larger /proc tcp settings)

This testing was done on some version prior to 2.6.0 also (I think it 
was a merge of some in-between release, so it is hard to pin down a 
version number).

It may also have something to do with the controller and local disks 
being used?  All of our local disk configurations are actually hardware 
raid 5 with some variety of the megaraid controller, and these are 
fairly new boxes.

-Phil