[Pvfs2-developers] TroveSyncData settings

[Phil Carns]

No.  Both alt aio and the normal dbpf method sync as a seperate step 
after the aio list operation completes.

This is technically possible with alt aio, though- you would just need 
to pass a flag through to tell the I/O thread to sync after the 
pwrite().  That would probably be pretty helpful, so the trove worker 
thread doesn't get stuck waiting on the sync...

-Phil


Rob Ross wrote:
> This is similar to using O_DIRECT, which has also shown benefits.
> 
> With alt aio, do we sync in the context of the I/O thread?
> 
> Thanks,
> 
> Rob
> 
> Phil Carns wrote:
> 
>>
>>> One thing that we noticed while testing for storage challenge was 
>>> that (and everyone correct me if I'm wrong here) enabling the 
>>> data-sync causes a flush/sync to occur after every sizeof(FlowBuffer) 
>>> bytes had been written.  I can imagine how this would help a SAN, but 
>>> I'm perplexed how it helps localdisk, what buffer size are you 
>>> playing with?
>>> We found that unless we were using HUGE (~size of cache on storage 
>>> controller) flowbuffers that this caused way too many syncs/seeks on 
>>> the disks and hurt performance quite a bit, maybe even as bad as 50% 
>>> performance because things were not being optimized for our disk 
>>> subsystems and we were issuing many small ops instead of fewer large 
>>> ones.
>>>
>>> Granted I havent been able to get 2.6.0 building properly yet to test 
>>> the latest out, but this was definitely the case for us on the 2.5 
>>> releases.
>>
>>
>> You are definitely right about the data sync option causing a 
>> flush/sync on every sizeof(FLowBuffer).  I don't really have a good 
>> explanation for why this doesn't seem to burn us anymore on local 
>> disk.  Our settings are standard, except for:
>>
>> - 512KB flow buffer size
>> - alt aio method
>> - 512KB tcp buffers (with larger /proc tcp settings)
>>
>> This testing was done on some version prior to 2.6.0 also (I think it 
>> was a merge of some in-between release, so it is hard to pin down a 
>> version number).
>>
>> It may also have something to do with the controller and local disks 
>> being used?  All of our local disk configurations are actually 
>> hardware raid 5 with some variety of the megaraid controller, and 
>> these are fairly new boxes.
>>
>> -Phil
>>
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>>