[Pvfs2-developers] acache timeout

[Phil Carns]

Dean Hildebrand wrote:
> Thanks for the info Phil.  While I'm running an I/O performance 
> experiment, I never want the attributes to expire, so I'm using the -a 
> option to set it to a really high value.
> But for the average pvfs2 setup, what is a useful value?  With 5msec, 
> the cache was continually expiring, and my data servers were contacting 
> the metadata server multiple times when writing a single file (which 
> takes on the order of seconds, if not minutes for large files), just to 
> retrieve the same information over and over.
> 
> I've seen single client file write throughput (with < 10 of data 
> servers) achieve roughly 40-60MB/s (with gigabit ethernet).  So taking a 
> 1G file around 20s to write to disk.  Of course there is no typical 
> setup or typical file size, but maybe a better default acache timeout
> value is in the 10s of seconds?
> Dean

I'm really not sure, to be honest with you.  Someone else may have more 
input.  I ran some limited tests at one point (very different from your 
workload, though) and found that there was a point of diminishing 
returns before the timeout got that high.  At some point the cost of 
occasional attribute retrieval just becomes noise.

I think that in your case you are probably triggering a few getattrs at 
open time, and then 1 getattr every 4 MB of the file (because that is 
the default buffer size the kernel module is breaking the file into).  I 
guess that works out to 10-15 getattr operations per second for your 
case that you could be avoiding.  Reducing the getattr traffic may also 
have a beneficial impact on the server request scheduling too.

-Phil