[PVFS2-developers] latest mpi-io-test numbers on jazz

[Robert Latham]

A little bit ago I queued up a bunch of mpi-io-test jobs on jazz over
the myrinet network and they finally finished.  

http://www.mcs.anl.gov/~robl/pvfs2/jazz-20050308/

[The -eb versions of the plots in that directory show the high, low,
and average over multiple runs (4 or 5 i think, depending on how many
jobs finished before Jazz recently had an "unscheduled maintenance" :> ]

As always for these mpi-io-test runs, these read numbers are with warm
caches.  Given that favorable condition, we peaked out at about 8.5
GiB/sec (128 servers and 100 clients).   

The big knee at 8 servers and 100 clients shouldn't be too alarming.
That's the point where the clients working set exceeds the servers'
memory (and caches).

Write numbers look rather unimpressive as usual.  The disks on Jazz
compute nodes are nothing special.  The recent system software upgrade
means we can compare the performance impact of TroveSyncMeta and
TroveSyncData.  jazz-gm-write.png is with those config file options
set to 'yes'.  jazz-gm-write-nosync is with them set to 'no'.  The
mpi-io-test program syncs after every MPI_File_write, so we are
still pushing data to disk, just not as often.   Rough calculations
give us 18 MiB/sec per server with TroveSync{Meta,Data} set to 'yes', 
but 23 MiB/sec per serfer with those options set to 'no'.   The
TroveSync options did not appear to have any impact on read
performance (nor would one expect to see such an impact).

The biggest change between this run and the previous run in September
is that Jazz was recently upgraded to Redhat Enterprise Linux 3.
pvfs2-server now gets to take advantage of working AIO callbacks in
glibc.  Just as Neill asserted, aio callbacks make a significant
difference in performance. 

For comparison, here are the old plots:
http://www-unix.mcs.anl.gov/~robl/pvfs2/jazz_gm.read-20040928.png
http://www-unix.mcs.anl.gov/~robl/pvfs2/jazz_gm.write-20040928.png

I guess I should write this up in a little more coherent manner but
I'm somewhat pressed for time ... let me know if you would like
anything explained in further detail.  

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA                B29D F333 664A 4280 315B