[Pvfs2-developers] IB status

[Kyle Schochenmaier]

Pete -
I've been working with the 1.4.0 release and got everything up and 
running, running tests, etc.
Currently I have only one bug which pops up randomly and causes the 
server to kill itself unexpectedly, but I'll hopefully find that soon....
So far I've managed to see a max rate of about 180MB/s from pvfs2-cp.
I havent really seen any single-client data for the vapi-Infiniband stuff.
    How are these results looking?
 - Does anyone have a link to, or data for tests similar to mine over 
the VAPI IB stack? (single-client && single-server)

My setup:
    1 meta/data server, 1 'client' which I'm running the tests from, 
note: not using the pvfs2-client stuff, just pvfs2-* tests.
    1x (meta + data) server on opteron2.   pvfs2-fs mounted to ram (/tmp)
    client tests run from ram (/tmp) on opteron1.
    (all tests here, obviously using the gen2 openIB stack)

64K   = 23MB/s
128K = 59MB/s
256K = 86MB/s
512K = 98.5MB/s
1M    = 115MB/s
2M    = 125.5MB/s
4M    = ( <100MB/s for some reason???)
8M    = 140MB/s
16M  = 142MB/s
32M  = 154MB/s
64M  = 178MB/s
128M = 166MB/s
256M = 161MB/s
(depending on the test 128MB+ will get ~170MB/s on average)

I'm not exactly sure where I'm standing in comparison to the VAPI 
version, nor how this scales yet.  Although I'm working on getting a 
multi-client system setup here hopefully by the end of the week..( 
waiting for a new switch to arrive )

I see some results/graphs from the SC'05 PVFS2 BOF documents, but am 
unsure exactly how everything was setup, and/or how all the clients were 
synchronized to get an accurate test.  Any advice would be helpful as 
well :)


thanks,
    - Kyle

-- 
Kyle Schochenmaier
kschoche at scl.ameslab.gov
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory