[Pvfs2-developers] QP Errors when varrying stripe_size over pvfs2-IB

[Kyle Schochenmaier]

Pete -

We've been playing with various FlowBufferSizes on the servers as well 
as varying the stripe_size when opening/modifying files for our 
benchmarking.  I ran across this error that caused our tests to hang, 
not sure where this should go, but it's reproducable with our setup.  I 
was wondering if anyone could tell me if there's an obvious problem with 
this setup:
(using mellanox ddr card)
FlowBufferSize 16MB
stripe_size 256KB
6 data servers

This problem occurs whenever we pick 256KB as a stripe size, however, it 
doesnt show up @ 64KB, or 1M or more (testing 512K right now to see if 
it occurs).  We also noticed that in general using 256KB stripes causes 
weird things, like eHCA errors which bring down the server completely...

(this shows up in the logs of all servers)
[E 10:09:51.709387] Warning: openib_check_async_events: 
IBV_EVENT_QP_ACCESS_ERR.
[E 10:10:22.026509] job_time_mgr_expire: job time out: cancelling flow 
operation
, job_id: 2028006.
[E 10:10:22.026567] fp_multiqueue_cancel: flow proto cancel called on 
0x2aaaabc2
1d20
[E 10:10:22.026578] handle_io_error: flow proto error cleanup started on 
0x2aaaa
bc21d20, error_code: -1610612737
[E 10:10:22.035861] handle_io_error: flow proto 0x2aaaabc21d20 canceled 
1 operat
ions, will clean up.
[E 10:10:22.036661] handle_io_error: flow proto 0x2aaaabc21d20 error 
cleanup fin
ished, error_code: -1610612737

dmesg returns this: (on every server)
ib_mthca 0000:01:00.0: modify QP 3->4 returned status 10.
ib_mthca 0000:01:00.0: modify QP 3->4 returned status 10.
ib_mthca 0000:01:00.0: modify QP 3->4 returned status 10.
ib_mthca 0000:01:00.0: modify QP 3->4 returned status 10.
ib_mthca 0000:01:00.0: modify QP 3->4 returned status 10.

Any ideas?

thanks
    --Kyle

-- 
Kyle Schochenmaier
kschoche at scl.ameslab.gov
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory