[Pvfs2-users] PVFS2 over Infiniband
Dardo D Kleiner - CONTRACTOR
dkleiner at cmf.nrl.navy.mil
Thu Mar 13 12:14:43 EST 2008
Hi folks! New to the list, trying to get PVFS2 running on our
IB fabric and hitting some problems (but also some "solutions").
A recent poster (Florin Isaila) was having problems with
ib_create_cq failures that I encountered as well, and I managed
to find a workaround though I don't really understand the cause.
We're using OFED 1.2 (with some minor mods by our IB gurus)
running on SLES10SP1 (kernel version 2.6.16.54-0.2.3-smp).
By tracing the call down through libibverbs, I discovered that
the call path went into some sort of compatibility layer in
OFED and ended up hitting __ibv_create_cq_1_0 (in
src/userspace/libibverbs/compat-1_0.c) instead of the
__ibv_create_cq function in src/userspace/libibverbs/src/verbs.c.
I was able to "correct" this by forcibly linking libpvfs2.so
with libibverbs, so there's likely some symbol versioning black
magic going on here. Perhaps Florin could confirm this to be
his problem as well.
So, I get it up and running (VFS interface and all), and
quickly hit a failure trying to do a basic
"dd if=/dev/zero of=testfile bs=4M count=10000" on my pvfs2
mount. Smaller tests seem to work ok (e.g. count=100).
Attached is the pvfs-client.log output - and before I try to get
our IB guys involved I wanted to see if anything jumped out to
the PVFS2 developer community (perhaps BMI related?) or if I
could get some help debugging it further.
I've got 6 I/O servers and 1 metadata server, pvfs2 storage
is on SRP-based LUNs on a DDN array, fs.conf is also attached.
Oh, and my pvfs2 configure options:
./configure --prefix=/afs/ld/software/sys \
--with-openib=/usr \
--with-openib-libs=/usr/lib64 \
--with-kernel=/usr/src/linux-2.6.16.54-0.2.3 \
--enable-shared --enable-trusted-connections \
--enable-mmap-racache --without-bmi-tcp
TIA!
- Dardo
-------------- next part --------------
[D 15:46:56.524470] [INFO]: Mapping pointer 0x2b09efdf4000 for I/O.
[D 15:46:56.550917] [INFO]: Mapping pointer 0x2b09f11f6000 for I/O.
[E 15:49:41.383093] fp_multiqueue_cancel: flow proto cancel called on 0x5acfe8
[E 15:49:41.383136] handle_io_error: flow proto error cleanup started on 0x5acfe8, error_code: -1610613121
[E 15:49:41.383241] handle_io_error: flow proto 0x5acfe8 canceled 1 operations, will clean up.
[E 15:49:41.383802] handle_io_error: flow proto 0x5acfe8 error cleanup finished, error_code: -1610613121
[E 15:49:41.403959] fp_multiqueue_cancel: flow proto cancel called on 0x5aae38
[E 15:49:41.403978] handle_io_error: flow proto error cleanup started on 0x5aae38, error_code: -1610613121
[E 15:49:41.404054] handle_io_error: flow proto 0x5aae38 canceled 1 operations, will clean up.
[E 15:49:41.404465] handle_io_error: flow proto 0x5aae38 error cleanup finished, error_code: -1610613121
[E 15:49:41.406991] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.406999] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.407057] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.407450] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.407883] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.407889] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.407962] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.408339] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.452563] fp_multiqueue_cancel: flow proto cancel called on 0x5abd48
[E 15:49:41.452592] handle_io_error: flow proto error cleanup started on 0x5abd48, error_code: -1610613121
[E 15:49:41.452663] handle_io_error: flow proto 0x5abd48 canceled 1 operations, will clean up.
[E 15:49:41.453081] handle_io_error: flow proto 0x5abd48 error cleanup finished, error_code: -1610613121
[E 15:49:41.475046] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.475063] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.475134] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.475545] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.477027] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.477034] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.477089] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.477452] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.519901] fp_multiqueue_cancel: flow proto cancel called on 0x5abd48
[E 15:49:41.519923] handle_io_error: flow proto error cleanup started on 0x5abd48, error_code: -1610613121
[E 15:49:41.520016] handle_io_error: flow proto 0x5abd48 canceled 1 operations, will clean up.
[E 15:49:41.520422] handle_io_error: flow proto 0x5abd48 error cleanup finished, error_code: -1610613121
[E 15:49:41.542756] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.542773] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.542908] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.543307] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.544543] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.544550] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.544615] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.545005] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.585183] fp_multiqueue_cancel: flow proto cancel called on 0x5abd48
[E 15:49:41.585208] handle_io_error: flow proto error cleanup started on 0x5abd48, error_code: -1610613121
[E 15:49:41.585283] handle_io_error: flow proto 0x5abd48 canceled 1 operations, will clean up.
[E 15:49:41.585697] handle_io_error: flow proto 0x5abd48 error cleanup finished, error_code: -1610613121
[E 15:49:41.608105] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.608124] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.608224] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.608635] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.609867] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.609873] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.609925] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.610316] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.650736] fp_multiqueue_cancel: flow proto cancel called on 0x5abd48
[E 15:49:41.650760] handle_io_error: flow proto error cleanup started on 0x5abd48, error_code: -1610613121
[E 15:49:41.650829] handle_io_error: flow proto 0x5abd48 canceled 1 operations, will clean up.
[E 15:49:41.651235] handle_io_error: flow proto 0x5abd48 error cleanup finished, error_code: -1610613121
[E 15:49:41.673691] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.673708] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.673780] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.674214] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.675311] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.675318] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.675376] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.675750] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.717098] fp_multiqueue_cancel: flow proto cancel called on 0x5abd48
[E 15:49:41.717120] handle_io_error: flow proto error cleanup started on 0x5abd48, error_code: -1610613121
[E 15:49:41.717189] handle_io_error: flow proto 0x5abd48 canceled 1 operations, will clean up.
[E 15:49:41.717611] handle_io_error: flow proto 0x5abd48 error cleanup finished, error_code: -1610613121
[E 15:49:41.739715] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.739732] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.739804] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.740228] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.741406] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.741413] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.741471] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.741855] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.782970] fp_multiqueue_cancel: flow proto cancel called on 0x5abd48
[E 15:49:41.782993] handle_io_error: flow proto error cleanup started on 0x5abd48, error_code: -1610613121
[E 15:49:41.783070] handle_io_error: flow proto 0x5abd48 canceled 1 operations, will clean up.
[E 15:49:41.783517] handle_io_error: flow proto 0x5abd48 error cleanup finished, error_code: -1610613121
[E 15:49:41.807792] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.807806] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.807877] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.808281] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.809356] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.809363] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.809424] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.809806] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.850968] fp_multiqueue_cancel: flow proto cancel called on 0x5abd48
[E 15:49:41.850994] handle_io_error: flow proto error cleanup started on 0x5abd48, error_code: -1610613121
[E 15:49:41.851064] handle_io_error: flow proto 0x5abd48 canceled 1 operations, will clean up.
[E 15:49:41.851491] handle_io_error: flow proto 0x5abd48 error cleanup finished, error_code: -1610613121
[E 15:49:41.870892] fp_multiqueue_cancel: flow proto cancel called on 0x5aae38
[E 15:49:41.870910] handle_io_error: flow proto error cleanup started on 0x5aae38, error_code: -1610613121
[E 15:49:41.870979] handle_io_error: flow proto 0x5aae38 canceled 1 operations, will clean up.
[E 15:49:41.871397] handle_io_error: flow proto 0x5aae38 error cleanup finished, error_code: -1610613121
[E 15:49:41.875071] fp_multiqueue_cancel: flow proto cancel called on 0x5ab5c0
[E 15:49:41.875078] handle_io_error: flow proto error cleanup started on 0x5ab5c0, error_code: -1610613121
[E 15:49:41.875136] handle_io_error: flow proto 0x5ab5c0 canceled 1 operations, will clean up.
[E 15:49:41.875499] handle_io_error: flow proto 0x5ab5c0 error cleanup finished, error_code: -1610613121
[E 15:49:41.876257] fp_multiqueue_cancel: flow proto cancel called on 0x5aa6b0
[E 15:49:41.876276] handle_io_error: flow proto error cleanup started on 0x5aa6b0, error_code: -1610613121
[E 15:49:41.876334] handle_io_error: flow proto 0x5aa6b0 canceled 1 operations, will clean up.
[E 15:49:41.876716] handle_io_error: flow proto 0x5aa6b0 error cleanup finished, error_code: -1610613121
[E 15:49:41.921704] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:41.921731] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:41.921809] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:41.922220] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:41.944486] fp_multiqueue_cancel: flow proto cancel called on 0x5aa680
[E 15:49:41.944505] handle_io_error: flow proto error cleanup started on 0x5aa680, error_code: -1610613121
[E 15:49:41.944573] handle_io_error: flow proto 0x5aa680 canceled 1 operations, will clean up.
[E 15:49:41.944977] handle_io_error: flow proto 0x5aa680 error cleanup finished, error_code: -1610613121
[E 15:49:41.946031] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:41.946038] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:41.946095] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:41.946481] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:41.987841] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:41.987865] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:41.987931] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:41.988361] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.010309] fp_multiqueue_cancel: flow proto cancel called on 0x5aa680
[E 15:49:42.010324] handle_io_error: flow proto error cleanup started on 0x5aa680, error_code: -1610613121
[E 15:49:42.010396] handle_io_error: flow proto 0x5aa680 canceled 1 operations, will clean up.
[E 15:49:42.010800] handle_io_error: flow proto 0x5aa680 error cleanup finished, error_code: -1610613121
[E 15:49:42.011978] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.011985] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.012045] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.012411] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.052851] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.052894] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.052990] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.053433] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.075337] fp_multiqueue_cancel: flow proto cancel called on 0x5aa680
[E 15:49:42.075357] handle_io_error: flow proto error cleanup started on 0x5aa680, error_code: -1610613121
[E 15:49:42.075457] handle_io_error: flow proto 0x5aa680 canceled 1 operations, will clean up.
[E 15:49:42.075880] handle_io_error: flow proto 0x5aa680 error cleanup finished, error_code: -1610613121
[E 15:49:42.077074] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.077081] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.077138] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.077533] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.117909] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.117935] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.118009] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.118409] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.138176] fp_multiqueue_cancel: flow proto cancel called on 0x5aae08
[E 15:49:42.138192] handle_io_error: flow proto error cleanup started on 0x5aae08, error_code: -1610613121
[E 15:49:42.138259] handle_io_error: flow proto 0x5aae08 canceled 1 operations, will clean up.
[E 15:49:42.138666] handle_io_error: flow proto 0x5aae08 error cleanup finished, error_code: -1610613121
[E 15:49:42.140695] fp_multiqueue_cancel: flow proto cancel called on 0x5aa680
[E 15:49:42.140702] handle_io_error: flow proto error cleanup started on 0x5aa680, error_code: -1610613121
[E 15:49:42.140761] handle_io_error: flow proto 0x5aa680 canceled 1 operations, will clean up.
[E 15:49:42.141156] handle_io_error: flow proto 0x5aa680 error cleanup finished, error_code: -1610613121
[E 15:49:42.142372] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.142378] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.142439] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.142804] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.184266] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.184293] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.184362] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.184786] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.206859] fp_multiqueue_cancel: flow proto cancel called on 0x5aa680
[E 15:49:42.206874] handle_io_error: flow proto error cleanup started on 0x5aa680, error_code: -1610613121
[E 15:49:42.207036] handle_io_error: flow proto 0x5aa680 canceled 1 operations, will clean up.
[E 15:49:42.207470] handle_io_error: flow proto 0x5aa680 error cleanup finished, error_code: -1610613121
[E 15:49:42.208792] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.208799] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.208860] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.209242] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.248403] fp_multiqueue_cancel: flow proto cancel called on 0x5a9ef8
[E 15:49:42.248421] handle_io_error: flow proto error cleanup started on 0x5a9ef8, error_code: -1610613121
[E 15:49:42.248493] handle_io_error: flow proto 0x5a9ef8 canceled 1 operations, will clean up.
[E 15:49:42.248960] handle_io_error: flow proto 0x5a9ef8 error cleanup finished, error_code: -1610613121
[E 15:49:42.249342] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.249349] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.249406] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.249786] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.270638] fp_multiqueue_cancel: flow proto cancel called on 0x5aae08
[E 15:49:42.270657] handle_io_error: flow proto error cleanup started on 0x5aae08, error_code: -1610613121
[E 15:49:42.270730] handle_io_error: flow proto 0x5aae08 canceled 1 operations, will clean up.
[E 15:49:42.271161] handle_io_error: flow proto 0x5aae08 error cleanup finished, error_code: -1610613121
[E 15:49:42.273868] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.273875] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.273954] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.274338] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.314154] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.314179] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.314267] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.314696] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.314720] fp_multiqueue_cancel: flow proto cancel called on 0x5a9ef8
[E 15:49:42.314729] handle_io_error: flow proto error cleanup started on 0x5a9ef8, error_code: -1610613121
[E 15:49:42.314779] handle_io_error: flow proto 0x5a9ef8 canceled 1 operations, will clean up.
[E 15:49:42.315163] handle_io_error: flow proto 0x5a9ef8 error cleanup finished, error_code: -1610613121
[E 15:49:42.337925] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.337945] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.338015] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.338438] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.378832] fp_multiqueue_cancel: flow proto cancel called on 0x5a9ef8
[E 15:49:42.378855] handle_io_error: flow proto error cleanup started on 0x5a9ef8, error_code: -1610613121
[E 15:49:42.378998] handle_io_error: flow proto 0x5a9ef8 canceled 1 operations, will clean up.
[E 15:49:42.379468] handle_io_error: flow proto 0x5a9ef8 error cleanup finished, error_code: -1610613121
[E 15:49:42.380417] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.380424] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.380481] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.380856] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.402216] fp_multiqueue_cancel: flow proto cancel called on 0x5aae08
[E 15:49:42.402234] handle_io_error: flow proto error cleanup started on 0x5aae08, error_code: -1610613121
[E 15:49:42.402302] handle_io_error: flow proto 0x5aae08 canceled 1 operations, will clean up.
[E 15:49:42.402738] handle_io_error: flow proto 0x5aae08 error cleanup finished, error_code: -1610613121
[E 15:49:42.405095] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.405101] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.405156] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.405527] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.445422] fp_multiqueue_cancel: flow proto cancel called on 0x5a9ef8
[E 15:49:42.445455] handle_io_error: flow proto error cleanup started on 0x5a9ef8, error_code: -1610613121
[E 15:49:42.445526] handle_io_error: flow proto 0x5a9ef8 canceled 1 operations, will clean up.
[E 15:49:42.445930] handle_io_error: flow proto 0x5a9ef8 error cleanup finished, error_code: -1610613121
[E 15:49:42.446319] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.446326] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.446391] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.446761] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.467709] fp_multiqueue_cancel: flow proto cancel called on 0x5aae08
[E 15:49:42.467731] handle_io_error: flow proto error cleanup started on 0x5aae08, error_code: -1610613121
[E 15:49:42.467798] handle_io_error: flow proto 0x5aae08 canceled 1 operations, will clean up.
[E 15:49:42.468216] handle_io_error: flow proto 0x5aae08 error cleanup finished, error_code: -1610613121
[E 15:49:42.470872] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.470879] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.470937] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.471334] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.511544] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.511568] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.511642] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.512069] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.531830] fp_multiqueue_cancel: flow proto cancel called on 0x5aae08
[E 15:49:42.531847] handle_io_error: flow proto error cleanup started on 0x5aae08, error_code: -1610613121
[E 15:49:42.531919] handle_io_error: flow proto 0x5aae08 canceled 1 operations, will clean up.
[E 15:49:42.532345] handle_io_error: flow proto 0x5aae08 error cleanup finished, error_code: -1610613121
[E 15:49:42.535038] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.535045] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.535103] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.535486] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.575591] fp_multiqueue_cancel: flow proto cancel called on 0x5a9ef8
[E 15:49:42.575618] handle_io_error: flow proto error cleanup started on 0x5a9ef8, error_code: -1610613121
[E 15:49:42.575691] handle_io_error: flow proto 0x5a9ef8 canceled 1 operations, will clean up.
[E 15:49:42.576141] handle_io_error: flow proto 0x5a9ef8 error cleanup finished, error_code: -1610613121
[E 15:49:42.576158] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.576168] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.576214] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.576607] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.597677] fp_multiqueue_cancel: flow proto cancel called on 0x5aae08
[E 15:49:42.597697] handle_io_error: flow proto error cleanup started on 0x5aae08, error_code: -1610613121
[E 15:49:42.597769] handle_io_error: flow proto 0x5aae08 canceled 1 operations, will clean up.
[E 15:49:42.598198] handle_io_error: flow proto 0x5aae08 error cleanup finished, error_code: -1610613121
[E 15:49:42.600934] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.600941] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.601008] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.601396] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.642142] fp_multiqueue_cancel: flow proto cancel called on 0x5abd18
[E 15:49:42.642161] handle_io_error: flow proto error cleanup started on 0x5abd18, error_code: -1610613121
[E 15:49:42.642231] handle_io_error: flow proto 0x5abd18 canceled 1 operations, will clean up.
[E 15:49:42.642645] handle_io_error: flow proto 0x5abd18 error cleanup finished, error_code: -1610613121
[E 15:49:42.654493] fp_multiqueue_cancel: flow proto cancel called on 0x5aae08
[E 15:49:42.654506] handle_io_error: flow proto error cleanup started on 0x5aae08, error_code: -1610613121
[E 15:49:42.654574] handle_io_error: flow proto 0x5aae08 canceled 1 operations, will clean up.
[E 15:49:42.654994] handle_io_error: flow proto 0x5aae08 error cleanup finished, error_code: -1610613121
[E 15:49:42.668858] fp_multiqueue_cancel: flow proto cancel called on 0x5ab590
[E 15:49:42.668871] handle_io_error: flow proto error cleanup started on 0x5ab590, error_code: -1610613121
[E 15:49:42.668939] handle_io_error: flow proto 0x5ab590 canceled 1 operations, will clean up.
[E 15:49:42.669333] handle_io_error: flow proto 0x5ab590 error cleanup finished, error_code: -1610613121
[E 15:49:42.697681] fp_multiqueue_cancel: flow proto cancel called on 0x5aa680
[E 15:49:42.697702] handle_io_error: flow proto error cleanup started on 0x5aa680, error_code: -1610613121
[E 15:49:42.697838] handle_io_error: flow proto 0x5aa680 canceled 1 operations, will clean up.
[E 15:49:42.697884] Error: ib_check_cq: unknown send state SQ_CANCELLED (10) of sq 0x552050.
[E 15:49:42.698381] [bt] /afs/ld/software/sys/lib/libpvfs2.so(error+0xce) [0x2b09ef23f4ae]
[E 15:49:42.698389] [bt] /afs/ld/software/sys/lib/libpvfs2.so [0x2b09ef23c6b1]
[E 15:49:42.698395] [bt] /afs/ld/software/sys/lib/libpvfs2.so [0x2b09ef23e36c]
[E 15:49:42.698400] [bt] /afs/ld/software/sys/lib/libpvfs2.so(BMI_testcontext+0xfa) [0x2b09ef243d6a]
[E 15:49:42.698405] [bt] /afs/ld/software/sys/lib/libpvfs2.so(PINT_thread_mgr_bmi_push+0x169) [0x2b09ef252e39]
[E 15:49:42.698410] [bt] /afs/ld/software/sys/lib/libpvfs2.so [0x2b09ef24e27a]
[E 15:49:42.698415] [bt] /afs/ld/software/sys/lib/libpvfs2.so(job_testcontext+0x106) [0x2b09ef24eab6]
[E 15:49:42.698420] [bt] /afs/ld/software/sys/lib/libpvfs2.so(PINT_client_state_machine_testsome+0x11e) [0x2b09ef206a9e]
[E 15:49:42.698425] [bt] pvfs2-client-core [0x40979f]
[E 15:49:42.698430] [bt] pvfs2-client-core(main+0xa60) [0x40aa30]
[E 15:49:42.698436] [bt] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b09ef874154]
[E 15:49:42.704091] pvfs2-client-core with pid 4754 exited with value 1
-------------- next part --------------
<Defaults>
UnexpectedRequests 50
EventLogging none
LogStamp datetime
BMIModules bmi_ib
FlowModules flowproto_multiqueue
PerfUpdateInterval 1000
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
ClientRetryLimit 5
ClientRetryDelayMilliSecs 2000
StorageSpace /pvfs2/storage
LogFile /var/log/pvfs2-server.log
</Defaults>
<Security>
TrustedPorts 0-65535
TrustedNetwork tcp://xxx.xxx.176.0@21
</Security>
<Aliases>
Alias SiteZ-XE-0 ib://SiteZ-XE-0-ib0:3335
Alias SiteZ-XE-1 ib://SiteZ-XE-1-ib0:3335
Alias SiteZ-XE-2 ib://SiteZ-XE-2-ib0:3335
Alias SiteZ-XE-3 ib://SiteZ-XE-3-ib0:3335
Alias SiteZ-XE-4 ib://SiteZ-XE-4-ib0:3335
Alias SiteZ-XE-5 ib://SiteZ-XE-5-ib0:3335
Alias SiteZ-XE-6 ib://SiteZ-XE-6-ib0:3335
</Aliases>
<Filesystem>
Name pvfs2-fs
ID 2024047019
RootHandle 1048576
<MetaHandleRanges>
Range SiteZ-XE-0 3-1317624576693539402
</MetaHandleRanges>
<DataHandleRanges>
Range SiteZ-XE-1 1317624576693539403-2635249153387078802
Range SiteZ-XE-2 2635249153387078803-3952873730080618202
Range SiteZ-XE-3 3952873730080618203-5270498306774157602
Range SiteZ-XE-4 5270498306774157603-6588122883467697002
Range SiteZ-XE-5 6588122883467697003-7905747460161236402
Range SiteZ-XE-6 7905747460161236403-9223372036854775802
</DataHandleRanges>
<StorageHints>
TroveSyncMeta yes
TroveSyncData no
AttrCacheKeywords dh,md,de,st
AttrCacheSize 4093
AttrCacheMaxNumElems 32768
</StorageHints>
SecretKey xxxxxxxxxxxxxxxxxxxxxxxxxxxx
</Filesystem>
More information about the Pvfs2-users
mailing list