[Pvfs2-developers] [Fwd: Molpro details]

Tue Oct 10 14:59:09 EDT 2006

We are seeing another application having issues accessing pvfs2 space 
via the client side mount via the kernel module.  I'm forwarding a 
description of things and looking for help on what info to gather to add 
to this report.  What I have so far is the client and server log output 
from during her runs of these test cases.

Client syslog messages:
> Oct 10 12:40:58 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017704, -- returning -2
> Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017704, -- returning -2
> Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017703, -- returning -2
> Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017703, -- returning -2
> Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017702, -- returning -2
> Oct 10 12:41:00 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017702, -- returning -2
> Oct 10 12:41:00 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017700, -- returning -2
> Oct 10 12:41:02 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017698, -- returning -2
> Oct 10 12:41:04 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017697, -- returning -2
> Oct 10 12:41:06 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017697, -- returning -2
> Oct 10 12:41:06 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017696, -- returning -2
> Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017695, -- returning -2
> Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017695, -- returning -2
> Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017694, -- returning -2
> Oct 10 12:41:08 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017694, -- returning -2
> Oct 10 12:41:08 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017693, -- returning -2
> Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017693, -- returning -2
> Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017692, -- returning -2
> Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017692, -- returning -2
> Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017691, -- returning -2
> Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017691, -- returning -2
> Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017690, -- returning -2
> Oct 10 12:41:10 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017689, -- returning -2
> Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017689, -- returning -2
> Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017688, -- returning -2
> Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017687, -- returning -2
> Oct 10 12:41:12 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017686, -- returning -2
> Oct 10 12:41:12 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017685, -- returning -2
> Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017684, -- returning -2
> Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017684, -- returning -2
> Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017683, -- returning -2
> Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017683, -- returning -2
> Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017682, -- returning -2
> Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017681, -- returning -2
> Oct 10 12:41:15 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017680, -- returning -2
> Oct 10 12:41:16 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017680, -- returning -2
> Oct 10 12:41:16 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017679, -- returning -2
> Oct 10 12:41:17 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017679, -- returning -2
> Oct 10 12:41:17 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017678, -- returning -2
> Oct 10 12:41:18 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017678, -- returning -2
> Oct 10 12:41:18 la099.landscapearch.arches kernel: pvfs2_file_write: error writing to handle 1017677, -- returning -2

Server pvfs2 logs showing last 40 lines from each of the 12 servers:
> [root at southwindow snapshot]# tail -40 ff*/var/log/pvfs*
> ==> ff001/var/log/pvfs2-server.log <==
> [E 01:28:36.025839] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.039854] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.042510] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.269407] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.269550] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.269672] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.565606] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.565808] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:36.566065] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.320512] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.391505] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.391707] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.605731] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.616642] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.643355] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.830465] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.831224] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.831336] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.925518] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.925677] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:37.925791] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:38.268132] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:38.268692] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:38.291768] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:38.511212] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:38.529310] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 01:28:38.529722] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: Key/data pair already exists
> [E 10:38:11.191169] Flow proto error cleanup started on 0x2a9e04b350, error_code: -1073741973
> [E 10:38:11.191884] Flow proto 0x2a9e04b350 canceling a total of 1 BMI or Trove operations
> [E 10:38:11.191910] Flow proto error cleanup finished 0x2a9e04b350, error_code: -1073741973
> [E 20:07:09.563212] ../../../src/pvfs2/1.5.1/src/server/final-response.sm line 136: Error: PINT_encode() failure.
> [E 20:07:09.573061] Flow proto error cleanup started on 0x2aa347b290, error_code: -1073741973
> [E 20:07:09.573157] Flow proto 0x2aa347b290 canceling a total of 1 BMI or Trove operations
> [E 20:07:09.573175] Flow proto error cleanup finished 0x2aa347b290, error_code: -1073741973
> [E 20:07:09.653967]     [bt] /uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server [0x446e8b]
> [E 20:07:09.654136]     [bt] /uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server(main+0x890) [0x40e110]
> [E 20:07:09.654152]     [bt] /lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x2a958b73fb]
> [E 20:07:09.654165]     [bt] /uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server [0x40c6fa]
> [E 20:07:09.654179] Server Response 0x2aa34b0cb8 is of type: PVFS_SERV_GETCONFIG
> [E 20:07:09.654194] FIXME: unimplemented resp type to print
> 
> ==> ff002/var/log/pvfs2-server.log <==
> [E 03:57:44.485617] tried to remove non-existant dataspace
> [E 03:57:44.485707] tried to remove non-existant dataspace
> [E 03:57:44.485781] tried to remove non-existant dataspace
> [E 03:57:44.485870] tried to remove non-existant dataspace
> [E 03:57:44.485959] tried to remove non-existant dataspace
> [E 03:57:44.486058] tried to remove non-existant dataspace
> [E 03:57:44.486134] tried to remove non-existant dataspace
> [E 03:57:44.486224] tried to remove non-existant dataspace
> [E 03:57:44.486313] tried to remove non-existant dataspace
> [E 03:57:44.486390] tried to remove non-existant dataspace
> [E 03:57:44.486467] tried to remove non-existant dataspace
> [E 03:57:44.486554] tried to remove non-existant dataspace
> [E 03:57:44.486641] tried to remove non-existant dataspace
> [E 03:57:44.486740] tried to remove non-existant dataspace
> [E 03:57:44.486826] tried to remove non-existant dataspace
> [E 03:57:44.486913] tried to remove non-existant dataspace
> [E 03:57:44.486986] tried to remove non-existant dataspace
> [E 03:57:44.487074] tried to remove non-existant dataspace
> [E 03:57:44.487163] tried to remove non-existant dataspace
> [E 03:57:44.487251] tried to remove non-existant dataspace
> [E 03:57:44.487338] tried to remove non-existant dataspace
> [E 03:57:44.487436] tried to remove non-existant dataspace
> [E 03:57:44.487512] tried to remove non-existant dataspace
> [E 03:57:44.487611] tried to remove non-existant dataspace
> [E 03:57:44.487710] tried to remove non-existant dataspace
> [E 03:57:44.487784] tried to remove non-existant dataspace
> [E 03:57:44.487868] tried to remove non-existant dataspace
> [E 03:57:44.487954] tried to remove non-existant dataspace
> [E 03:57:44.488041] tried to remove non-existant dataspace
> [E 03:57:44.488139] tried to remove non-existant dataspace
> [E 03:57:44.488216] tried to remove non-existant dataspace
> [E 03:57:44.488301] tried to remove non-existant dataspace
> [E 03:57:44.488399] tried to remove non-existant dataspace
> [E 03:57:44.488475] tried to remove non-existant dataspace
> [E 03:57:44.488568] tried to remove non-existant dataspace
> [E 03:57:44.488655] tried to remove non-existant dataspace
> [E 03:57:44.488753] tried to remove non-existant dataspace
> [E 03:57:44.488840] tried to remove non-existant dataspace
> [E 03:57:44.488927] tried to remove non-existant dataspace
> [E 03:57:44.489001] tried to remove non-existant dataspace
> 
> ==> ff003/var/log/pvfs2-server.log <==
> [E 19:34:11.810075] Flow proto error cleanup finished 0x2a96915f30, error_code: -1610612737
> [E 19:34:11.810170] Flow proto error cleanup started on 0x2a966bd8e0, error_code: -1610612737
> [E 19:34:11.810485] Flow proto 0x2a966bd8e0 canceling a total of 1 BMI or Trove operations
> [E 19:34:11.810541] Flow proto error cleanup finished 0x2a966bd8e0, error_code: -1610612737
> [E 19:34:11.810586] Flow proto error cleanup finished 0x2a96600660, error_code: -1610612737
> [E 19:34:59.872719] Job time out: cancelling flow operation, job_id: 10768593.
> [E 19:34:59.873205] Flow proto cancel called on 0x2a96600160
> [E 19:35:15.284126] Flow proto error cleanup started on 0x2a96600160, error_code: -1610612737
> [E 19:35:15.286646] Flow proto 0x2a96600160 canceling a total of 2 BMI or Trove operations
> [E 19:35:15.286691] Flow proto error cleanup finished 0x2a96600160, error_code: -1610612737
> [E 12:57:45.872203] Job time out: cancelling flow operation, job_id: 174028738.
> [E 12:57:45.872733] Flow proto cancel called on 0x2a98b56d50
> [E 12:58:40.673616] Flow proto error cleanup started on 0x2a98b56d50, error_code: -1610612737
> [E 12:58:40.674631] Flow proto 0x2a98b56d50 canceling a total of 2 BMI or Trove operations
> [E 12:58:40.674713] Flow proto error cleanup finished 0x2a98b56d50, error_code: -1610612737
> [E 11:47:31.847418] Job time out: cancelling flow operation, job_id: 260685652.
> [E 11:47:31.848020] Flow proto cancel called on 0x13957f0
> [E 11:47:59.228124] Flow proto error cleanup started on 0x13957f0, error_code: -1610612737
> [E 11:47:59.228577] Flow proto 0x13957f0 canceling a total of 1 BMI or Trove operations
> [E 11:47:59.228600] Flow proto error cleanup finished 0x13957f0, error_code: -1610612737
> [E 11:50:27.162829]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:43.563953]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 00:37:00.584569] Job time out: cancelling flow operation, job_id: 60716511.
> [E 00:37:00.585797] Flow proto cancel called on 0x2a96f44a90
> [E 00:37:17.511475] Flow proto error cleanup started on 0x2a96f44a90, error_code: -1610612737
> [E 00:37:17.513090] Flow proto 0x2a96f44a90 canceling a total of 2 BMI or Trove operations
> [E 00:37:17.513179] Flow proto error cleanup finished 0x2a96f44a90, error_code: -1610612737
> [E 09:02:15.843391]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 04:44:49.759259] Flow proto error cleanup started on 0x2a9acb6ce0, error_code: -104
> [E 04:44:49.760934] Flow proto 0x2a9acb6ce0 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.760955] Flow proto error cleanup finished 0x2a9acb6ce0, error_code: -104
> [E 10:38:06.672188] Flow proto error cleanup started on 0x29b12a0, error_code: -1073741973
> [E 10:38:06.672815] Flow proto 0x29b12a0 canceling a total of 1 BMI or Trove operations
> [E 10:38:06.672841] Flow proto error cleanup finished 0x29b12a0, error_code: -1073741973
> [E 20:06:54.081971] Flow proto error cleanup started on 0x2aa2fc7240, error_code: -1073741973
> [E 20:06:54.082764] Flow proto 0x2aa2fc7240 canceling a total of 1 BMI or Trove operations
> [E 20:06:54.082791] Flow proto error cleanup finished 0x2aa2fc7240, error_code: -1073741973
> 
> ==> ff004/var/log/pvfs2-server.log <==
> [E 11:40:46.991214] Flow proto cancel called on 0x2a991ae760
> [E 11:40:46.991246] Flow proto error cleanup started on 0x2a991ae760, error_code: -1610612737
> [E 11:41:03.301929] Flow proto 0x2a991ae760 canceling a total of 2 BMI or Trove operations
> [E 11:41:03.304713] Job time out: cancelling flow operation, job_id: 260524129.
> [E 11:41:03.304752] Flow proto cancel called on 0x2a994258c0
> [E 11:41:03.305135] Flow proto error cleanup finished 0x2a991ae760, error_code: -1610612737
> [E 11:41:03.305136] Flow proto error cleanup started on 0x2a994258c0, error_code: -1610612737
> [E 11:41:03.305260] Flow proto 0x2a994258c0 canceling a total of 2 BMI or Trove operations
> [E 11:41:03.305407] Flow proto error cleanup finished 0x2a994258c0, error_code: -1610612737
> [E 11:48:29.583000] Job time out: cancelling flow operation, job_id: 260526158.
> [E 11:48:29.583714] Flow proto cancel called on 0x2a98fc8680
> [E 11:48:37.291564] Flow proto error cleanup started on 0x2a98fc8680, error_code: -1610612737
> [E 11:48:37.336903] Flow proto 0x2a98fc8680 canceling a total of 1 BMI or Trove operations
> [E 11:48:37.366595] Flow proto error cleanup finished 0x2a98fc8680, error_code: -1610612737
> [E 11:50:28.698951]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:45.480138]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:52:10.988318] Flow proto error cleanup started on 0x2a971436e0, error_code: -1073741973
> [E 11:52:10.990092] Flow proto 0x2a971436e0 canceling a total of 0 BMI or Trove operations
> [E 11:52:10.990114] Flow proto error cleanup finished 0x2a971436e0, error_code: -1073741973
> [E 00:37:00.993757] Job time out: cancelling flow operation, job_id: 60633518.
> [E 00:37:00.994276] Flow proto cancel called on 0x2a9730ca90
> [E 00:37:19.107120] Flow proto error cleanup started on 0x2a9730ca90, error_code: -1610612737
> [E 00:37:19.109208] Flow proto 0x2a9730ca90 canceling a total of 2 BMI or Trove operations
> [E 00:37:19.109309] Flow proto error cleanup finished 0x2a9730ca90, error_code: -1610612737
> [E 09:02:18.012373]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 04:44:49.758554] Flow proto error cleanup started on 0x2a9aac1040, error_code: -104
> [E 04:44:49.760161] Flow proto 0x2a9aac1040 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.760182] Flow proto error cleanup finished 0x2a9aac1040, error_code: -104
> [E 10:32:12.887396] Flow proto error cleanup started on 0x24788f0, error_code: -1073741973
> [E 10:32:12.888016] Flow proto 0x24788f0 canceling a total of 0 BMI or Trove operations
> [E 10:32:12.888036] Flow proto error cleanup finished 0x24788f0, error_code: -1073741973
> [E 10:38:16.226774] Flow proto error cleanup started on 0x2a9e01b9d0, error_code: -1073741973
> [E 10:38:16.227377] Flow proto 0x2a9e01b9d0 canceling a total of 1 BMI or Trove operations
> [E 10:38:16.227401] Flow proto error cleanup finished 0x2a9e01b9d0, error_code: -1073741973
> [E 20:07:08.266341] Flow proto error cleanup started on 0x2aa320eec0, error_code: -1073741973
> [E 20:07:08.267061] Flow proto 0x2aa320eec0 canceling a total of 1 BMI or Trove operations
> [E 20:07:08.267086] Flow proto error cleanup finished 0x2aa320eec0, error_code: -1073741973
> 
> ==> ff005/var/log/pvfs2-server.log <==
> [E 19:28:12.878831] Flow proto cancel called on 0x2a96620de0
> [E 19:28:12.878851] Flow proto error cleanup started on 0x2a96620de0, error_code: -1610612737
> [E 19:28:35.501633] Flow proto 0x2a96620de0 canceling a total of 2 BMI or Trove operations
> [E 19:28:35.502569] Flow proto error cleanup finished 0x2a96620de0, error_code: -1610612737
> [E 19:34:59.682130] Job time out: cancelling flow operation, job_id: 10697506.
> [E 19:34:59.682756] Flow proto cancel called on 0x2a965fc1c0
> [E 19:35:02.829823] Flow proto error cleanup started on 0x2a965fc1c0, error_code: -1610612737
> [E 19:35:02.831224] Flow proto 0x2a965fc1c0 canceling a total of 2 BMI or Trove operations
> [E 19:35:02.831323] Flow proto error cleanup finished 0x2a965fc1c0, error_code: -1610612737
> [E 12:57:45.167229] Job time out: cancelling flow operation, job_id: 173801599.
> [E 12:57:45.167954] Flow proto cancel called on 0x2a98aac730
> [E 12:58:42.592382] Flow proto error cleanup started on 0x2a98aac730, error_code: -1610612737
> [E 12:58:42.593366] Flow proto 0x2a98aac730 canceling a total of 2 BMI or Trove operations
> [E 12:58:42.593438] Flow proto error cleanup finished 0x2a98aac730, error_code: -1610612737
> [E 11:50:30.957911]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:47.463113]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 00:37:00.895289] Job time out: cancelling flow operation, job_id: 60714443.
> [E 00:37:00.897106] Flow proto cancel called on 0xd0b1b0
> [E 00:37:17.927882] Flow proto error cleanup started on 0xd0b1b0, error_code: -1610612737
> [E 00:37:17.929563] Flow proto 0xd0b1b0 canceling a total of 2 BMI or Trove operations
> [E 00:37:17.929641] Flow proto error cleanup finished 0xd0b1b0, error_code: -1610612737
> [E 09:02:20.070652]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 04:44:49.759129] Flow proto error cleanup started on 0x2a9b1487f0, error_code: -104
> [E 04:44:49.760142] Flow proto 0x2a9b1487f0 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.760162] Flow proto error cleanup finished 0x2a9b1487f0, error_code: -104
> [E 10:32:12.894177] Flow proto error cleanup started on 0x2a9d08ad90, error_code: -1073741973
> [E 10:32:12.894805] Flow proto 0x2a9d08ad90 canceling a total of 0 BMI or Trove operations
> [E 10:32:12.894823] Flow proto error cleanup finished 0x2a9d08ad90, error_code: -1073741973
> [E 10:38:15.483254] Flow proto error cleanup started on 0x22f0470, error_code: -1073741973
> [E 10:38:15.483885] Flow proto 0x22f0470 canceling a total of 1 BMI or Trove operations
> [E 10:38:15.483919] Flow proto error cleanup finished 0x22f0470, error_code: -1073741973
> [E 20:07:02.904136] Flow proto error cleanup started on 0x2aa3756870, error_code: -1073741973
> [E 20:07:02.904619] Flow proto 0x2aa3756870 canceling a total of 0 BMI or Trove operations
> [E 20:07:02.904642] Flow proto error cleanup finished 0x2aa3756870, error_code: -1073741973
> [E 20:07:02.914124] Flow proto error cleanup started on 0x2aa3738bf0, error_code: -1073741973
> [E 20:07:02.914145] Flow proto 0x2aa3738bf0 canceling a total of 1 BMI or Trove operations
> [E 20:07:02.914160] Flow proto error cleanup finished 0x2aa3738bf0, error_code: -1073741973
> 
> ==> ff006/var/log/pvfs2-server.log <==
> [E 19:01:47.683377]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:31:24.201720]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 08:34:03.161205] Flow proto error cleanup started on 0x2a971354f0, error_code: -104
> [E 08:34:03.162306] Flow proto 0x2a971354f0 canceling a total of 1 BMI or Trove operations
> [E 08:34:03.162326] Flow proto error cleanup finished 0x2a971354f0, error_code: -104
> [E 08:36:52.935164] Job time out: cancelling flow operation, job_id: 43252530.
> [E 08:36:52.935787] Flow proto cancel called on 0x2a97137800
> [E 08:37:00.709266] Flow proto error cleanup started on 0x2a97137800, error_code: -1610612737
> [E 08:37:00.710092] Flow proto 0x2a97137800 canceling a total of 2 BMI or Trove operations
> [E 08:37:00.710171] Flow proto error cleanup finished 0x2a97137800, error_code: -1610612737
> [E 19:34:59.160804] Job time out: cancelling flow operation, job_id: 10909203.
> [E 19:34:59.161705] Flow proto cancel called on 0x2a965e5e00
> [E 19:34:59.161728] Flow proto error cleanup started on 0x2a965e5e00, error_code: -1610612737
> [E 19:35:15.147747] Flow proto 0x2a965e5e00 canceling a total of 2 BMI or Trove operations
> [E 19:35:15.148983] Job time out: cancelling flow operation, job_id: 10909222.
> [E 19:35:15.149005] Flow proto cancel called on 0x2a9622dbd0
> [E 19:35:15.149214] Flow proto error cleanup finished 0x2a965e5e00, error_code: -1610612737
> [E 11:50:32.274652]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:49.400128]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 00:37:00.177509] Job time out: cancelling flow operation, job_id: 60674262.
> [E 00:37:00.178757] Flow proto cancel called on 0x2a97285740
> [E 00:37:18.801059] Flow proto error cleanup started on 0x2a97285740, error_code: -1610612737
> [E 00:37:18.803307] Flow proto 0x2a97285740 canceling a total of 2 BMI or Trove operations
> [E 00:37:18.803392] Flow proto error cleanup finished 0x2a97285740, error_code: -1610612737
> [E 09:02:22.157152]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 04:44:49.758640] Flow proto error cleanup started on 0x1949060, error_code: -104
> [E 04:44:49.759769] Flow proto 0x1949060 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.759790] Flow proto error cleanup finished 0x1949060, error_code: -104
> [E 10:38:07.226797] Flow proto error cleanup started on 0x2216e60, error_code: -1073741973
> [E 10:38:07.227431] Flow proto 0x2216e60 canceling a total of 1 BMI or Trove operations
> [E 10:38:07.227485] Flow proto error cleanup finished 0x2216e60, error_code: -1073741973
> [E 20:07:05.613263] Flow proto error cleanup started on 0x2aa36d34c0, error_code: -1073741973
> [E 20:07:05.614048] Flow proto 0x2aa36d34c0 canceling a total of 1 BMI or Trove operations
> [E 20:07:05.614069] Flow proto error cleanup finished 0x2aa36d34c0, error_code: -1073741973
> 
> ==> ff007/var/log/pvfs2-server.log <==
> [E 11:29:25.541128] Flow proto cancel called on 0x2a98cd0850
> [E 11:30:03.976777] Flow proto error cleanup started on 0x2a98cd0850, error_code: -1610612737
> [E 11:30:03.978901] Flow proto 0x2a98cd0850 canceling a total of 2 BMI or Trove operations
> [E 11:30:03.978962] Flow proto error cleanup finished 0x2a98cd0850, error_code: -1610612737
> [E 19:34:59.671092] Job time out: cancelling flow operation, job_id: 10737005.
> [E 19:34:59.672028] Flow proto cancel called on 0x2a9659c910
> [E 19:35:10.519122] Flow proto error cleanup started on 0x2a9659c910, error_code: -1610612737
> [E 19:35:10.520406] Flow proto 0x2a9659c910 canceling a total of 2 BMI or Trove operations
> [E 19:35:10.520536] Flow proto error cleanup finished 0x2a9659c910, error_code: -1610612737
> [E 12:57:45.661058] Job time out: cancelling flow operation, job_id: 173821587.
> [E 12:57:45.661538] Flow proto cancel called on 0x11f50f0
> [E 12:58:43.551111] Flow proto error cleanup started on 0x11f50f0, error_code: -1610612737
> [E 12:58:43.552166] Flow proto 0x11f50f0 canceling a total of 2 BMI or Trove operations
> [E 12:58:43.552347] Flow proto error cleanup finished 0x11f50f0, error_code: -1610612737
> [E 11:50:33.936269]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:51.724085]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:52:10.992619] Flow proto error cleanup started on 0x2a97091240, error_code: -1073741973
> [E 11:52:10.995148] Flow proto 0x2a97091240 canceling a total of 0 BMI or Trove operations
> [E 11:52:10.995169] Flow proto error cleanup finished 0x2a97091240, error_code: -1073741973
> [E 00:37:00.689602] Job time out: cancelling flow operation, job_id: 60688232.
> [E 00:37:00.690275] Flow proto cancel called on 0x2a97604c80
> [E 00:37:18.944316] Flow proto error cleanup started on 0x2a97604c80, error_code: -1610612737
> [E 00:37:18.946217] Flow proto 0x2a97604c80 canceling a total of 2 BMI or Trove operations
> [E 00:37:18.946300] Flow proto error cleanup finished 0x2a97604c80, error_code: -1610612737
> [E 09:02:24.269926]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:14:02.832222] Flow proto error cleanup started on 0x2a98040310, error_code: -104
> [E 11:14:02.833211] Flow proto 0x2a98040310 canceling a total of 0 BMI or Trove operations
> [E 11:14:02.833232] Flow proto error cleanup finished 0x2a98040310, error_code: -104
> [E 04:44:49.757671] Flow proto error cleanup started on 0x1920720, error_code: -104
> [E 04:44:49.759001] Flow proto 0x1920720 canceling a total of 0 BMI or Trove operations
> [E 04:44:49.759020] Flow proto error cleanup finished 0x1920720, error_code: -104
> [E 10:38:13.450369] Flow proto error cleanup started on 0x2a9df8ff20, error_code: -1073741973
> [E 10:38:13.450973] Flow proto 0x2a9df8ff20 canceling a total of 1 BMI or Trove operations
> [E 10:38:13.451011] Flow proto error cleanup finished 0x2a9df8ff20, error_code: -1073741973
> [E 20:06:46.818453] Flow proto error cleanup started on 0x2aa352fca0, error_code: -1073741973
> [E 20:06:46.819234] Flow proto 0x2aa352fca0 canceling a total of 1 BMI or Trove operations
> [E 20:06:46.819258] Flow proto error cleanup finished 0x2aa352fca0, error_code: -1073741973
> 
> ==> ff008/var/log/pvfs2-server.log <==
> [E 11:23:15.053985] Job time out: cancelling flow operation, job_id: 92899743.
> [E 11:23:15.054655] Flow proto cancel called on 0x175fc80
> [E 11:23:55.566371] Flow proto error cleanup started on 0x175fc80, error_code: -1610612737
> [E 11:23:55.567591] Flow proto 0x175fc80 canceling a total of 2 BMI or Trove operations
> [E 11:23:55.567683] Flow proto error cleanup finished 0x175fc80, error_code: -1610612737
> [E 11:29:25.334361] Job time out: cancelling flow operation, job_id: 92911807.
> [E 11:29:25.334988] Flow proto cancel called on 0x1744720
> [E 11:30:02.090680] Flow proto error cleanup started on 0x1744720, error_code: -1610612737
> [E 11:30:02.092233] Flow proto 0x1744720 canceling a total of 2 BMI or Trove operations
> [E 11:30:02.092307] Flow proto error cleanup finished 0x1744720, error_code: -1610612737
> [E 19:34:59.158017] Job time out: cancelling flow operation, job_id: 10839796.
> [E 19:34:59.158967] Flow proto cancel called on 0x2a966d75d0
> [E 19:35:16.526547] Flow proto error cleanup started on 0x2a966d75d0, error_code: -1610612737
> [E 19:35:16.528973] Flow proto 0x2a966d75d0 canceling a total of 2 BMI or Trove operations
> [E 19:35:16.529032] Flow proto error cleanup finished 0x2a966d75d0, error_code: -1610612737
> [E 12:57:45.865585] Job time out: cancelling flow operation, job_id: 173982292.
> [E 12:57:45.866208] Flow proto cancel called on 0x11a2500
> [E 12:58:41.864228] Flow proto error cleanup started on 0x11a2500, error_code: -1610612737
> [E 12:58:41.865154] Flow proto 0x11a2500 canceling a total of 2 BMI or Trove operations
> [E 12:58:41.865225] Flow proto error cleanup finished 0x11a2500, error_code: -1610612737
> [E 11:50:36.038756]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:53.861118]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 00:37:00.184936] Job time out: cancelling flow operation, job_id: 60608771.
> [E 00:37:00.186029] Flow proto cancel called on 0xd22e60
> [E 00:37:18.149785] Flow proto error cleanup started on 0xd22e60, error_code: -1610612737
> [E 00:37:18.151685] Flow proto 0xd22e60 canceling a total of 2 BMI or Trove operations
> [E 00:37:18.151774] Flow proto error cleanup finished 0xd22e60, error_code: -1610612737
> [E 09:02:26.320790]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 04:44:49.757312] Flow proto error cleanup started on 0x2a9a9939e0, error_code: -104
> [E 04:44:49.758739] Flow proto 0x2a9a9939e0 canceling a total of 0 BMI or Trove operations
> [E 04:44:49.758758] Flow proto error cleanup finished 0x2a9a9939e0, error_code: -104
> [E 10:38:03.863313] Flow proto error cleanup started on 0x2f580a0, error_code: -1073741973
> [E 10:38:03.863920] Flow proto 0x2f580a0 canceling a total of 1 BMI or Trove operations
> [E 10:38:03.863942] Flow proto error cleanup finished 0x2f580a0, error_code: -1073741973
> [E 20:07:12.494161] Flow proto error cleanup started on 0x2aa26f24a0, error_code: -1073741973
> [E 20:07:12.494667] Flow proto 0x2aa26f24a0 canceling a total of 1 BMI or Trove operations
> [E 20:07:12.494691] Flow proto error cleanup finished 0x2aa26f24a0, error_code: -1073741973
> 
> ==> ff009/var/log/pvfs2-server.log <==
> [E 08:36:59.582120] Flow proto 0xd455b0 canceling a total of 2 BMI or Trove operations
> [E 08:36:59.582230] Flow proto error cleanup finished 0xd455b0, error_code: -1610612737
> [E 11:29:25.131942] Job time out: cancelling flow operation, job_id: 93337739.
> [E 11:29:25.132558] Flow proto cancel called on 0x2a98cf6de0
> [E 11:30:04.808605] Flow proto error cleanup started on 0x2a98cf6de0, error_code: -1610612737
> [E 11:30:04.810303] Flow proto 0x2a98cf6de0 canceling a total of 2 BMI or Trove operations
> [E 11:30:04.810377] Flow proto error cleanup finished 0x2a98cf6de0, error_code: -1610612737
> [E 19:34:59.875193] Job time out: cancelling flow operation, job_id: 10673674.
> [E 19:34:59.876212] Flow proto cancel called on 0x8bf5c0
> [E 19:35:15.284127] Flow proto error cleanup started on 0x8bf5c0, error_code: -1610612737
> [E 19:35:15.287235] Flow proto 0x8bf5c0 canceling a total of 2 BMI or Trove operations
> [E 19:35:15.287309] Flow proto error cleanup finished 0x8bf5c0, error_code: -1610612737
> [E 12:57:45.762669] Job time out: cancelling flow operation, job_id: 173824052.
> [E 12:57:45.763290] Flow proto cancel called on 0x2a98a6b4a0
> [E 12:58:43.317763] Flow proto error cleanup started on 0x2a98a6b4a0, error_code: -1610612737
> [E 12:58:43.318778] Flow proto 0x2a98a6b4a0 canceling a total of 2 BMI or Trove operations
> [E 12:58:43.318851] Flow proto error cleanup finished 0x2a98a6b4a0, error_code: -1610612737
> [E 11:50:38.093306]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:55.493550]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:52:11.051873] Flow proto error cleanup started on 0x2a971567e0, error_code: -1073741973
> [E 11:52:11.052839] Flow proto 0x2a971567e0 canceling a total of 0 BMI or Trove operations
> [E 11:52:11.052858] Flow proto error cleanup finished 0x2a971567e0, error_code: -1073741973
> [E 00:37:00.609802] Job time out: cancelling flow operation, job_id: 60604133.
> [E 00:37:00.610308] Flow proto cancel called on 0x2a9734b890
> [E 00:37:21.082539] Flow proto error cleanup started on 0x2a9734b890, error_code: -1610612737
> [E 00:37:21.084859] Flow proto 0x2a9734b890 canceling a total of 2 BMI or Trove operations
> [E 00:37:21.084975] Flow proto error cleanup finished 0x2a9734b890, error_code: -1610612737
> [E 09:02:28.364450]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 04:44:49.757475] Flow proto error cleanup started on 0x2a9abc37c0, error_code: -104
> [E 04:44:49.758771] Flow proto 0x2a9abc37c0 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.758791] Flow proto error cleanup finished 0x2a9abc37c0, error_code: -104
> [E 10:38:09.247852] Flow proto error cleanup started on 0x2a9db1e680, error_code: -1073741973
> [E 10:38:09.248487] Flow proto 0x2a9db1e680 canceling a total of 1 BMI or Trove operations
> [E 10:38:09.248525] Flow proto error cleanup finished 0x2a9db1e680, error_code: -1073741973
> [E 20:07:03.036272] Flow proto error cleanup started on 0x2aa260b930, error_code: -1073741973
> [E 20:07:03.037042] Flow proto 0x2aa260b930 canceling a total of 1 BMI or Trove operations
> [E 20:07:03.037067] Flow proto error cleanup finished 0x2aa260b930, error_code: -1073741973
> 
> ==> ff010/var/log/pvfs2-server.log <==
> [E 12:57:45.662629] Flow proto cancel called on 0x2a98c771f0
> [E 12:58:37.177711] Flow proto error cleanup started on 0x2a98c771f0, error_code: -1610612737
> [E 12:58:37.178690] Flow proto 0x2a98c771f0 canceling a total of 2 BMI or Trove operations
> [E 12:58:37.178771] Flow proto error cleanup finished 0x2a98c771f0, error_code: -1610612737
> [E 11:41:06.579585] Job time out: cancelling flow operation, job_id: 261075524.
> [E 11:41:06.580350] Flow proto cancel called on 0x12a1550
> [E 11:41:17.661067] Flow proto error cleanup started on 0x12a1550, error_code: -1610612737
> [E 11:41:17.663997] Flow proto 0x12a1550 canceling a total of 2 BMI or Trove operations
> [E 11:41:17.664514] Flow proto error cleanup finished 0x12a1550, error_code: -1610612737
> [E 11:47:05.893222] Job time out: cancelling flow operation, job_id: 261077518.
> [E 11:47:05.893890] Flow proto cancel called on 0x1283ba0
> [E 11:47:05.893926] Flow proto error cleanup started on 0x1283ba0, error_code: -1610612737
> [E 11:48:23.943123] Flow proto 0x1283ba0 canceling a total of 1 BMI or Trove operations
> [E 11:48:23.943696] Flow proto error cleanup finished 0x1283ba0, error_code: -1610612737
> [E 11:50:40.247367]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:25:58.400013]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:52:11.007555] Flow proto error cleanup started on 0x2a96eefff0, error_code: -1073741973
> [E 11:52:11.008692] Flow proto 0x2a96eefff0 canceling a total of 0 BMI or Trove operations
> [E 11:52:11.008712] Flow proto error cleanup finished 0x2a96eefff0, error_code: -1073741973
> [E 00:37:00.590231] Job time out: cancelling flow operation, job_id: 60722692.
> [E 00:37:00.590849] Flow proto cancel called on 0x2a97326e90
> [E 00:37:19.214512] Flow proto error cleanup started on 0x2a97326e90, error_code: -1610612737
> [E 00:37:19.217037] Flow proto 0x2a97326e90 canceling a total of 2 BMI or Trove operations
> [E 00:37:19.217121] Flow proto error cleanup finished 0x2a97326e90, error_code: -1610612737
> [E 09:02:30.496085]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:14:02.909857] Flow proto error cleanup started on 0xd83060, error_code: -1073741973
> [E 11:14:02.910777] Flow proto 0xd83060 canceling a total of 0 BMI or Trove operations
> [E 11:14:02.910797] Flow proto error cleanup finished 0xd83060, error_code: -1073741973
> [E 04:44:49.757205] Flow proto error cleanup started on 0x1902ef0, error_code: -104
> [E 04:44:49.758324] Flow proto 0x1902ef0 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.758344] Flow proto error cleanup finished 0x1902ef0, error_code: -104
> [E 10:38:12.786615] Flow proto error cleanup started on 0x2a9e17bd40, error_code: -1073741973
> [E 10:38:12.787231] Flow proto 0x2a9e17bd40 canceling a total of 1 BMI or Trove operations
> [E 10:38:12.787269] Flow proto error cleanup finished 0x2a9e17bd40, error_code: -1073741973
> [E 20:06:58.044763] Flow proto error cleanup started on 0x2aa37d9600, error_code: -1073741973
> [E 20:06:58.045488] Flow proto 0x2aa37d9600 canceling a total of 1 BMI or Trove operations
> [E 20:06:58.045512] Flow proto error cleanup finished 0x2aa37d9600, error_code: -1073741973
> 
> ==> ff011/var/log/pvfs2-server.log <==
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 08:34:03.160140] Flow proto error cleanup started on 0x2a971ea2b0, error_code: -104
> [E 08:34:03.160818] Flow proto 0x2a971ea2b0 canceling a total of 1 BMI or Trove operations
> [E 08:34:03.160838] Flow proto error cleanup finished 0x2a971ea2b0, error_code: -104
> [E 08:36:52.014916] Job time out: cancelling flow operation, job_id: 43584119.
> [E 08:36:52.015589] Flow proto cancel called on 0xd7e710
> [E 08:36:56.733708] Flow proto error cleanup started on 0xd7e710, error_code: -1610612737
> [E 08:36:56.734636] Flow proto 0xd7e710 canceling a total of 2 BMI or Trove operations
> [E 08:36:56.734749] Flow proto error cleanup finished 0xd7e710, error_code: -1610612737
> [E 11:29:25.143922] Job time out: cancelling flow operation, job_id: 93291033.
> [E 11:29:25.144571] Flow proto cancel called on 0x2a99107a90
> [E 11:30:03.333847] Flow proto error cleanup started on 0x2a99107a90, error_code: -1610612737
> [E 11:30:03.335664] Flow proto 0x2a99107a90 canceling a total of 2 BMI or Trove operations
> [E 11:30:03.335726] Flow proto error cleanup finished 0x2a99107a90, error_code: -1610612737
> [E 12:57:45.882066] Job time out: cancelling flow operation, job_id: 174098170.
> [E 12:57:45.882894] Flow proto cancel called on 0x2a988d06d0
> [E 12:58:44.439322] Flow proto error cleanup started on 0x2a988d06d0, error_code: -1610612737
> [E 12:58:44.440301] Flow proto 0x2a988d06d0 canceling a total of 2 BMI or Trove operations
> [E 12:58:44.440381] Flow proto error cleanup finished 0x2a988d06d0, error_code: -1610612737
> [E 11:50:42.315431]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:26:00.403797]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:52:11.008605] Flow proto error cleanup started on 0x2a98154cc0, error_code: -1073741973
> [E 11:52:11.010067] Flow proto 0x2a98154cc0 canceling a total of 0 BMI or Trove operations
> [E 11:52:11.010087] Flow proto error cleanup finished 0x2a98154cc0, error_code: -1073741973
> [E 09:02:32.419305]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:14:02.910124] Flow proto error cleanup started on 0x2a98297510, error_code: -1073741973
> [E 11:14:02.911096] Flow proto 0x2a98297510 canceling a total of 0 BMI or Trove operations
> [E 11:14:02.911115] Flow proto error cleanup finished 0x2a98297510, error_code: -1073741973
> [E 04:44:49.758449] Flow proto error cleanup started on 0x2a9af08d50, error_code: -104
> [E 04:44:49.759700] Flow proto 0x2a9af08d50 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.759720] Flow proto error cleanup finished 0x2a9af08d50, error_code: -104
> [E 10:38:07.559495] Flow proto error cleanup started on 0x2a9df4d8b0, error_code: -1073741973
> [E 10:38:07.560125] Flow proto 0x2a9df4d8b0 canceling a total of 1 BMI or Trove operations
> [E 10:38:07.560150] Flow proto error cleanup finished 0x2a9df4d8b0, error_code: -1073741973
> [E 20:07:13.662001] Flow proto error cleanup started on 0x2aa2f658f0, error_code: -1073741973
> [E 20:07:13.662675] Flow proto 0x2aa2f658f0 canceling a total of 1 BMI or Trove operations
> [E 20:07:13.662703] Flow proto error cleanup finished 0x2aa2f658f0, error_code: -1073741973
> 
> ==> ff012/var/log/pvfs2-server.log <==
> [E 11:22:11.302587] dbpf_finalize: No such file or directory
> [E 08:34:03.160263] Flow proto error cleanup started on 0x2a9738d7e0, error_code: -104
> [E 08:34:03.161145] Flow proto 0x2a9738d7e0 canceling a total of 1 BMI or Trove operations
> [E 08:34:03.161165] Flow proto error cleanup finished 0x2a9738d7e0, error_code: -104
> [E 08:36:52.825555] Job time out: cancelling flow operation, job_id: 43493526.
> [E 08:36:52.826176] Flow proto cancel called on 0x2a97376500
> [E 08:36:56.949573] Flow proto error cleanup started on 0x2a97376500, error_code: -1610612737
> [E 08:36:56.950684] Flow proto 0x2a97376500 canceling a total of 2 BMI or Trove operations
> [E 08:36:56.950821] Flow proto error cleanup finished 0x2a97376500, error_code: -1610612737
> [E 11:29:25.947687] Job time out: cancelling flow operation, job_id: 93095488.
> [E 11:29:25.948328] Flow proto cancel called on 0x14209f0
> [E 11:30:03.829131] Flow proto error cleanup started on 0x14209f0, error_code: -1610612737
> [E 11:30:03.830901] Flow proto 0x14209f0 canceling a total of 2 BMI or Trove operations
> [E 11:30:03.830976] Flow proto error cleanup finished 0x14209f0, error_code: -1610612737
> [E 12:57:45.976144] Job time out: cancelling flow operation, job_id: 174239452.
> [E 12:57:45.977003] Flow proto cancel called on 0x2a98b3d0b0
> [E 12:58:37.054246] Flow proto error cleanup started on 0x2a98b3d0b0, error_code: -1610612737
> [E 12:58:37.055212] Flow proto 0x2a98b3d0b0 canceling a total of 2 BMI or Trove operations
> [E 12:58:37.055292] Flow proto error cleanup finished 0x2a98b3d0b0, error_code: -1610612737
> [E 11:50:44.409229]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 12:26:01.613807]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 11:52:11.015067] Flow proto error cleanup started on 0x2a971e8bd0, error_code: -1073741973
> [E 11:52:11.016821] Flow proto 0x2a971e8bd0 canceling a total of 0 BMI or Trove operations
> [E 11:52:11.016847] Flow proto error cleanup finished 0x2a971e8bd0, error_code: -1073741973
> [E 12:41:07.917434] Flow proto error cleanup started on 0x2a9771d180, error_code: -1073741973
> [E 12:41:07.917915] Flow proto 0x2a9771d180 canceling a total of 0 BMI or Trove operations
> [E 12:41:07.917934] Flow proto error cleanup finished 0x2a9771d180, error_code: -1073741973
> [E 09:02:34.417028]
> PVFS2 server got signal 15 (server_status_flag: 262143)
> [E 04:44:49.758474] Flow proto error cleanup started on 0x2a9ad31b10, error_code: -104
> [E 04:44:49.759557] Flow proto 0x2a9ad31b10 canceling a total of 1 BMI or Trove operations
> [E 04:44:49.759578] Flow proto error cleanup finished 0x2a9ad31b10, error_code: -104
> [E 10:38:11.060375] Flow proto error cleanup started on 0x2a9dd0a2c0, error_code: -1073741973
> [E 10:38:11.061020] Flow proto 0x2a9dd0a2c0 canceling a total of 1 BMI or Trove operations
> [E 10:38:11.061044] Flow proto error cleanup finished 0x2a9dd0a2c0, error_code: -1073741973
> [E 20:07:04.830849] Flow proto error cleanup started on 0x2aa2e753f0, error_code: -1073741973
> [E 20:07:04.831496] Flow proto 0x2aa2e753f0 canceling a total of 1 BMI or Trove operations
> [E 20:07:04.831523] Flow proto error cleanup finished 0x2aa2e753f0, error_code: -1073741973

-- 
Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112-0190
Phone: (801) 558-1150, Fax: (801) 585-5366
http://www.map.utah.edu/umaplink/0019.html
-------------- next part --------------
An embedded message was scrubbed...
From: "Anita Orendt" <Anita.Orendt at utah.edu>
Subject: Molpro details
Date: Tue, 10 Oct 2006 12:17:27 -0600
Size: 2616
Url: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20061010/eeb1eebf/Molprodetails-0001.eml