<HTML>
<HEAD>
<TITLE>Re: [Pvfs2-users] PVFS server won't start </TITLE>
</HEAD>
<BODY>
<FONT FACE="Courier, Courier New"><SPAN STYLE='font-size:11pt'>I noticed a similar thread where someone ran a fsck and recovered. I tried a fsck with no luck. I ran db_verify on all of the .db files and it didn’t show anything. Below is the debug output of the server:<BR>
<BR>
[D 06/29 15:29] Passing tcp://oss004-4:3337 as BMI listen address.<BR>
[D 06/29 15:29] BMI_tcp_initialize: Initializing TCP/IP module.<BR>
[D 06/29 15:29] BMI_tcp_initialize: TCP/IP module successfully initialized.<BR>
[D 06/29 15:29] Server using shm key hint: 373672738<BR>
[D 06/29 15:29] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 11<BR>
[D 06/29 15:29] Default socket buffers send:16384 receive:87380<BR>
[D 06/29 15:29] Setting socket buffer size for send:0 receive:0 <BR>
[D 06/29 15:29] Reread socket buffers send:16384 receive:87380<BR>
[D 06/29 15:29] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 12<BR>
[D 06/29 15:29] Default socket buffers send:16384 receive:87380<BR>
[D 06/29 15:29] Setting socket buffer size for send:0 receive:0 <BR>
[D 06/29 15:29] Reread socket buffers send:16384 receive:87380<BR>
[D 06/29 15:29] dbpf_thread_initialize: initialized<BR>
[D 06/29 15:29] [SYNC_COALESCE]: dbpf_sync_context_init for context 0 called<BR>
[D 06/29 15:29] dbpf_collection_lookup of coll: pvfs2-fs<BR>
[D 06/29 15:29] dbpf using default db cache size.<BR>
[D 06/29 15:29] dbpf using shm key: 1020239961<BR>
[D 06/29 15:29] collection lookup: version is 0.1.4<BR>
[D 06/29 15:29] [SYNC_COALESCE]: dbpf_sync_context_init for context 1 called<BR>
[D 06/29 15:29] dbpf collection 373672578 - Setting handle timeout to 360000000 microseconds<BR>
[D 06/29 15:29] - set handle re-use timeout to 360 seconds (ret=0)<BR>
[D 06/29 15:29] dbpf collection 373672578 - Setting cache keywords of attribute cache to dh,<BR>
[D 06/29 15:29] Setting dbpf_attr_cache keywords to:<BR>
dh,<BR>
[D 06/29 15:29] dbpf collection 373672578 - Setting cache size of attribute cache to 511<BR>
[D 06/29 15:29] dbpf collection 373672578 - Setting maximum elements of attribute cache to 1024<BR>
[D 06/29 15:29] dbpf collection 373672578 - Initialize collection attr. cache<BR>
[D 06/29 15:29] There are 1 cacheable keywords registered<BR>
[D 06/29 15:29] dbpf_attr_cache_initialize: initialized<BR>
[D 06/29 15:29] dbpf collection 373672578 - Setting collection handle ranges to 4323455642275676148-4611686018427387890,8935141660703064036-9223372036854775778<BR>
[D 06/29 15:29] op_queue add: 0x9f96380<BR>
[D 06/29 15:29] dbpf_thread_function started<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] handle_new_connection: Assigning socket 11 to new method addr.<BR>
[D 06/29 15:29] tcp_do_work_recv: Reading header for new op.<BR>
[D 06/29 15:29] tcp_do_work_recv: Received new message; mode: 2.<BR>
[D 06/29 15:29] tcp_do_work_recv: tag: 5865658<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9f96380<BR>
[D 06/29 15:29] handle_new_connection: Assigning socket 12 to new method addr.<BR>
[D 06/29 15:29] op_queue add: 0x9f9da50<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9f9da50<BR>
[D 06/29 15:29] op_queue add: 0x9fa63d0<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fa63d0<BR>
[D 06/29 15:29] op_queue add: 0x9fad360<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fad360<BR>
[D 06/29 15:29] op_queue add: 0x9fb0bf0<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fb0bf0<BR>
[D 06/29 15:29] op_queue add: 0x9fb2f90<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fb2f90<BR>
[D 06/29 15:29] op_queue add: 0x9fb5ab0<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fb5ab0<BR>
[D 06/29 15:29] op_queue add: 0x9fc7a30<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fc7a30<BR>
[D 06/29 15:29] op_queue add: 0x9fca500<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fca500<BR>
[D 06/29 15:29] op_queue add: 0x9fca690<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fca690<BR>
[D 06/29 15:29] op_queue add: 0x9fe1980<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1)<BR>
[D 06/29 15:29] op_queue add: 0x9fe1980<BR>
[D 06/29 15:29] op_queue add: 0x9fe2330<BR>
[D 06/29 15:29] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES)<BR>
[E 06/29 15:29] dbpf_dspace_iterate_handles_op_svc: Invalid argument<BR>
[D 06/29 15:29] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: -1073742095)<BR>
[D 06/29 15:29] op_queue add: 0x9fe2330<BR>
[D 06/29 15:29] trove_dspace_iterate_handles failed<BR>
[E 06/29 15:29] Error adding handle range 4323455642275676148-4611686018427387890,8935141660703064036-9223372036854775778 to filesystem pvfs2-fs<BR>
[E 06/29 15:29] Error: Could not initialize server interfaces; aborting.<BR>
[E 06/29 15:29] Error: Could not initialize server; aborting.<BR>
[D 06/29 15:29] *** server shutdown in progress ***<BR>
<BR>
<BR>
-Randy<BR>
<BR>
<HR ALIGN=CENTER SIZE="3" WIDTH="95%"><B>From: </B>Randall Martin <<a href="wolf@clemson.edu">wolf@clemson.edu</a>><BR>
<B>Date: </B>Mon, 29 Jun 2009 14:05:33 -0400<BR>
<B>To: </B><<a href="pvfs2-users@beowulf-underground.org">pvfs2-users@beowulf-underground.org</a>><BR>
<B>Subject: </B>[Pvfs2-users] PVFS server won't start <BR>
<BR>
One of our PVFS servers crashed and now it won’t start back. It was previously working since June 2 until today’s crash. Any ideas on how to fix it? I was running the 2.8.1 released version, but I also tried the HEAD version with no change in symptoms.<BR>
<BR>
>From the server log:<BR>
<BR>
[D 06/29 13:49] PVFS2 Server version 2.8.1pre1-2009-06-26-182521 starting.<BR>
[E 06/29 13:49] dbpf_dspace_iterate_handles_op_svc: Invalid argument<BR>
[E 06/29 13:49] Error adding handle range 4323455642275676148-4611686018427387890,8935141660703064036-9223372036854775778 to filesystem pvfs2-fs<BR>
[E 06/29 13:49] Error: Could not initialize server interfaces; aborting.<BR>
[E 06/29 13:49] Error: Could not initialize server; aborting.<BR>
<BR>
My config file:<BR>
<BR>
<BR>
<Defaults><BR>
UnexpectedRequests 50<BR>
EventLogging none<BR>
EnableTracing no<BR>
LogStamp datetime<BR>
BMIModules bmi_tcp<BR>
FlowModules flowproto_multiqueue<BR>
PerfUpdateInterval 1000<BR>
ServerJobBMITimeoutSecs 30<BR>
ServerJobFlowTimeoutSecs 30<BR>
ClientJobBMITimeoutSecs 300<BR>
ClientJobFlowTimeoutSecs 300<BR>
ClientRetryLimit 60 <BR>
ClientRetryDelayMilliSecs 10000<BR>
PrecreateBatchSize 512<BR>
PrecreateLowThreshold 256<BR>
</Defaults><BR>
<BR>
<Aliases><BR>
Alias oss001-1 tcp://oss001-1:3334<BR>
Alias oss001-2 tcp://oss001-2:3335<BR>
Alias oss001-3 tcp://oss001-3:3336<BR>
Alias oss001-4 tcp://oss001-4:3337<BR>
<BR>
Alias oss002-1 tcp://oss002-1:3334<BR>
Alias oss002-2 tcp://oss002-2:3335<BR>
Alias oss002-3 tcp://oss002-3:3336<BR>
Alias oss002-4 tcp://oss002-4:3337<BR>
<BR>
Alias oss003-1 tcp://oss003-1:3334<BR>
Alias oss003-2 tcp://oss003-2:3335<BR>
Alias oss003-3 tcp://oss003-3:3336<BR>
Alias oss003-4 tcp://oss003-4:3337<BR>
<BR>
Alias oss004-1 tcp://oss004-1:3334<BR>
Alias oss004-2 tcp://oss004-2:3335<BR>
Alias oss004-3 tcp://oss004-3:3336<BR>
Alias oss004-4 tcp://oss004-4:3337<BR>
</Aliases><BR>
<BR>
<BR>
<ServerOptions><BR>
Server oss001-1<BR>
StorageSpace /ost1<BR>
LogFile /var/log/pvfs2-server.oss001-1.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss001-2<BR>
StorageSpace /ost2<BR>
LogFile /var/log/pvfs2-server.oss001-2.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss001-3<BR>
StorageSpace /ost3<BR>
LogFile /var/log/pvfs2-server.oss001-3.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss001-4<BR>
StorageSpace /ost4<BR>
LogFile /var/log/pvfs2-server.oss001-4.log<BR>
</ServerOptions><BR>
<BR>
<BR>
<ServerOptions><BR>
Server oss002-1<BR>
StorageSpace /ost5<BR>
LogFile /var/log/pvfs2-server.oss002-1.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss002-2<BR>
StorageSpace /ost6<BR>
LogFile /var/log/pvfs2-server.oss002-2.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss002-3<BR>
StorageSpace /ost7<BR>
LogFile /var/log/pvfs2-server.oss002-3.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss002-4<BR>
StorageSpace /ost8<BR>
LogFile /var/log/pvfs2-server.oss002-4.log<BR>
</ServerOptions><BR>
<BR>
<BR>
<ServerOptions><BR>
Server oss003-1<BR>
StorageSpace /ost9<BR>
LogFile /var/log/pvfs2-server.oss003-1.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss003-2<BR>
StorageSpace /ost10<BR>
LogFile /var/log/pvfs2-server.oss003-2.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss003-3<BR>
StorageSpace /ost11<BR>
LogFile /var/log/pvfs2-server.oss003-3.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss003-4<BR>
StorageSpace /ost12<BR>
LogFile /var/log/pvfs2-server.oss003-4.log<BR>
</ServerOptions><BR>
<BR>
<BR>
<ServerOptions><BR>
Server oss004-1<BR>
StorageSpace /ost13<BR>
LogFile /var/log/pvfs2-server.oss004-1.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss004-2<BR>
StorageSpace /ost14<BR>
LogFile /var/log/pvfs2-server.oss004-2.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss004-3<BR>
StorageSpace /ost15<BR>
LogFile /var/log/pvfs2-server.oss004-3.log<BR>
</ServerOptions><BR>
<ServerOptions><BR>
Server oss004-4<BR>
StorageSpace /ost16<BR>
LogFile /var/log/pvfs2-server.oss004-4.log<BR>
</ServerOptions><BR>
<BR>
<Filesystem><BR>
Name pvfs2-fs<BR>
ID 373672578<BR>
RootHandle 1048576<BR>
FileStuffing yes<BR>
<MetaHandleRanges><BR>
Range oss001-1 3-288230376151711745<BR>
Range oss001-2 288230376151711746-576460752303423488<BR>
Range oss001-3 576460752303423489-864691128455135231<BR>
Range oss001-4 864691128455135232-1152921504606846974<BR>
Range oss002-1 1152921504606846975-1441151880758558717<BR>
Range oss002-2 1441151880758558718-1729382256910270460<BR>
Range oss002-3 1729382256910270461-2017612633061982203<BR>
Range oss002-4 2017612633061982204-2305843009213693946<BR>
Range oss003-1 2305843009213693947-2594073385365405689<BR>
Range oss003-2 2594073385365405690-2882303761517117432<BR>
Range oss003-3 2882303761517117433-3170534137668829175<BR>
Range oss003-4 3170534137668829176-3458764513820540918<BR>
Range oss004-1 3458764513820540919-3746994889972252661<BR>
Range oss004-2 3746994889972252662-4035225266123964404<BR>
Range oss004-3 4035225266123964405-4323455642275676147<BR>
Range oss004-4 4323455642275676148-4611686018427387890<BR>
</MetaHandleRanges><BR>
<DataHandleRanges><BR>
Range oss001-1 4611686018427387891-4899916394579099633<BR>
Range oss001-2 4899916394579099634-5188146770730811376<BR>
Range oss001-3 5188146770730811377-5476377146882523119<BR>
Range oss001-4 5476377146882523120-5764607523034234862<BR>
Range oss002-1 5764607523034234863-6052837899185946605<BR>
Range oss002-2 6052837899185946606-6341068275337658348<BR>
Range oss002-3 6341068275337658349-6629298651489370091<BR>
Range oss002-4 6629298651489370092-6917529027641081834<BR>
Range oss003-1 6917529027641081835-7205759403792793577<BR>
Range oss003-2 7205759403792793578-7493989779944505320<BR>
Range oss003-3 7493989779944505321-7782220156096217063<BR>
Range oss003-4 7782220156096217064-8070450532247928806<BR>
Range oss004-1 8070450532247928807-8358680908399640549<BR>
Range oss004-2 8358680908399640550-8646911284551352292<BR>
Range oss004-3 8646911284551352293-8935141660703064035<BR>
Range oss004-4 8935141660703064036-9223372036854775778<BR>
</DataHandleRanges><BR>
<StorageHints><BR>
TroveSyncMeta no <BR>
TroveSyncData no<BR>
TroveMethod alt-aio<BR>
</StorageHints><BR>
<Distribution><BR>
Name simple_stripe<BR>
Param strip_size<BR>
Value 1048576<BR>
</Distribution><BR>
</Filesystem><BR>
<BR>
<BR>
Thanks,<BR>
Randy<BR>
<BR>
<HR ALIGN=CENTER SIZE="3" WIDTH="95%"></SPAN><FONT SIZE="2"><SPAN STYLE='font-size:10pt'>_______________________________________________<BR>
Pvfs2-users mailing list<BR>
<a href="Pvfs2-users@beowulf-underground.org">Pvfs2-users@beowulf-underground.org</a><BR>
<a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users</a><BR>
</SPAN></FONT></FONT>
</BODY>
</HTML>