Hello Phil,<br><br>Thanks for you answer.<br>Yes I delete the storage dir every time I make a new configuration<br>and I run the pvfs2-server -f command before starting the daemons.<br><br>The only thing that I get from the servers is the batch_create, starting server, and the "PVFS2 server got signal 15 (server_status_flag: 507903" <br>
error message. Do you want me to try on an other log level?<br><br>Also, this is how the server is configured:<br>***** Displaying PVFS Configuration Information *****<br>------------------------------------------------------<br>
PVFS2 configured to build karma gui : no<br>PVFS2 configured to perform coverage analysis : no<br>PVFS2 configured for aio threaded callbacks : yes<br>PVFS2 configured to use FUSE : no<br>
PVFS2 configured for the 2.6.x kernel module : no<br>PVFS2 configured for the 2.4.x kernel module : no<br>PVFS2 configured for using the mmap-ra-cache : no<br>PVFS2 will use workaround for redhat 2.4 kernels : no<br>
PVFS2 will use workaround for buggy NPTL : no<br>PVFS2 server will be built : yes<br><br>PVFS2 version string: 2.8.1<br><br><br>thanks again,<br><div class="gmail_quote">On Mon, Apr 6, 2009 at 5:21 PM, Phil Carns <span dir="ltr"><<a href="mailto:carns@mcs.anl.gov" target="_blank">carns@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hello,<br>
<br>
I'm not sure what would cause that "Invalid argument" error.<br>
<br>
Could you try the following steps:<br>
<br>
- kill both servers<br>
- modify your configuration files to set "EventLogging" to "none"<br>
- delete your old log files (or move them to another directory)<br>
- start the servers<br>
<br>
You can then send us the complete contents of both log files and we can go from there. The "all" level is a little hard to interpret because it generates a lot of information even when servers are idle.<br>
<br>
Also, when you went from one server to two, did you delete your old storage space (/pvfs) and start over, or are you trying to keep that data and add servers to it?<br>
<br>
thanks!<br>
-Phil<br>
<br>
Asterios Katsifodimos wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div>
Hello all,<br>
<br>
I have been trying to install PVFS 2.8.1 on Ubuntu server, Centos4 and<br>
Scientific Linux 4. I compile it and can run it on a "single host" configuration<br>
without any problems.<br>
<br>
However, when I add more nodes to the configuration(always using the<br>
pvfs2-genconfig defaults ) I have the following problem:<br>
<br>
*On the metadata node I get these messages:*<br>
[E 04/02 20:16] batch_create request got: Invalid argument<br>
[E 04/02 20:16] batch_create request got: Invalid argument<br>
[E 04/02 20:16] batch_create request got: Invalid argument<br>
[E 04/02 20:16] batch_create request got: Invalid argument<br>
<br>
<br>
*In the IO nodes I get:*<br>
[root@wn140 ~]# tail -50 /tmp/pvfs2-server.log<br>
[D 04/02 23:53] BMI_testcontext completing: 18446744072456767880<br>
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:complete (status: 1)<br>
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1<br>
[D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00 status_user_tag 1 msgarray_count 1<br>
[D 04/02 23:53] msgpairarray: 1 operations remain<br>
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:complete (error code: -1073742006), (action: DEFERRED)<br>
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:complete (status: 0)<br>
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1<br>
[D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00 status_user_tag 0 msgarray_count 1<br>
[D 04/02 23:53] msgpairarray: all operations complete<br>
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:complete (error code: 190), (action: COMPLETE)<br>
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:completion_fn (status: 0)<br>
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1<br>
[D 04/02 23:53] (0x88f8b00) msgpairarray state: completion_fn<br>
[E 04/02 23:53] Warning: msgpair failed to tcp://wn141:3334, will retry: Connection refused<br>
[D 04/02 23:53] *** msgpairarray_completion_fn: msgpair 0 failed, retry 1<br>
[D 04/02 23:53] *** msgpairarray_completion_fn: msgpair retrying after delay.<br>
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:completion_fn (error code: 191), (action: COMPLETE)<br>
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:post_retry (status: 0)<br>
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1<br>
[D 04/02 23:53] msgpairarray_post_retry: sm 0x88f8b00, wait 2000 ms<br>
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:post_retry (error code: 0), (action: DEFERRED)<br>
[D 04/02 23:53] [SM Entering]: (0x89476c0) perf_update_sm:do_work (status: 0)<br>
[P 04/02 23:53] Start times (hr:min:sec): 23:53:11.330 23:53:10.310 23:53:09.287 23:53:08.268 23:53:07.245 23:53:06.225<br>
[P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.026 00:00:01.020 00:00:01.023 00:00:01.019 00:00:01.023 00:00:01.020<br>
[P 04/02 23:53] -------------------------------------------------------------------------------------------------------------<br>
[P 04/02 23:53] bytes read : 0 0 0 0 0 0<br>
[P 04/02 23:53] bytes written : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata reads : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata writes : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata dspace ops : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata keyval ops : 1 1 1 1 1 1<br>
[P 04/02 23:53] request scheduler : 0 0 0 0 0 0<br>
[D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work (error code: 0), (action: DEFERRED)<br>
[D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work (status: 0)<br>
[D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work (error code: 0), (action: DEFERRED)<br>
[D 04/02 23:53] [SM Entering]: (0x89476c0) perf_update_sm:do_work (status: 0)<br>
[P 04/02 23:53] Start times (hr:min:sec): 23:53:12.356 23:53:11.330 23:53:10.310 23:53:09.287 23:53:08.268 23:53:07.245<br>
[P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.020 00:00:01.026 00:00:01.020 00:00:01.023 00:00:01.019 00:00:01.023<br>
[P 04/02 23:53] -------------------------------------------------------------------------------------------------------------<br>
[P 04/02 23:53] bytes read : 0 0 0 0 0 0<br>
[P 04/02 23:53] bytes written : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata reads : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata writes : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata dspace ops : 0 0 0 0 0 0<br>
[P 04/02 23:53] metadata keyval ops : 1 1 1 1 1 1<br>
[P 04/02 23:53] request scheduler : 0 0 0 0 0 0<br>
[D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work (error code: 0), (action: DEFERRED)<br>
[D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work (status: 0)<br>
[D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work (error code: 0), (action: DEFERRED)<br>
<br>
<br>
The metadata node keeps asking for something that the IO nodes cannot give<br>
the right way. So it complains. This makes the nodes and the metadata node<br>
not to work.<br>
<br>
I have installed those services many times. I have tested this using berkeley<br>
db 4.2 and 4.3 on Redhat systems(centos, scientific linnux) and on Ubuntu server.<br>
<br>
I have also tried the PVFS version 2.6.3 and I get the same problem.<br>
<br>
*My config files look like:*<br>
[root@wn140 ~]# more /etc/pvfs2-fs.conf<br>
<Defaults><br>
UnexpectedRequests 50<br>
EventLogging all<br>
EnableTracing no<br>
LogStamp datetime<br>
BMIModules bmi_tcp<br>
FlowModules flowproto_multiqueue<br>
PerfUpdateInterval 1000<br>
ServerJobBMITimeoutSecs 30<br>
ServerJobFlowTimeoutSecs 30<br>
ClientJobBMITimeoutSecs 300<br>
ClientJobFlowTimeoutSecs 300<br>
ClientRetryLimit 5<br>
ClientRetryDelayMilliSecs 2000<br>
PrecreateBatchSize 512<br>
PrecreateLowThreshold 256<br>
<br>
StorageSpace /pvfs<br>
LogFile /tmp/pvfs2-server.log<br>
</Defaults><br>
<br>
<Aliases><br>
Alias wn140 tcp://wn140:3334<br>
Alias wn141 tcp://wn141:3334<br>
</Aliases><br>
<br>
<Filesystem><br>
Name pvfs2-fs<br>
ID 320870944<br>
RootHandle 1048576<br>
FileStuffing yes<br>
<MetaHandleRanges><br>
Range wn140 3-2305843009213693953<br>
Range wn141 2305843009213693954-4611686018427387904<br>
</MetaHandleRanges><br>
<DataHandleRanges><br>
Range wn140 4611686018427387905-6917529027641081855<br>
Range wn141 6917529027641081856-9223372036854775806<br>
</DataHandleRanges><br>
<StorageHints><br>
TroveSyncMeta yes<br>
TroveSyncData no<br>
TroveMethod alt-aio<br>
</StorageHints><br>
</Filesystem><br>
<br>
<br>
My setup is made from two nodes that are both IO and Metadata nodes. I have also tried<br>
a 4 node setup with 2I/O - 2 MD nodes resulting in the same thing.<br>
<br>
Any suggestions?<br>
<br>
thank you in advance,<br>
--<br>
Asterios Katsifodimos<br>
High Performance Computing systems Lab<br>
Department of Computer Science, University of Cyprus<br>
</div></div><a href="http://www.asteriosk.gr" target="_blank">http://www.asteriosk.gr</a> <<a href="http://www.asteriosk.gr/" target="_blank">http://www.asteriosk.gr/</a>><br>
<br>
<br>
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Pvfs2-users mailing list<br>
<a href="mailto:Pvfs2-users@beowulf-underground.org" target="_blank">Pvfs2-users@beowulf-underground.org</a><br>
<a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users</a><br>
</blockquote>
<br>
</blockquote></div><br>