[Pvfs2-users] PVFS2 2.8.1 - batch_create request got: Invalid
argument
Asterios Katsifodimos
asteriosk at gmail.com
Mon Apr 6 12:41:28 EDT 2009
Yes both of them. Because now both are Metadata servers. When I had one
metadata and
one IO server, the metadata server was not producing the errors until the IO
server got up.
>From the time that the IO server gets up, the Metadata server is getting
crazy...
I have uploaded the log files here:
http://grid.ucy.ac.cy/file/pvfs_logwn140.grid.ucy.ac.cy
http://grid.ucy.ac.cy/file/pvfs_logwn141.grid.ucy.ac.cy
have a look!
thanks!
On Mon, Apr 6, 2009 at 7:00 PM, Phil Carns <carns at mcs.anl.gov> wrote:
> Ok. Could you try "verbose" now as the log level? It is close to the
> "all" level but should only print information while the server is busy.
>
> Are both wn140 and wn141 showing the same batch create errors, or just one
> of them?
>
> thanks,
> -Phil
>
> Asterios Katsifodimos wrote:
>
>> Hello Phil,
>>
>> Thanks for you answer.
>> Yes I delete the storage dir every time I make a new configuration
>> and I run the pvfs2-server -f command before starting the daemons.
>>
>> The only thing that I get from the servers is the batch_create, starting
>> server, and the "PVFS2 server got signal 15 (server_status_flag: 507903"
>> error message. Do you want me to try on an other log level?
>>
>> Also, this is how the server is configured:
>> ***** Displaying PVFS Configuration Information *****
>> ------------------------------------------------------
>> PVFS2 configured to build karma gui : no
>> PVFS2 configured to perform coverage analysis : no
>> PVFS2 configured for aio threaded callbacks : yes
>> PVFS2 configured to use FUSE : no
>> PVFS2 configured for the 2.6.x kernel module : no
>> PVFS2 configured for the 2.4.x kernel module : no
>> PVFS2 configured for using the mmap-ra-cache : no
>> PVFS2 will use workaround for redhat 2.4 kernels : no
>> PVFS2 will use workaround for buggy NPTL : no
>> PVFS2 server will be built : yes
>>
>> PVFS2 version string: 2.8.1
>>
>>
>> thanks again,
>> On Mon, Apr 6, 2009 at 5:21 PM, Phil Carns <carns at mcs.anl.gov <mailto:
>> carns at mcs.anl.gov>> wrote:
>>
>> Hello,
>>
>> I'm not sure what would cause that "Invalid argument" error.
>>
>> Could you try the following steps:
>>
>> - kill both servers
>> - modify your configuration files to set "EventLogging" to "none"
>> - delete your old log files (or move them to another directory)
>> - start the servers
>>
>> You can then send us the complete contents of both log files and we
>> can go from there. The "all" level is a little hard to interpret
>> because it generates a lot of information even when servers are idle.
>>
>> Also, when you went from one server to two, did you delete your old
>> storage space (/pvfs) and start over, or are you trying to keep that
>> data and add servers to it?
>>
>> thanks!
>> -Phil
>>
>> Asterios Katsifodimos wrote:
>>
>> Hello all,
>>
>> I have been trying to install PVFS 2.8.1 on Ubuntu server,
>> Centos4 and
>> Scientific Linux 4. I compile it and can run it on a "single
>> host" configuration
>> without any problems.
>>
>> However, when I add more nodes to the configuration(always using
>> the
>> pvfs2-genconfig defaults ) I have the following problem:
>>
>> *On the metadata node I get these messages:*
>> [E 04/02 20:16] batch_create request got: Invalid argument
>> [E 04/02 20:16] batch_create request got: Invalid argument
>> [E 04/02 20:16] batch_create request got: Invalid argument
>> [E 04/02 20:16] batch_create request got: Invalid argument
>>
>>
>> *In the IO nodes I get:*
>> [root at wn140 ~]# tail -50 /tmp/pvfs2-server.log
>> [D 04/02 23:53] BMI_testcontext completing: 18446744072456767880
>> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
>> msgpairarray_sm:complete (status: 1)
>> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
>> base-frm: 1
>> [D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00
>> status_user_tag 1 msgarray_count 1
>> [D 04/02 23:53] msgpairarray: 1 operations remain
>> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
>> msgpairarray_sm:complete (error code: -1073742006), (action:
>> DEFERRED)
>> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
>> msgpairarray_sm:complete (status: 0)
>> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
>> base-frm: 1
>> [D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00
>> status_user_tag 0 msgarray_count 1
>> [D 04/02 23:53] msgpairarray: all operations complete
>> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
>> msgpairarray_sm:complete (error code: 190), (action: COMPLETE)
>> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
>> msgpairarray_sm:completion_fn (status: 0)
>> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
>> base-frm: 1
>> [D 04/02 23:53] (0x88f8b00) msgpairarray state: completion_fn
>> [E 04/02 23:53] Warning: msgpair failed to tcp://wn141:3334,
>> will retry: Connection refused
>> [D 04/02 23:53] *** msgpairarray_completion_fn: msgpair 0
>> failed, retry 1
>> [D 04/02 23:53] *** msgpairarray_completion_fn: msgpair retrying
>> after delay.
>> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
>> msgpairarray_sm:completion_fn (error code: 191), (action: COMPLETE)
>> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
>> msgpairarray_sm:post_retry (status: 0)
>> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
>> base-frm: 1
>> [D 04/02 23:53] msgpairarray_post_retry: sm 0x88f8b00, wait 2000 ms
>> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
>> msgpairarray_sm:post_retry (error code: 0), (action: DEFERRED)
>> [D 04/02 23:53] [SM Entering]: (0x89476c0)
>> perf_update_sm:do_work (status: 0)
>> [P 04/02 23:53] Start times (hr:min:sec): 23:53:11.330
>> 23:53:10.310 23:53:09.287 23:53:08.268 23:53:07.245
>> 23:53:06.225
>> [P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.026
>> 00:00:01.020 00:00:01.023 00:00:01.019 00:00:01.023
>> 00:00:01.020
>> [P 04/02 23:53]
>>
>> -------------------------------------------------------------------------------------------------------------
>> [P 04/02 23:53] bytes read : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] bytes written : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata reads : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata writes : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata dspace ops : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata keyval ops : 1
>> 1 1 1 1 1
>> [P 04/02 23:53] request scheduler : 0
>> 0 0 0 0 0
>> [D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work
>> (error code: 0), (action: DEFERRED)
>> [D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work
>> (status: 0)
>> [D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work
>> (error code: 0), (action: DEFERRED)
>> [D 04/02 23:53] [SM Entering]: (0x89476c0)
>> perf_update_sm:do_work (status: 0)
>> [P 04/02 23:53] Start times (hr:min:sec): 23:53:12.356
>> 23:53:11.330 23:53:10.310 23:53:09.287 23:53:08.268
>> 23:53:07.245
>> [P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.020
>> 00:00:01.026 00:00:01.020 00:00:01.023 00:00:01.019
>> 00:00:01.023
>> [P 04/02 23:53]
>>
>> -------------------------------------------------------------------------------------------------------------
>> [P 04/02 23:53] bytes read : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] bytes written : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata reads : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata writes : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata dspace ops : 0
>> 0 0 0 0 0
>> [P 04/02 23:53] metadata keyval ops : 1
>> 1 1 1 1 1
>> [P 04/02 23:53] request scheduler : 0
>> 0 0 0 0 0
>> [D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work
>> (error code: 0), (action: DEFERRED)
>> [D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work
>> (status: 0)
>> [D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work
>> (error code: 0), (action: DEFERRED)
>>
>>
>> The metadata node keeps asking for something that the IO nodes
>> cannot give
>> the right way. So it complains. This makes the nodes and the
>> metadata node
>> not to work.
>>
>> I have installed those services many times. I have tested this
>> using berkeley
>> db 4.2 and 4.3 on Redhat systems(centos, scientific linnux) and
>> on Ubuntu server.
>>
>> I have also tried the PVFS version 2.6.3 and I get the same
>> problem.
>>
>> *My config files look like:*
>> [root at wn140 ~]# more /etc/pvfs2-fs.conf
>> <Defaults>
>> UnexpectedRequests 50
>> EventLogging all
>> EnableTracing no
>> LogStamp datetime
>> BMIModules bmi_tcp
>> FlowModules flowproto_multiqueue
>> PerfUpdateInterval 1000
>> ServerJobBMITimeoutSecs 30
>> ServerJobFlowTimeoutSecs 30
>> ClientJobBMITimeoutSecs 300
>> ClientJobFlowTimeoutSecs 300
>> ClientRetryLimit 5
>> ClientRetryDelayMilliSecs 2000
>> PrecreateBatchSize 512
>> PrecreateLowThreshold 256
>>
>> StorageSpace /pvfs
>> LogFile /tmp/pvfs2-server.log
>> </Defaults>
>>
>> <Aliases>
>> Alias wn140 tcp://wn140:3334
>> Alias wn141 tcp://wn141:3334
>> </Aliases>
>>
>> <Filesystem>
>> Name pvfs2-fs
>> ID 320870944
>> RootHandle 1048576
>> FileStuffing yes
>> <MetaHandleRanges>
>> Range wn140 3-2305843009213693953
>> Range wn141 2305843009213693954-4611686018427387904
>> </MetaHandleRanges>
>> <DataHandleRanges>
>> Range wn140 4611686018427387905-6917529027641081855
>> Range wn141 6917529027641081856-9223372036854775806
>> </DataHandleRanges>
>> <StorageHints>
>> TroveSyncMeta yes
>> TroveSyncData no
>> TroveMethod alt-aio
>> </StorageHints>
>> </Filesystem>
>>
>>
>> My setup is made from two nodes that are both IO and Metadata
>> nodes. I have also tried
>> a 4 node setup with 2I/O - 2 MD nodes resulting in the same thing.
>>
>> Any suggestions?
>>
>> thank you in advance,
>> --
>> Asterios Katsifodimos
>> High Performance Computing systems Lab
>> Department of Computer Science, University of Cyprus
>> http://www.asteriosk.gr <http://www.asteriosk.gr/>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> <mailto:Pvfs2-users at beowulf-underground.org>
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20090406/5fae6f47/attachment-0001.htm
More information about the Pvfs2-users
mailing list