[Pvfs2-users] PVFS2 2.8.1 - batch_create request got: Invalid
argument
Phil Carns
carns at mcs.anl.gov
Mon Apr 6 12:00:20 EDT 2009
Ok. Could you try "verbose" now as the log level? It is close to the
"all" level but should only print information while the server is busy.
Are both wn140 and wn141 showing the same batch create errors, or just
one of them?
thanks,
-Phil
Asterios Katsifodimos wrote:
> Hello Phil,
>
> Thanks for you answer.
> Yes I delete the storage dir every time I make a new configuration
> and I run the pvfs2-server -f command before starting the daemons.
>
> The only thing that I get from the servers is the batch_create, starting
> server, and the "PVFS2 server got signal 15 (server_status_flag: 507903"
> error message. Do you want me to try on an other log level?
>
> Also, this is how the server is configured:
> ***** Displaying PVFS Configuration Information *****
> ------------------------------------------------------
> PVFS2 configured to build karma gui : no
> PVFS2 configured to perform coverage analysis : no
> PVFS2 configured for aio threaded callbacks : yes
> PVFS2 configured to use FUSE : no
> PVFS2 configured for the 2.6.x kernel module : no
> PVFS2 configured for the 2.4.x kernel module : no
> PVFS2 configured for using the mmap-ra-cache : no
> PVFS2 will use workaround for redhat 2.4 kernels : no
> PVFS2 will use workaround for buggy NPTL : no
> PVFS2 server will be built : yes
>
> PVFS2 version string: 2.8.1
>
>
> thanks again,
> On Mon, Apr 6, 2009 at 5:21 PM, Phil Carns <carns at mcs.anl.gov
> <mailto:carns at mcs.anl.gov>> wrote:
>
> Hello,
>
> I'm not sure what would cause that "Invalid argument" error.
>
> Could you try the following steps:
>
> - kill both servers
> - modify your configuration files to set "EventLogging" to "none"
> - delete your old log files (or move them to another directory)
> - start the servers
>
> You can then send us the complete contents of both log files and we
> can go from there. The "all" level is a little hard to interpret
> because it generates a lot of information even when servers are idle.
>
> Also, when you went from one server to two, did you delete your old
> storage space (/pvfs) and start over, or are you trying to keep that
> data and add servers to it?
>
> thanks!
> -Phil
>
> Asterios Katsifodimos wrote:
>
> Hello all,
>
> I have been trying to install PVFS 2.8.1 on Ubuntu server,
> Centos4 and
> Scientific Linux 4. I compile it and can run it on a "single
> host" configuration
> without any problems.
>
> However, when I add more nodes to the configuration(always using the
> pvfs2-genconfig defaults ) I have the following problem:
>
> *On the metadata node I get these messages:*
> [E 04/02 20:16] batch_create request got: Invalid argument
> [E 04/02 20:16] batch_create request got: Invalid argument
> [E 04/02 20:16] batch_create request got: Invalid argument
> [E 04/02 20:16] batch_create request got: Invalid argument
>
>
> *In the IO nodes I get:*
> [root at wn140 ~]# tail -50 /tmp/pvfs2-server.log
> [D 04/02 23:53] BMI_testcontext completing: 18446744072456767880
> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
> msgpairarray_sm:complete (status: 1)
> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
> base-frm: 1
> [D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00
> status_user_tag 1 msgarray_count 1
> [D 04/02 23:53] msgpairarray: 1 operations remain
> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
> msgpairarray_sm:complete (error code: -1073742006), (action:
> DEFERRED)
> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
> msgpairarray_sm:complete (status: 0)
> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
> base-frm: 1
> [D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00
> status_user_tag 0 msgarray_count 1
> [D 04/02 23:53] msgpairarray: all operations complete
> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
> msgpairarray_sm:complete (error code: 190), (action: COMPLETE)
> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
> msgpairarray_sm:completion_fn (status: 0)
> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
> base-frm: 1
> [D 04/02 23:53] (0x88f8b00) msgpairarray state: completion_fn
> [E 04/02 23:53] Warning: msgpair failed to tcp://wn141:3334,
> will retry: Connection refused
> [D 04/02 23:53] *** msgpairarray_completion_fn: msgpair 0
> failed, retry 1
> [D 04/02 23:53] *** msgpairarray_completion_fn: msgpair retrying
> after delay.
> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
> msgpairarray_sm:completion_fn (error code: 191), (action: COMPLETE)
> [D 04/02 23:53] [SM Entering]: (0x88f8b00)
> msgpairarray_sm:post_retry (status: 0)
> [D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0
> base-frm: 1
> [D 04/02 23:53] msgpairarray_post_retry: sm 0x88f8b00, wait 2000 ms
> [D 04/02 23:53] [SM Exiting]: (0x88f8b00)
> msgpairarray_sm:post_retry (error code: 0), (action: DEFERRED)
> [D 04/02 23:53] [SM Entering]: (0x89476c0)
> perf_update_sm:do_work (status: 0)
> [P 04/02 23:53] Start times (hr:min:sec): 23:53:11.330
> 23:53:10.310 23:53:09.287 23:53:08.268 23:53:07.245
> 23:53:06.225
> [P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.026
> 00:00:01.020 00:00:01.023 00:00:01.019 00:00:01.023
> 00:00:01.020
> [P 04/02 23:53]
> -------------------------------------------------------------------------------------------------------------
> [P 04/02 23:53] bytes read : 0
> 0 0 0 0 0
> [P 04/02 23:53] bytes written : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata reads : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata writes : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata dspace ops : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata keyval ops : 1
> 1 1 1 1 1
> [P 04/02 23:53] request scheduler : 0
> 0 0 0 0 0
> [D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work
> (error code: 0), (action: DEFERRED)
> [D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work
> (status: 0)
> [D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work
> (error code: 0), (action: DEFERRED)
> [D 04/02 23:53] [SM Entering]: (0x89476c0)
> perf_update_sm:do_work (status: 0)
> [P 04/02 23:53] Start times (hr:min:sec): 23:53:12.356
> 23:53:11.330 23:53:10.310 23:53:09.287 23:53:08.268
> 23:53:07.245
> [P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.020
> 00:00:01.026 00:00:01.020 00:00:01.023 00:00:01.019
> 00:00:01.023
> [P 04/02 23:53]
> -------------------------------------------------------------------------------------------------------------
> [P 04/02 23:53] bytes read : 0
> 0 0 0 0 0
> [P 04/02 23:53] bytes written : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata reads : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata writes : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata dspace ops : 0
> 0 0 0 0 0
> [P 04/02 23:53] metadata keyval ops : 1
> 1 1 1 1 1
> [P 04/02 23:53] request scheduler : 0
> 0 0 0 0 0
> [D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work
> (error code: 0), (action: DEFERRED)
> [D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work
> (status: 0)
> [D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work
> (error code: 0), (action: DEFERRED)
>
>
> The metadata node keeps asking for something that the IO nodes
> cannot give
> the right way. So it complains. This makes the nodes and the
> metadata node
> not to work.
>
> I have installed those services many times. I have tested this
> using berkeley
> db 4.2 and 4.3 on Redhat systems(centos, scientific linnux) and
> on Ubuntu server.
>
> I have also tried the PVFS version 2.6.3 and I get the same problem.
>
> *My config files look like:*
> [root at wn140 ~]# more /etc/pvfs2-fs.conf
> <Defaults>
> UnexpectedRequests 50
> EventLogging all
> EnableTracing no
> LogStamp datetime
> BMIModules bmi_tcp
> FlowModules flowproto_multiqueue
> PerfUpdateInterval 1000
> ServerJobBMITimeoutSecs 30
> ServerJobFlowTimeoutSecs 30
> ClientJobBMITimeoutSecs 300
> ClientJobFlowTimeoutSecs 300
> ClientRetryLimit 5
> ClientRetryDelayMilliSecs 2000
> PrecreateBatchSize 512
> PrecreateLowThreshold 256
>
> StorageSpace /pvfs
> LogFile /tmp/pvfs2-server.log
> </Defaults>
>
> <Aliases>
> Alias wn140 tcp://wn140:3334
> Alias wn141 tcp://wn141:3334
> </Aliases>
>
> <Filesystem>
> Name pvfs2-fs
> ID 320870944
> RootHandle 1048576
> FileStuffing yes
> <MetaHandleRanges>
> Range wn140 3-2305843009213693953
> Range wn141 2305843009213693954-4611686018427387904
> </MetaHandleRanges>
> <DataHandleRanges>
> Range wn140 4611686018427387905-6917529027641081855
> Range wn141 6917529027641081856-9223372036854775806
> </DataHandleRanges>
> <StorageHints>
> TroveSyncMeta yes
> TroveSyncData no
> TroveMethod alt-aio
> </StorageHints>
> </Filesystem>
>
>
> My setup is made from two nodes that are both IO and Metadata
> nodes. I have also tried
> a 4 node setup with 2I/O - 2 MD nodes resulting in the same thing.
>
> Any suggestions?
>
> thank you in advance,
> --
> Asterios Katsifodimos
> High Performance Computing systems Lab
> Department of Computer Science, University of Cyprus
> http://www.asteriosk.gr <http://www.asteriosk.gr/>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> <mailto:Pvfs2-users at beowulf-underground.org>
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>
>
>
More information about the Pvfs2-users
mailing list