[Pvfs2-developers] PVFS2: files with ?---?--? permissions

[Phil Carns]

Thanks for the log files.  I don't see any DB errors like Becky and 
Randy were seeing at their site, so this may be a different problem.

One thing that jumps out in the server log file is that there is 
occasionally a getattr state machine that starts, but never gets past 
the PVFS request scheduler.  In this case, the client will eventually 
get a timeout like you are seeing in your client side logs.

An example is on line 1433 in your pvfs2-server.log file:

[D 09/16 14:08] (0x2aaab055eff0) getattr (prelude sm) state: req_sched

That (0x2aaab055eff0) number should show up later in the log file as 
that getattr state machine proceeds, but it never does.  It doesn't look 
like it even got to the first step of retrieving metadata.

We need to figure out why that request is getting stuck.  What 
"EventLogging" setting are you using on the server side to collect this 
log?  Could you add "access,access_detail" to the list and try to 
capture a failure case again?

You can also use pvfs2-set-debugmask to set this at runtime without 
restarting the server if that helps any.

thanks,
-Phil

Kumar, Amit H. wrote:
> Hi Everyone!
> 
> Thank you for responding. I enabled PVFS2_DEBUGMASK to see what was going on, before I responded.
> 
>  Attached are the /tmp/pvfs2-client.log and /tmp/pvfs2-server.log and /var/log/pvfs2.debug.log(PVFS2_DEBUGFILE) files
> 
> I do have the latest version, but not from the CVS. And I don't have a backup, so we will potentially lose some data :-(
> 
> I ran pvfs-fsck on the metadata server that has a local mount of the pvfs2-FS, and get the following errors/warnings. Surprisingly I don't have any problem browsing the file system until I hit the files with corrupt attributes. 
> 
> # /opt/pvfs2/bin/pvfs2-fsck -vp -m /scratch/pvfs2
> # Current FSID is 1415574627.
> [E 12:09:56.029492] job_time_mgr_expire: job time out: cancelling bmi operation, job_id: 18.
> [E 12:09:56.030046] Warning: msgpair failed to ib://pvfs2-io-0-0.local-ib:3335, will retry: Connection timed out
> [E 12:09:56.030118] *** msgpairarray_completion_fn: msgpair to server ib://pvfs2-io-0-0.local-ib:3335 failed: Connection timed out
> [E 12:09:56.030164] *** Non-BMI failure.
> PVFS_mgmt_setparam_list: Connection timed out (error class: 0)
> 
> I believe the time out is a result of not being able to access the files with corrupt attributes.
> 
> 
> Our Metadata server & one of the I/O server =>[root at pvfs2-io-0-0 tmp]# ps -ef | grep pvfs
> root     19964     1  0 Sep14 ?        00:24:02 /opt/pvfs2/sbin/pvfs2-server --pidfile /var/run/pvfs2.pid -a pvfs2-io-0-0.local-ib /opt/pvfs2/etc/pvfs2-fs.conf
> root     20434     1  0 Sep14 ?        00:00:00 /opt/pvfs2/sbin/pvfs2-client -p /opt/pvfs2/sbin/pvfs2-client-core
> root     20435 20434  0 Sep14 ?        00:00:05 pvfs2-client-core --child -a 5 -n 5 --logtype file -L /tmp/pvfs2-client.log
> root     23623 19719  0 15:28 pts/2    00:00:00 grep pvfs
> 
> 
> Any thoughts??? greatly appreciated!!!
> -Amit
> 
>> -----Original Message-----
>> From: Phil Carns [mailto:carns at mcs.anl.gov]
>> Sent: Wednesday, September 16, 2009 2:55 PM
>> To: ligon at clemson.edu
>> Cc: Kumar, Amit H.; pvfs2-developers at beowulf-underground.org
>> Subject: Re: [Pvfs2-developers] PVFS2: files with ?---?--? permissions
>>
>> Becky Ligon wrote:
>>> Amit:
>>>
>>> This means that the PVFS system cannot access the attributes database
>>> containing the information about the particular file.  It also means
>> that
>>> the file is unusable.  You need to determine which metadata server is
>>> having problems.  If you don't have a backup, then you may not be
>> able to
>>> recover your file.
>>>
>>> Here at Clemson, we just went through this painful process, when one
>> of
>>> the Berkeley DB holding metadata became corrupt.  You might try
>>> db_recover, but it didn't help us.  You might also try pvfs2-fsck.
>> If
>>> your file is unrecoverable, then pvfs2-fsck just simply cleans up the
>>> orphan objects.
>>>
>>> Becky
>> Hi Amit,
>>
>> Do you have anything in your server logs or the /tmp/pvfs2-client.log
>> file on the client side?
>>
>> thanks,
>> -Phil