[Pvfs2-developers] PVFS2: files with ?---?--? permissions

[Kumar, Amit H.]

Hi Everyone!

Thank you for responding. I enabled PVFS2_DEBUGMASK to see what was going on, before I responded.

 Attached are the /tmp/pvfs2-client.log and /tmp/pvfs2-server.log and /var/log/pvfs2.debug.log(PVFS2_DEBUGFILE) files

I do have the latest version, but not from the CVS. And I don't have a backup, so we will potentially lose some data :-(

I ran pvfs-fsck on the metadata server that has a local mount of the pvfs2-FS, and get the following errors/warnings. Surprisingly I don't have any problem browsing the file system until I hit the files with corrupt attributes. 

# /opt/pvfs2/bin/pvfs2-fsck -vp -m /scratch/pvfs2
# Current FSID is 1415574627.
[E 12:09:56.029492] job_time_mgr_expire: job time out: cancelling bmi operation, job_id: 18.
[E 12:09:56.030046] Warning: msgpair failed to ib://pvfs2-io-0-0.local-ib:3335, will retry: Connection timed out
[E 12:09:56.030118] *** msgpairarray_completion_fn: msgpair to server ib://pvfs2-io-0-0.local-ib:3335 failed: Connection timed out
[E 12:09:56.030164] *** Non-BMI failure.
PVFS_mgmt_setparam_list: Connection timed out (error class: 0)

I believe the time out is a result of not being able to access the files with corrupt attributes.


Our Metadata server & one of the I/O server =>[root at pvfs2-io-0-0 tmp]# ps -ef | grep pvfs
root     19964     1  0 Sep14 ?        00:24:02 /opt/pvfs2/sbin/pvfs2-server --pidfile /var/run/pvfs2.pid -a pvfs2-io-0-0.local-ib /opt/pvfs2/etc/pvfs2-fs.conf
root     20434     1  0 Sep14 ?        00:00:00 /opt/pvfs2/sbin/pvfs2-client -p /opt/pvfs2/sbin/pvfs2-client-core
root     20435 20434  0 Sep14 ?        00:00:05 pvfs2-client-core --child -a 5 -n 5 --logtype file -L /tmp/pvfs2-client.log
root     23623 19719  0 15:28 pts/2    00:00:00 grep pvfs


Any thoughts??? greatly appreciated!!!
-Amit

> -----Original Message-----
> From: Phil Carns [mailto:carns at mcs.anl.gov]
> Sent: Wednesday, September 16, 2009 2:55 PM
> To: ligon at clemson.edu
> Cc: Kumar, Amit H.; pvfs2-developers at beowulf-underground.org
> Subject: Re: [Pvfs2-developers] PVFS2: files with ?---?--? permissions
> 
> Becky Ligon wrote:
> > Amit:
> >
> > This means that the PVFS system cannot access the attributes database
> > containing the information about the particular file.  It also means
> that
> > the file is unusable.  You need to determine which metadata server is
> > having problems.  If you don't have a backup, then you may not be
> able to
> > recover your file.
> >
> > Here at Clemson, we just went through this painful process, when one
> of
> > the Berkeley DB holding metadata became corrupt.  You might try
> > db_recover, but it didn't help us.  You might also try pvfs2-fsck.
> If
> > your file is unrecoverable, then pvfs2-fsck just simply cleans up the
> > orphan objects.
> >
> > Becky
> 
> Hi Amit,
> 
> Do you have anything in your server logs or the /tmp/pvfs2-client.log
> file on the client side?
> 
> thanks,
> -Phil
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2.debug.log
Type: application/octet-stream
Size: 2195 bytes
Desc: pvfs2.debug.log
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20090916/46c7e37b/pvfs2.debug-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2-client.log
Type: application/octet-stream
Size: 14791 bytes
Desc: pvfs2-client.log
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20090916/46c7e37b/pvfs2-client-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2-server.log
Type: application/octet-stream
Size: 146437 bytes
Desc: pvfs2-server.log
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20090916/46c7e37b/pvfs2-server-0001.obj