[Pvfs2-users] filesystem errors - pvfs2-fsck failing

[Bill Wichser]

The reboot of the I/O servers have now cleared up this failure.  There 
is still some inconsistency
in the filesystem while running pvfs2-fsck but I believe that I can 
clean these up.

Thanks,
Bill

Michael Moore wrote:
> Hi Bill,
>
> Sorry for the delay. The difference appears to be between what the 
> management iterate handles call returns and what statfs returns to fsck. 
> I'm looking now to get a better understanding how statfs and the trove 
> ledger stuff gets it's counts versus how iterate handles counts them. 
>
> In the mean time, have the server processes been restarted since this 
> behavior started occurring? If not, is that a possibility?
>
> Sorry again for the delay in getting back with you on this issue.
>
> Thanks,
> Michael
>
> On Tue, Jan 04, 2011 at 01:48:20PM -0500, Bill Wichser wrote:
>   
>> I've deleted those files with the native pvfs2-rm command which informed 
>> me to run pvfs2-fsck.  Running pvfs2-validate turned up a number more 
>> which I removed.  So there is nothing to pvfs2-viewdist on.
>>
>> FWIW I'm running a meta on the head and the I/O servers on 16 compute 
>> nodes, version 2.8.2
>>
>> [root at della3 bill]# pvfs2-stat /scratch/pvfs2
>> -------------------------------------------------------
>>   File Name     : /scratch/pvfs2
>>   Relative Name : /
>>   fs ID         : 1922795883
>>   Handle        : 1048576
>>   Mask          : 504000177
>>   Permissions   : 777
>>   Type          : Directory
>>   Size          : 4096
>>   Owner         : 0 (root)
>>   Group         : 0 (root)
>>   atime         : 1294130281 (Tue Jan  4 03:38:01 2011)
>>   mtime         : 1293499466 (Mon Dec 27 20:24:26 2010)
>>   ctime         : 1293499462 (Mon Dec 27 20:24:22 2010)
>>   dir entries   : 6
>>
>> [root at della3 bill]# pvfs2-validate -d /scratch/pvfs2/
>> pvfs2-validate starting validation at object [/scratch/pvfs2]
>> pvfs2-validate done validating object tree at [/scratch/pvfs2]
>>
>> [root at della3 bill]# pvfs2-fsck -p -m /scratch/pvfs2
>> # Current FSID is 1922795883.
>> Ugh! Server 1, Received 64789 total handles instead of 64792
>>
>> So the total handles have changed, as expected because of the removals, 
>> but the difference is the same.  Now to be honest, when I made that 
>> filesystem, I didn't run an fsck so it could be a remnant from last 
>> month.  I don't know.  But we have a bunch of Genomics people wrecking 
>> havoc with those strange files in kernel space.  I was able to do an 
>> pvfs2-ls on them (user space) but didn't really pursue, hoping instead 
>> to just make the problem go away!
>>
>> Thanks,
>> Bill
>>
>> Michael Moore wrote:
>>     
>>> Hi Bill,
>>>
>>> Can you provide the output of pvfs2-stat on the parent directory 
>>> and affected files and 'pvfs2-viewdist -f <path>' on the affected files?
>>>
>>> Do you see any complaints in the server logs related to accessing these 
>>> files?
>>>
>>> Michael
>>>
>>> On Mon, Jan 03, 2011 at 08:04:02AM -0500, Bill Wichser wrote:
>>>   
>>>       
>>>> Having some trouble with my filesystem.  There are a few files which did 
>>>> not get written correctly by one of the users and some corruption looks 
>>>> to be present.
>>>>
>>>> # ls -lR
>>>> ./3689_old:
>>>> total 0
>>>> ?--------- ? ? ? ?            ? clusmax.out
>>>>
>>>> ./3764_old:
>>>> total 0
>>>> ?--------- ? ? ? ?            ? traj.xtc
>>>>
>>>> These cannot be removed.  In the past, a run of pvfs2-fsck seemed to 
>>>> correct these types of problems but this time all I get is the following 
>>>> message and the fsck terminates.  I'm not sure how to correct this.  
>>>> Googling leads me to the source code.  Anyone have any suggestions?
>>>>
>>>> # pvfs2-fsck -p -v -m /scratch/pvfs2
>>>> # Current FSID is 1922795883.
>>>> Ugh! Server 1, Received 64796 total handles instead of 64800
>>>>
>>>>
>>>> Thanks, and Happy New Year to all!
>>>> Bill
>>>>
>>>> _______________________________________________
>>>> Pvfs2-users mailing list
>>>> Pvfs2-users at beowulf-underground.org
>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>>     
>>>>         
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>