[Pvfs2-developers] Bad File Entries

Wed Jun 30 15:56:31 EDT 2010

On Jun 30, 2010, at 12:42 PM, Bart Taylor wrote:

> Hi Sam,
> 
> First to answer your underlying file system question, this is using ext3.
> 
> Thanks for the patch. I did a quick rebuild for my client using it and have had mixed results. pvfs2-rm will now get the job done for files that return ENOENT, but /bin/rm does not remove them. Getting /bin/rm to remove the files would be great, but having pvfs2-rm get them is definitely helpful. Thanks for that.

Hmm..  it looks like the rm util is actually doing an lstat first, and getting an ENOENT from that.  You may be able to get around that by passing the -r option.  It sounds weird, but rm is doing the stat to get the type of file -- it won't remove if it if its a directory -- and telling rm to recurse just might avoid the stat.  Probably not, but I'm curious if you get a chance to give it a try.

> 
> I put that test script in a loop and left it running overnight, and I stumbled into a few bits of information that might be useful. 
> 
> The first is that I seem to be getting two different return codes from some of these bad file entries. The files that return ENOENT can be removed with pvfs2-rm after your patch, but some of the entries are now returning "Input/output error" messages. I assume this could be related to the EIO messages you mentioned on XFS. The files that return this message cannot be removed by either /bin/rm or pvfs2-rm.

Right.  The patch I sent allows the rmdirent to proceed even when the remove of the metadata fails first.  In the ENOENT case, this is ok, because the server is indicating that the metadata is gone anyway.  For other errors, I want to be careful not to remove the directory entry when somehow the metadata (and data!) is still actually in place.  It may be that EIO is getting returned from the local file system, because it has entered a bad state.  unmounting, running fsck, and remounting may get rid of the EIO errors.  I would check the server logs and see what error messages they contain if you're seeing EIO.

> 
> The second is that apparently some files are being affected that are not touched by my test at all. That script creates some directories and then creates files within those directories. I had a large file in the same directory where my script was running but unrelated to the test (different file name completely, so no chance for accidentally hitting the same name) that was left in this bad state when the tests were finished. Does the fact that other files are being affected help shed any light on where the cause may lie?

No that leaves me more puzzled.  I should go back and check, but do you get other error messages in the server logs besides timeouts?
-sam

> 
> I attached a text files with the results of trying to delete files that return ENOENT and the Input/output error message. Let me know if I can provide anything else.
> 
> Thanks for your help,
> Bart.
> 
> 
> 
> 
> On Tue, Jun 29, 2010 at 12:40 PM, Sam Lang <slang at mcs.anl.gov> wrote:
> 
> Hi Bart,
> 
> I haven't had a chance to do a whole lot of debugging of this problem, although I did run your test once.  I was using XFS as the underlying filesystem, and after running successfully for a while, the XFS mount just kind of blew up on me, and started returning EIO errors for everything. I will try the test again with ext2 (is that what you're using?), but in the meantime, I wanted to address the actual problem as you described, which is that you can't remove those bad file entries easily.  I've attached a patch which I think will fix the issue, allowing you to remove those bad entries with either rm or pvfs2-rm.  I haven't been able to test this fully, but its only a one-liner, so there shouldn't be any unintended side-affects.  Do you want to give it a try?
> 
> -sam
> 
> 
> 
> 
> On Jun 18, 2010, at 3:21 PM, Bart Taylor wrote:
> 
> > Yes. I reran the test again so that I could grab the actual messages, and this time it was a bit more aggressive. Instead of leaving a file in a bad state, it left my whole directory structure under the file system root in that state. I attached a chunk of log messages from the client and from the server that was timing out. The other servers did not log anything.
> >
> > I am currently getting this back from pvfs2-fsck:
> >
> > server 1, exceeding number of handles it declared (42923), currently (43000)
> > pvfs28-fsck: ../pvfs2_src/src/apps/admin/pvfs2-fsck.c:1325: handlelist_add_handles: Assertion `0' failed.
> > Aborted
> >
> >
> > Bart.
> >
> >
> >
> > On Fri, Jun 18, 2010 at 8:15 AM, Sam Lang <slang at mcs.anl.gov> wrote:
> >
> > Hi Bart,
> >
> > When you run the script, do you see any timeout error messages in the client log?
> >
> > -sam
> >
> > On Jun 18, 2010, at 9:03 AM, Bart Taylor wrote:
> >
> > > Hey Phil,
> > >
> > > Yes, it is running 2.8.2.  My setup was using 3 servers with 2.6.18-194.el5 kernels and High Availability. I have not had a chance yet to try it on another file system, so I do not know if it is specific to that setup. It has been triggered from more than one client, but the only know I know for certain was running a 2.6.9-89.ELsmp kernel.
> > >
> > > Bart.
> > >
> > >
> > > On Fri, Jun 18, 2010 at 7:39 AM, Phil Carns <carns at mcs.anl.gov> wrote:
> > > Hi Bart,
> > >
> > > Is this on 2.8.2?  Do you happen to know how many servers are needed to trigger the problem?
> > >
> > > thanks,
> > > -Phil
> > >
> > >
> > > On 06/17/2010 04:08 PM, Bart Taylor wrote:
> > >>
> > >> Hey guys,
> > >>
> > >> We have had some problems in the past on 2.6 with file creations leaving bad
> > >> files that we cannot delete. Most utilities like ls and rm return "No such file
> > >> or directory", and pvfs utilities like viewdist, pvfs2-ls, and pvfs2-rm return
> > >> various errors. We have resorted to looking up the parent handle, the fsid, and
> > >> filename and using pvfs2-remove-object to delete the entry. But we weren't ever
> > >> able to intentionally recreate the problem.
> > >>
> > >> Recently while testing 2.8, I have been able to reliably trigger a similar
> > >> scenario where a file creation fails and leaves a garbage entry that cannot be
> > >> deleted in any of the normal ways requiring the pvfs2-remove-object approach to
> > >> clean up. The file and various outputs for this case:
> > >>
> > >> [root at client dir]# ls -l 2010.06.10.28050
> > >> total 0
> > >> ?---------  ? ? ? ?           ? File17027
> > >>
> > >> [root at client dir]# rm 2010.06.10.28050/File17027
> > >> rm: cannot lstat `2010.06.10.28050/File17027': No such file or directory
> > >>
> > >> [root at client dir]# rm -rf 2010.06.10.28050
> > >> rm: cannot remove directory `2010.06.10.28050': Directory not empty
> > >>
> > >> [root at client dir]# pvfs2-rm 2010.06.10.28050/File17027
> > >> Error: An error occurred while removing 2010.06.10.28050/File17027
> > >> PVFS_sys_remove: No such file or directory (error class: 0)
> > >>
> > >> [root at client dir]# pvfs2-stat 2010.06.10.28050/File17027
> > >> PVFS_sys_lookup: No such file or directory (error class: 0)
> > >> Error stating [2010.06.10.28050/File17027]
> > >>
> > >> [root at client dir]# pvfs2-viewdist -f 2010.06.10.28050/File17027
> > >> PVFS_sys_lookup: No such file or directory (error class: 0)
> > >> Could not open 2010.06.10.28050/File17027
> > >>
> > >> [root at client dir]# ls -l 2010.06.10.28050
> > >> total 0
> > >> ?---------  ? ? ? ?           ? File17027
> > >>
> > >>
> > >> I have included a test script that will spawn off a number of processes, open a
> > >> bunch of files, write to each of them, then close them. You can tweak the
> > >> options as you want but using 5 processes and 50,000 files will usually create
> > >> at least one of these files. Here is an example command:
> > >>
> > >> $> ulimit -n 1000000 && ./open-file-limit --num-files=50000 --sleep-time=1 --num-processes=5 --directory=/mnt/pvfs2/ --file-size=1
> > >>
> > >> You may have to do a long listing on any left-over directories to find the file(s).
> > >>
> > >> I will give any help I can to help recreate the bad file or find the cause.
> > >> Until then, is there a better (simpler) way to remove these entries, maybe
> > >> some sort of utility that doesn't require doing manual handle lookups before
> > >> getting the file removed? It would ease some support pain if it were simpler to
> > >> fix.
> > >>
> > >> Thanks for your help,
> > >> Bart.
> > >>
> > >> _______________________________________________
> > >> Pvfs2-developers mailing list
> > >>
> > >> Pvfs2-developers at beowulf-underground.org
> > >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> > >>
> > >>
> > >>
> > >
> > >
> > > _______________________________________________
> > > Pvfs2-developers mailing list
> > > Pvfs2-developers at beowulf-underground.org
> > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> > >
> > >
> > > _______________________________________________
> > > Pvfs2-developers mailing list
> > > Pvfs2-developers at beowulf-underground.org
> > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> >
> >
> > <server2-log.TXT><client-log.TXT>
> 
> 
> 
> <rm-results.TXT>