[Pvfs2-developers] duplicate entries in directory listing

[Rob Ross]

Hi Phil,

That's been around for a while (years). It's an artifact of ls reading 
pieces of the directory list at a time, and the directory list changing 
between directory reads.

We've talked about having pvfs2-client pull out duplicates (or the 
kernel module) in the cases where one of those chooses to break a 
readdir into multiple operations, but we haven't spent much time 
investigating where the replication is actually happening in order to 
accomplish this.

Other solutions could include locking the directory (not going to 
happen), restarting the ls entirely if the directory changes during the 
read (would cause starvation), improving ls to remove duplicates on its 
own (probably realistic for pvfs2-ls, unlikely to get accepted by GNU 
tools group for stock ls), and reordering directory entries returned so 
that most recently changing entries are returned last (high overhead on 
server to sort, lots of coding probably).

Any other ideas?

Rob

Phil Carns wrote:
> We are seeing a strange bug where if we list the contents of a directory
> while files are being created in it, we sometimes get duplicates and/or
> missing files in the output.
> 
> I can reproduce it on a single machine by running these two scripts at
> the same time:
> 
> tester.sh:
> -----------------------------------
>    #!/bin/tcsh
> 
>    foreach file ( `seq 1 10000` )
>       touch /mnt/pvfs2/testdir/${file}
>    end
> 
> watcher.sh:
> -----------------------------------
>    #!/bin/tcsh
> 
>    there:
>       set foo=`ls /mnt/pvfs2/testdir | wc -l`
>       set bar=`ls /mnt/pvfs2/testdir | uniq -d | wc -l`
>       echo listing count: $foo, duplicates: $bar
>       sleep 1
>    goto there
> 
> 
> The test machine that I am using is pretty slow.  On faster machines you
> may need to create more than 10,000 files, or maybe slow it down by
> actually writing a little bit of data into each file.
> 
> At any rate, the output looks normal for a while, but then we start
> seeing results like this from watcher.sh:
> 
> ...
> listing count: 6310, duplicates: 0
> listing count: 6320, duplicates: 0
> listing count: 6334, duplicates: 0
> listing count: 6371, duplicates: 0
> listing count: 6382, duplicates: 0
> listing count: 6396, duplicates: 5024
> listing count: 6406, duplicates: 0
> listing count: 10896, duplicates: 5344
> listing count: 6430, duplicates: 5120
> listing count: 6434, duplicates: 0
> listing count: 11574, duplicates: 6048
> listing count: 6472, duplicates: 0
> ...
> 
> The listing count is supposed to steadily increase, and the duplicates
> field should always be zero.  The problem only occurs while files are
> being created.  Once tester.sh is done, the listing looks perfectly
> normal.
> 
> Anyone have any ideas?  I think this problem has been hanging around for
> a little while but we just now figured out how to reliably trigger it. 
> It is at least in current cvs head and was in a snapshot from August 21.
> 
> -Phil