[Pvfs2-developers] PVFS2 Removal of large files

[Phil Carns]

Thanks David, this looks good.  I committed the patch to our cvs trunk 
so that it can get some mileage in nightly testing.

At some point before releasing this we will need a way to clean up the 
stranded bstreams directory if the server dies before the files are 
unlinked.  That may just be a matter of adding files in that stranded 
directory to the unlink queue on startup so that the server can work on 
them in the background.

Here is a little more information about #3 that David mentioned in case 
anyone is interested.  The remove stall pointed out a flaw in the BMI 
reference counting that at least led to some confusing log messages. 
I'm not sure if there were any more serious side effects or not:

http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:pcarns:20081008183827

-Phil

David Metheny wrote:
> XFS might be a good change for us in the future, thanks for the suggestion.
> For me, it won't be an option I can introduce immediately. 
> 
> I've created a patch that basically starts a persistent thread to take care
> of 'unlink' requests. This is patched against the 2.6.3 release. 
> 
> The unlink now renames the file (puts it in the stranded-bstreams dir), adds
> it to a shared queue, signals the unlink thread to delete it, and returns.. 
> 
> There were a several side effects that the block on deletion caused us. 
> 1) the pvfs2-check-server called via heartbeat would timeout, getting caught
> behind the remove, and would send heartbeat STONITH'ing away. After STONITH
> on a 'rm' command, that sent our 3+TB devices into fsck for a bit, and
> client jobs canceled away. 
> 2) Other accesses, like 'ls' and reads/writes also queued behind deletes,
> and then they started canceling as well. 
> 3) Once the BMI jobs started timing out and retries sent, there were some
> cases where canceling those jobs didn't quite stop accesses to the jobs
> memory. Phil Carns sent me a patch to fix this. 
> 
> I've ran a small sample of tests on a single PVFS2 server with a separate
> machine for the PVFS2 client. Both machines are running RHEL4 U7 i386 WS.
> The short version is that there was little affect on current performance for
> small files, and a large performance increase from a clients perspective for
> large files. 
> 
> +-----------+-----------+--------------+----------------+
> + Num Files + File Size + Time (stock) + Time (threaded)+
> +-----------+-----------+--------------+----------------+
> +   10	+ 0 bytes	+  0.55 (secs) + 0.51 (secs)    +
> +-----------+-----------+--------------+----------------+
> +   500 	+ 100 bytes	+  35 (secs)   + 33 (secs)      +
> +-----------+-----------+--------------+----------------+
> +  10,000   + 100 bytes	+  728 (secs)  + 702 (secs)     +
> +-----------+-----------+--------------+----------------+
> +   10	+ 500 MB	+  7 (secs)    + 0.73 (secs)    +
> +-----------+-----------+--------------+----------------+
> 
> 
> -----Original Message-----
> From: Emmanuel Florac [mailto:eflorac at intellique.com]
> Sent: Friday, October 03, 2008 1:50 AM
> To: david.metheny at gmail.com
> Cc: pvfs2-developers at beowulf-underground.org
> Subject: Re: [Pvfs2-developers] PVFS2 Removal of large files
> 
> Le Thu, 2 Oct 2008 16:57:19 -0500 vous écriviez:
> 
>> What are you thinking along the lines of tuning the client side?
> 
> I'd strongly encourage you NOT to use ext3. ext3 is by far the slowest
> filesystem under Linux, and it behaves very poorly on big filesystems.
> I'd definetely go for XFS ( or JFS ) on the underlying device.
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers