Thanks Phil, that patch works great! I successfully ran the previous script on a few thousand mounts and did not have any more problems. <br><br>It looks like you nailed down where the problem lies even if it isn't fixed in every case. I ran the 'for' loop you mentioned that bounces off the acache perf-counter just to see what would happen, and it still causes a hang as I think you expected.<br>
<br>Bart.<br><br><br><br><div class="gmail_quote">On Tue, May 13, 2008 at 7:55 AM, Phil Carns <<a href="mailto:carns@mcs.anl.gov">carns@mcs.anl.gov</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Bart,<br>
<br>
Could you try this patch out and see if it fixes your problem? This is checked into trunk as well. This won't eliminate the inode alloc warning, but I think it does actually fix the umount hang.<br>
<br>
I also suspect that this same issue may affect a few other cases as well, but it would be good if you could confirm this much for starters.<br>
<br>
I think the same class of bug is affecting some of the proc file handlers, for example. Cases like this also cause a pvfs2-client-core hang:<br>
<br>
"for i in `seq 1 100`; do echo $i; cat /proc/sys/pvfs2/perf-counters/acache; done"<br>
<br>
thanks!<br>
-Phil<br>
<br>
Bart Taylor wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div></div><div class="Wj3C7c">
Hey guys,<br>
<br>
I have been running some tests against the 271 release, and I am having some trouble with multiple mounts on one client. My setup has 2 servers (both meta and io servers on local disk) and one client all of which are running RHEL4 update 6. All that was done on the test client is loading the kernel module and starting pvfs2-client. I can mount the file system once and use it without any problem, but I have attached a test script - takes file system information and a number of times to mount it - that keeps failing. Here are the steps it executes:<br>
<br>
- For the number of mounts requested<br>
- Create a new directory (defaults to /tmp/mount_limit.#)<br>
- Mount the specified file system on the new dir<br>
<br>
- For the number of mounts requested<br>
- Do a recursive ls comparison (keep a copy the first time through and compare subsequent mounts to the first)<br>
- Unmount the dir<br>
- Delete the dir<br>
<br>
I have been able to consistently reproduce the problem running the attached script like this:<br>
./test-mount-limit.pl pvfs2-server1:3334/pvfs2-fs 100<br>
It stalls every time with either 36 or 37 mounts remaining. The script has been successfully run on previous versions of pvfs2 up to several thousand mounts.<br>
<br>
The problem comes at the umount step. Eventually the process just hangs, strands a bunch of mounts, and umount doesn't work as expected after that even from the command line. When it stalls, I start seeing messages like this one in dmesg and syslog:<br>
May 2 15:02:44 client-node kernel: pvfs2_kill_sb: (WARNING) number of inode allocs (4100) != number of inode deallocs (2665)<br>
<br>
I am running this against an almost empty file system since the recursive ls would take a while if it were large. Am I doing something wrong/strange here, or is there a client/kernel problem? The test seems pretty straight-forward, and I've never had an issue with the script before. I'm not sure if it was run against the 2.7.0 release though.<br>
<br>
Bart.<br>
<br>
<br></div></div>
------------------------------------------------------------------------<div class="Ih2E3d"><br>
<br>
_______________________________________________<br>
Pvfs2-developers mailing list<br>
<a href="mailto:Pvfs2-developers@beowulf-underground.org" target="_blank">Pvfs2-developers@beowulf-underground.org</a><br>
<a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><br>
</div></blockquote>
<br>
</blockquote></div><br>