Sam -<br><br>I'm testing with only 50 files now...initially the tests were run in 1000 file increments from 1000 to 6000 files. The times scale almost perfectly linearly at all data points.<br><br>- Dave<br><br><div class="gmail_quote">
On Wed, Jul 1, 2009 at 4:08 PM, Sam Lang <span dir="ltr"><<a href="mailto:slang@mcs.anl.gov">slang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style=""><div><br></div><div><div><div class="im"><div>On Jul 1, 2009, at 5:05 PM, David Bonnie wrote:</div><br><blockquote type="cite">Rob -<br><br>Performance is down across all PVFS2 installations. The benchmark simply creates files of a random size (between 1 and 25 MB) in a single folder on the mounted PVFS2 partition, 16 KB at a time. It's not anywhere near ideal, but it's the workload I'm working with.<br>
</blockquote><div><br></div></div>How many files?<div>-sam</div><div class="im"><div><br><div></div></div><blockquote type="cite"> <br>Prior to this problem we were getting ~22 MB/s write throughput and we're down to about 2.5 MB/s for no apparent reason. Reads are down from about 55 MB/s to 30 MB/s. No hardware has changed and as far as I can tell no hardware has died either.<br>
<br>- Dave<br><br><br><div class="gmail_quote">On Wed, Jul 1, 2009 at 4:00 PM, Rob Ross <span dir="ltr"><<a href="mailto:rross@mcs.anl.gov" target="_blank">rross@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Do you mean that 2.8.0 is fast and 2.8.1 is slow? Can you describe the benchmark and how you are doing your measurements?<br> <br> Rob<div><div></div><div><br> <br> On Jul 1, 2009, at 4:43 PM, David Bonnie wrote:<br> <br>
</div></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div> Hello all -<br> <br> I'm having trouble figuring out a problem with performance depredation on a simple 10 node cluster. Prior runs on the cluster (before this problem manifested itself) resulted in bandwidth and IOPS about 10 times higher on a small file creation workload. Each node is running as a metadata server and a data server.<br>
<br> The problem is persistent between versions and installations of PVFS2 2.8.0 and 2.8.1. Rebooting all of the nodes didn't improve anything. The network connections (simple GigE) showed no errors or dropped packets. Using different physical disks (both SAS and FC) didn't improve things. The kernel logs didn't show anything out of place nor did the pvfs2 server or client logs. It seems like a network issue but I can't seem to find anything wrong with any of the connections.<br>
<br> Has anyone seen this kind of problem before? I seem to remember something on the list before about performance suddenly dropping but I can't find the message now (of course). Any insight would be appreciated!<br>
<br> Thanks,<br> <br> - Dave<br></div></div> _______________________________________________<br> Pvfs2-developers mailing list<br> <a href="mailto:Pvfs2-developers@beowulf-underground.org" target="_blank">Pvfs2-developers@beowulf-underground.org</a><br>
<a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><br> </blockquote> <br> </blockquote></div><br> _______________________________________________<br>
Pvfs2-developers mailing list<br><a href="mailto:Pvfs2-developers@beowulf-underground.org" target="_blank">Pvfs2-developers@beowulf-underground.org</a><br><a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><br>
</blockquote></div></div><br></div></div></blockquote></div><br>