<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div>David,<div><br></div><div>I hate to question what you've said, but are you sure that you were getting good performance with 2.8.0? Is it possible that you only got good performance with 2.7.1, and that switching to 2.8.0 (and 2.8.1) has caused this performance degradation? I ask because (as Rob has hinted at) we changed the way we manage the side of datafiles in releases >=2.8.0, and we've seen performance drops for serial, small file workloads. Its a bug, and we've fixed it in CVS, but you may be seeing it in your setup.</div><div><br></div><div>-sam</div><div><br><div><div>On Jul 1, 2009, at 5:57 PM, David Bonnie wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Sam -<br><br>All of the nodes checked out fine with netpipe, still no errors on any of the adapters.<br><br>- Dave<br><br><div class="gmail_quote">On Wed, Jul 1, 2009 at 4:47 PM, Sam Lang <span dir="ltr"><<a href="mailto:slang@mcs.anl.gov">slang@mcs.anl.gov</a>></span> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style=""><br><div><div class="im"><div>On Jul 1, 2009, at 5:45 PM, David Bonnie wrote:</div> <br><blockquote type="cite">I'll run it on each node and let you know if anything is out of place. I believe the above results are fine for GigE, yes?<br></blockquote><div><br></div></div>They certainly don't match with the numbers you're getting from PVFS.</div> <div>-sam</div><div><div></div><div class="h5"><div><br><blockquote type="cite"><br>- Dave<br><br><div class="gmail_quote">On Wed, Jul 1, 2009 at 4:20 PM, Sam Lang <span dir="ltr"><<a href="mailto:slang@mcs.anl.gov" target="_blank">slang@mcs.anl.gov</a>></span> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div><br></div>David,<div><br></div><div>It sounds like your initial thought (that there is a network problem) could be correct. I would probably explore that first. What sort of numbers do you get from netpipe runs (or even bmi_pingpong) between client and server?</div> <div><br></div><font color="#888888"><div>-sam</div></font><div><div></div><div><div><br><div><div>On Jul 1, 2009, at 5:15 PM, David Bonnie wrote:</div><br><blockquote type="cite">Sorry for not being clear.<br> <br>The hardware and software is unchanged. Runs from a few months ago (on 2.8.0) performed as expected. Current runs (on both 2.8.0 and 2.8.1) are slow.<br> <br>The nodes are sitting there with very low CPU usage even when running the benchmark. I'm the only one running any jobs and there aren't any processes running (the system load is < .02 and the cpu usage is pretty much 0%).<br> <br>The local disks haven't changed and are empty except for the pvfs2 storage space; performance is bad even when I put the PVFS2 file system storage onto a very fast (>300 MB/s local bandwidth) Atrato vlun connected over fiber channel.<br> <br>My initial thought is that some hardware along the line died but I can't seem to pinpoint it. All of the network interfaces show 0 errors and 0 dropped packets.<br><br>- Dave<br><br><div class="gmail_quote">On Wed, Jul 1, 2009 at 4:10 PM, Rob Ross <span dir="ltr"><<a href="mailto:rross@mcs.anl.gov" target="_blank">rross@mcs.anl.gov</a>></span> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi David,<br> <br> I still don't get it: when was the performance good? Same software and hardware, just some time in the past? Or is there a software change?<br> <br> The nodes aren't being used for anything else, there are no rogue processes, and the local file systems are otherwise empty?<br> <br> Thanks,<br><font color="#888888"> <br> Rob<br></font><div> <br> On Jul 1, 2009, at 5:05 PM, David Bonnie wrote:<br> <br> </div><div><div></div><div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Rob -<br> <br> Performance is down across all PVFS2 installations. The benchmark simply creates files of a random size (between 1 and 25 MB) in a single folder on the mounted PVFS2 partition, 16 KB at a time. It's not anywhere near ideal, but it's the workload I'm working with.<br> <br> Prior to this problem we were getting ~22 MB/s write throughput and we're down to about 2.5 MB/s for no apparent reason. Reads are down from about 55 MB/s to 30 MB/s. No hardware has changed and as far as I can tell no hardware has died either.<br> <br> - Dave<br> <br> <br> On Wed, Jul 1, 2009 at 4:00 PM, Rob Ross <<a href="mailto:rross@mcs.anl.gov" target="_blank">rross@mcs.anl.gov</a>> wrote:<br> Do you mean that 2.8.0 is fast and 2.8.1 is slow? Can you describe the benchmark and how you are doing your measurements?<br> <br> Rob<br> <br> <br> On Jul 1, 2009, at 4:43 PM, David Bonnie wrote:<br> <br> Hello all -<br> <br> I'm having trouble figuring out a problem with performance depredation on a simple 10 node cluster. Prior runs on the cluster (before this problem manifested itself) resulted in bandwidth and IOPS about 10 times higher on a small file creation workload. Each node is running as a metadata server and a data server.<br> <br> The problem is persistent between versions and installations of PVFS2 2.8.0 and 2.8.1. Rebooting all of the nodes didn't improve anything. The network connections (simple GigE) showed no errors or dropped packets. Using different physical disks (both SAS and FC) didn't improve things. The kernel logs didn't show anything out of place nor did the pvfs2 server or client logs. It seems like a network issue but I can't seem to find anything wrong with any of the connections.<br> <br> Has anyone seen this kind of problem before? I seem to remember something on the list before about performance suddenly dropping but I can't find the message now (of course). Any insight would be appreciated!<br> <br> Thanks,<br> <br> - Dave<br> _______________________________________________<br> Pvfs2-developers mailing list<br> <a href="mailto:Pvfs2-developers@beowulf-underground.org" target="_blank">Pvfs2-developers@beowulf-underground.org</a><br> <a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><br> <br> <br> </blockquote> <br> </div></div></blockquote> </div><br> _______________________________________________<br>Pvfs2-developers mailing list<br><a href="mailto:Pvfs2-developers@beowulf-underground.org" target="_blank">Pvfs2-developers@beowulf-underground.org</a><br><a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><br> </blockquote></div><br></div></div></div></div><br>_______________________________________________<br> Pvfs2-developers mailing list<br> <a href="mailto:Pvfs2-developers@beowulf-underground.org" target="_blank">Pvfs2-developers@beowulf-underground.org</a><br> <a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><br> <br></blockquote></div><br></blockquote></div><br> </div></div></div><br>_______________________________________________<br> Pvfs2-developers mailing list<br> <a href="mailto:Pvfs2-developers@beowulf-underground.org">Pvfs2-developers@beowulf-underground.org</a><br> <a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers" target="_blank">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><br> <br></blockquote></div><br></blockquote></div><br></div></body></html>