[Pvfs2-developers] PVFS2 Performance Problem -

[Sam Lang]

David,

It sounds like your initial thought (that there is a network problem)  
could be correct.  I would probably explore that first.  What sort of  
numbers do you get from netpipe runs (or even bmi_pingpong) between  
client and server?

-sam

On Jul 1, 2009, at 5:15 PM, David Bonnie wrote:

> Sorry for not being clear.
>
> The hardware and software is unchanged.  Runs from a few months ago  
> (on 2.8.0) performed as expected.  Current runs (on both 2.8.0 and  
> 2.8.1) are slow.
>
> The nodes are sitting there with very low CPU usage even when  
> running the benchmark.  I'm the only one running any jobs and there  
> aren't any processes running (the system load is < .02 and the cpu  
> usage is pretty much 0%).
>
> The local disks haven't changed and are empty except for the pvfs2  
> storage space; performance is bad even when I put the PVFS2 file  
> system storage onto a very fast (>300 MB/s local bandwidth) Atrato  
> vlun connected over fiber channel.
>
> My initial thought is that some hardware along the line died but I  
> can't seem to pinpoint it.  All of the network interfaces show 0  
> errors and 0 dropped packets.
>
> - Dave
>
> On Wed, Jul 1, 2009 at 4:10 PM, Rob Ross <rross at mcs.anl.gov> wrote:
> Hi David,
>
> I still don't get it: when was the performance good? Same software  
> and hardware, just some time in the past? Or is there a software  
> change?
>
> The nodes aren't being used for anything else, there are no rogue  
> processes, and the local file systems are otherwise empty?
>
> Thanks,
>
> Rob
>
> On Jul 1, 2009, at 5:05 PM, David Bonnie wrote:
>
> Rob -
>
> Performance is down across all PVFS2 installations.  The benchmark  
> simply creates files of a random size (between 1 and 25 MB) in a  
> single folder on the mounted PVFS2 partition, 16 KB at a time.  It's  
> not anywhere near ideal, but it's the workload I'm working with.
>
> Prior to this problem we were getting ~22 MB/s write throughput and  
> we're down to about 2.5 MB/s for no apparent reason.  Reads are down  
> from about 55 MB/s to 30 MB/s.  No hardware has changed and as far  
> as I can tell no hardware has died either.
>
> - Dave
>
>
> On Wed, Jul 1, 2009 at 4:00 PM, Rob Ross <rross at mcs.anl.gov> wrote:
> Do you mean that 2.8.0 is fast and 2.8.1 is slow? Can you describe  
> the benchmark and how you are doing your measurements?
>
> Rob
>
>
> On Jul 1, 2009, at 4:43 PM, David Bonnie wrote:
>
> Hello all -
>
> I'm having trouble figuring out a problem with performance  
> depredation on a simple 10 node cluster.  Prior runs on the cluster  
> (before this problem manifested itself) resulted in bandwidth and  
> IOPS about 10 times higher on a small file creation workload.  Each  
> node is running as a metadata server and a data server.
>
> The problem is persistent between versions and installations of  
> PVFS2 2.8.0 and 2.8.1.  Rebooting all of the nodes didn't improve  
> anything.  The network connections (simple GigE) showed no errors or  
> dropped packets.  Using different physical disks (both SAS and FC)  
> didn't improve things.  The kernel logs didn't show anything out of  
> place nor did the pvfs2 server or client logs.  It seems like a  
> network issue but I can't seem to find anything wrong with any of  
> the connections.
>
> Has anyone seen this kind of problem before?  I seem to remember  
> something on the list before about performance suddenly dropping but  
> I can't find the message now (of course).  Any insight would be  
> appreciated!
>
> Thanks,
>
> - Dave
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>
>
>
>
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20090701/0ecb725a/attachment.htm