[Pvfs2-developers] PVFS2 Performance Problem -

[David Bonnie]

Rob -

Performance is down across all PVFS2 installations.  The benchmark simply
creates files of a random size (between 1 and 25 MB) in a single folder on
the mounted PVFS2 partition, 16 KB at a time.  It's not anywhere near ideal,
but it's the workload I'm working with.

Prior to this problem we were getting ~22 MB/s write throughput and we're
down to about 2.5 MB/s for no apparent reason.  Reads are down from about 55
MB/s to 30 MB/s.  No hardware has changed and as far as I can tell no
hardware has died either.

- Dave


On Wed, Jul 1, 2009 at 4:00 PM, Rob Ross <rross at mcs.anl.gov> wrote:

> Do you mean that 2.8.0 is fast and 2.8.1 is slow? Can you describe the
> benchmark and how you are doing your measurements?
>
> Rob
>
>
> On Jul 1, 2009, at 4:43 PM, David Bonnie wrote:
>
>  Hello all -
>>
>> I'm having trouble figuring out a problem with performance depredation on
>> a simple 10 node cluster.  Prior runs on the cluster (before this problem
>> manifested itself) resulted in bandwidth and IOPS about 10 times higher on a
>> small file creation workload.  Each node is running as a metadata server and
>> a data server.
>>
>> The problem is persistent between versions and installations of PVFS2
>> 2.8.0 and 2.8.1.  Rebooting all of the nodes didn't improve anything.  The
>> network connections (simple GigE) showed no errors or dropped packets.
>>  Using different physical disks (both SAS and FC) didn't improve things.
>>  The kernel logs didn't show anything out of place nor did the pvfs2 server
>> or client logs.  It seems like a network issue but I can't seem to find
>> anything wrong with any of the connections.
>>
>> Has anyone seen this kind of problem before?  I seem to remember something
>> on the list before about performance suddenly dropping but I can't find the
>> message now (of course).  Any insight would be appreciated!
>>
>> Thanks,
>>
>> - Dave
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20090701/c4f34ae5/attachment.htm