[Pvfs2-developers] PVFS2 Performance Problem -

David Bonnie dbonnie at clemson.edu
Wed Jul 1 18:11:01 EDT 2009


Sam -

I'm testing with only 50 files now...initially the tests were run in 1000
file increments from 1000 to 6000 files.  The times scale almost perfectly
linearly at all data points.

- Dave

On Wed, Jul 1, 2009 at 4:08 PM, Sam Lang <slang at mcs.anl.gov> wrote:

>
> On Jul 1, 2009, at 5:05 PM, David Bonnie wrote:
>
> Rob -
>
> Performance is down across all PVFS2 installations.  The benchmark simply
> creates files of a random size (between 1 and 25 MB) in a single folder on
> the mounted PVFS2 partition, 16 KB at a time.  It's not anywhere near ideal,
> but it's the workload I'm working with.
>
>
> How many files?-sam
>
>
> Prior to this problem we were getting ~22 MB/s write throughput and we're
> down to about 2.5 MB/s for no apparent reason.  Reads are down from about 55
> MB/s to 30 MB/s.  No hardware has changed and as far as I can tell no
> hardware has died either.
>
> - Dave
>
>
> On Wed, Jul 1, 2009 at 4:00 PM, Rob Ross <rross at mcs.anl.gov> wrote:
>
>> Do you mean that 2.8.0 is fast and 2.8.1 is slow? Can you describe the
>> benchmark and how you are doing your measurements?
>>
>> Rob
>>
>>
>> On Jul 1, 2009, at 4:43 PM, David Bonnie wrote:
>>
>>  Hello all -
>>>
>>> I'm having trouble figuring out a problem with performance depredation on
>>> a simple 10 node cluster.  Prior runs on the cluster (before this problem
>>> manifested itself) resulted in bandwidth and IOPS about 10 times higher on a
>>> small file creation workload.  Each node is running as a metadata server and
>>> a data server.
>>>
>>> The problem is persistent between versions and installations of PVFS2
>>> 2.8.0 and 2.8.1.  Rebooting all of the nodes didn't improve anything.  The
>>> network connections (simple GigE) showed no errors or dropped packets.
>>>  Using different physical disks (both SAS and FC) didn't improve things.
>>>  The kernel logs didn't show anything out of place nor did the pvfs2 server
>>> or client logs.  It seems like a network issue but I can't seem to find
>>> anything wrong with any of the connections.
>>>
>>> Has anyone seen this kind of problem before?  I seem to remember
>>> something on the list before about performance suddenly dropping but I can't
>>> find the message now (of course).  Any insight would be appreciated!
>>>
>>> Thanks,
>>>
>>> - Dave
>>> _______________________________________________
>>> Pvfs2-developers mailing list
>>> Pvfs2-developers at beowulf-underground.org
>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>>>
>>
>>
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20090701/9ff818f4/attachment.htm


More information about the Pvfs2-developers mailing list