[Pvfs2-users] pvfs performance over IB

[Rob Ross]

Hi,

Ok, so given those results, it's pretty easy to understand how pvfs2-cp 
obtained the performance that it did.

Thanks,

Rob

hyeyoung cho wrote:
> Hi,
> Kyle and Rob.
> 
> Thank you for your reply.
> I used 2.6.3 at the last test.
> 
> When I tested using iozone, the results was read 145MB and write 100MB(MAX).
> 
> I will send to you test results file.
> 
> Kyle.
> There are no logs.
> After the server is staring, There are no logs.
> 
> [D 10/10 12:41] PVFS2 Server version 2.6.3 starting.
> 
> Thanks,
> Hyeyoung Cho
> 
> -----Original Message-----
> From: Rob Ross [mailto:rross at mcs.anl.gov] 
> Sent: Wednesday, October 10, 2007 1:13 PM
> To: hyeyoung cho
> Cc: 'Kyle Schochenmaier'; pvfs2-users at beowulf-underground.org
> Subject: Re: [Pvfs2-users] pvfs performance over IB
> 
> Hi,
> 
> Have you run iozone or something similar to test the performance of the 
> file system on which you're storing your PVFS server's data?
> 
> You list pvfs version 2.6.1 below, but you listed 2.6.3 before. I assume 
> you're using 2.6.3?
> 
> Thanks,
> 
> Rob
> 
> hyeyoung cho wrote:
>>  
>>> Hi Kyle.
>>  
>>
>> I tested PVFS performance on IB with moving to a PCI-E(x8) platform.
>>
>> I got about 850-950MB network performance.
>>
>> (using perf_main in IBGD or using ib_write_bw, ib_send_bw on OFED -
>>
>> I tested the PVFS performance on both IBGD and OFDE)
>>
>>  
>>
>> But the performance was also terrible. (50~60MB using pvfs2-cp with 
>> changing buffer size)
>>
>>  
>>
>> Maybe I got other problems.
>>
>> But now I have no clue why that would happen.
>>
>> Anybody can give me any clues and help me?
>>
>>  
>>
>>  
>>
>> My test platform
>>
>> Hardware
>>
>> -         Intel XeonRProcessor 2.80GHz - dual CPU
>>
>> -         Memory 1GB
>>
>> -         PCI Express (X8)( using Riser)
>>
>> -         SCSI Controller : 2 Channel LSI Logic U 320
>>
>>
>>
>> Software
>>
>> -         OS : 2.6.9-5.EL-smp-i686
>>
>> -         Pvfs2 2.6.1
>>
>> -         IBGD-1.8.2 or OFED-1.0 ( I tested on both )
>>
>>  
>>
>>  
>>
>>  
>>
>> ------------------------------------------------------------------------
>>
>> *From:* Kyle Schochenmaier [mailto:kschoche at gmail.com]
>> *Sent:* Thursday, October 04, 2007 11:03 AM
>> *To:* hyeyoung cho
>> *Cc:* Rob Ross; pvfs2-users at beowulf-underground.org
>> *Subject:* Re: [Pvfs2-users] pvfs performance over IB
>>
>>  
>>
>> Yes, PCI-X should be a huge bottleneck here, but doesnt explain how you 
>> were able to get 400MB/s with the test utilities and not on the fs.
>> You should have a definite and measurable performance increase moving to 
>> a pci-e platform of course.
>>
>> Could you let us know if this turns out to be the final solution?
>>
>> ~Kyle
>>
>> On 10/3/07, *hyeyoung cho* < chohy at kisti.re.kr 
>> <mailto:chohy at kisti.re.kr>> wrote:
>>
>>
>> Hi Rob.
>> But I already tested with "--without-bmi-tcp" and the flag gave about 10%
>> improving performance.
>>
>> I found my problem.
>> I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz.
>> Maybe I think the PIC-X performance was my bottle neck.
>>
>> Thank you for your comments.
>>
>> Regards,
>> Hyeyoung Cho
>>
>> -----Original Message-----
>> From: Rob Ross [mailto:rross at mcs.anl.gov <mailto:rross at mcs.anl.gov>]
>> Sent: Thursday, October 04, 2007 12:14 AM
>> To: hyeyoung cho
>> Cc: pvfs2-users at beowulf-underground.org 
>> <mailto:pvfs2-users at beowulf-underground.org>
>> Subject: Re: [Pvfs2-users] pvfs performance over IB
>>
>> you should disable TCP. -- rob
>>
>> hyeyoung cho wrote:
>>>  Hello.
>>>
>>>
>>>
>>>  I am working on the performance test of PVFS2 over IB(IBGD-1.8.2).
>>>
>>>  But the performance was terrible.  (70~80MB using pvfs2-cp)
>>>
>>>  So just wonder anybody has observed the same behavior earlier?
>>>
>>>  And anybody can help me with some insight here?
>>>
>>>
>>>
>>>  When I was IB performance using perf_main, which is a benchmark tool in
>>>  IBGD,
>>>
>>>  The IB performance was about 400MB.
>>>
>>>
>>>
>>>  my settings
>>>
>>>  - AMD Opteron Processor 240 - dual CPU
>>>
>>>  - kernel 2.6.9-55.0.2.ELsmp
>>>
>>>  - running pvfs2-2.6.3
>>>
>>>  - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB)
>>>
>>>
>>>
>>>  I compiled pvfs2 with following flags:
>>>
>>>  ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd
>>>
>>>  --with-ib-includes=/usr/local/ibgd/driver/infinihost/include
>>>
>>>  --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64
>>>
>>>  --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64
>>>
>>>
>>>
>>>
>>>
>>>  configuration
>>>
>>>  1 server, 1 client
>>>
>>>
>>>
>>>  - Client mount command:
>>>
>>>   mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2
>>>
>>>
>>>
>>>  - server pvfs2-fs.conf:
>>>
>>>  <Aliases>
>>>
>>>         Alias c0-10-ib ib://c0-10-ib:3335
>>>
>>>  </Aliases>
>>>
>>>
>>>
>>>  - pvfs2-server.conf-c0-10-ib:
>>>
>>>  StorageSpace /state/pvfs-part/pvfs2-stoarge-space
>>>
>>>  HostID "ib://c0-10-ib:3335"
>>>
>>>  LogFile /tmp/pvfs2-server-ib1server.log
>>>
>>>
>>>
>>>  Test pvfs2-cp
>>>
>>>   [root at compute-0-0 ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5
>>>
>>>   Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds
>>>
>>>   [root at compute-0-0 ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY
>>>  /mnt/pvfs2/aa
>>>
>>>   Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds
>>>
>>>  [root at compute-0-0 ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY
>>>  /mnt/pvfs2/bb
>>>
>>>  Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds
>>>
>>>
>>>
>>>  Test IB performance(usingperf_main)
>>>
>>>   perf_main --send -trc -mbw -s10240 -n1000
>>>
>>>  BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000]
>>>
>>>
>>>
>>>  Regards,
>>>
>>>  Thank you in advance for valuable comments.
>>>
>>>  Hyeyoung cho
>>>
>>>
>>>
>>>
>>>  ------------------------------------------------------------------------
>>>
>>>  _______________________________________________
>>>  Pvfs2-users mailing list
>>>  Pvfs2-users at beowulf-underground.org 
>> <mailto:Pvfs2-users at beowulf-underground.org>
>>>  http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org 
>> <mailto:Pvfs2-users at beowulf-underground.org>
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>>
>>
>>
>> -- 
>> Kyle Schochenmaier
>>
>