[Pvfs2-users] pvfs performance over IB

Wed Oct 10 00:13:20 EDT 2007

Hi,

Have you run iozone or something similar to test the performance of the 
file system on which you're storing your PVFS server's data?

You list pvfs version 2.6.1 below, but you listed 2.6.3 before. I assume 
you're using 2.6.3?

Thanks,

Rob

hyeyoung cho wrote:
>  
> 
> Hi Kyle.
> 
>  
> 
> I tested PVFS performance on IB with moving to a PCI-E(x8) platform.
> 
> I got about 850-950MB network performance.
> 
> (using perf_main in IBGD or using ib_write_bw, ib_send_bw on OFED -
> 
> I tested the PVFS performance on both IBGD and OFDE)
> 
>  
> 
> But the performance was also terrible. (50~60MB using pvfs2-cp with 
> changing buffer size)
> 
>  
> 
> Maybe I got other problems.
> 
> But now I have no clue why that would happen.
> 
> Anybody can give me any clues and help me?
> 
>  
> 
>  
> 
> My test platform
> 
> Hardware
> 
> -         Intel Xeon®Processor 2.80GHz - dual CPU
> 
> -         Memory 1GB
> 
> -         PCI Express (X8)( using Riser)
> 
> -         SCSI Controller : 2 Channel LSI Logic U 320
> 
>  
> 
> Software
> 
> -         OS : 2.6.9-5.EL-smp-i686
> 
> -         Pvfs2 2.6.1
> 
> -         IBGD-1.8.2 or OFED-1.0 ( I tested on both )
> 
>  
> 
>  
> 
>  
> 
> ------------------------------------------------------------------------
> 
> *From:* Kyle Schochenmaier [mailto:kschoche at gmail.com]
> *Sent:* Thursday, October 04, 2007 11:03 AM
> *To:* hyeyoung cho
> *Cc:* Rob Ross; pvfs2-users at beowulf-underground.org
> *Subject:* Re: [Pvfs2-users] pvfs performance over IB
> 
>  
> 
> Yes, PCI-X should be a huge bottleneck here, but doesnt explain how you 
> were able to get 400MB/s with the test utilities and not on the fs.
> You should have a definite and measurable performance increase moving to 
> a pci-e platform of course.
> 
> Could you let us know if this turns out to be the final solution?
> 
> ~Kyle
> 
> On 10/3/07, *hyeyoung cho* < chohy at kisti.re.kr 
> <mailto:chohy at kisti.re.kr>> wrote:
> 
> 
> Hi Rob.
> But I already tested with "--without-bmi-tcp" and the flag gave about 10%
> improving performance.
> 
> I found my problem.
> I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz.
> Maybe I think the PIC-X performance was my bottle neck.
> 
> Thank you for your comments.
> 
> Regards,
> Hyeyoung Cho
> 
> -----Original Message-----
> From: Rob Ross [mailto:rross at mcs.anl.gov <mailto:rross at mcs.anl.gov>]
> Sent: Thursday, October 04, 2007 12:14 AM
> To: hyeyoung cho
> Cc: pvfs2-users at beowulf-underground.org 
> <mailto:pvfs2-users at beowulf-underground.org>
> Subject: Re: [Pvfs2-users] pvfs performance over IB
> 
> you should disable TCP. -- rob
> 
> hyeyoung cho wrote:
>>  Hello.
>>
>>
>>
>>  I am working on the performance test of PVFS2 over IB(IBGD-1.8.2).
>>
>>  But the performance was terrible.  (70~80MB using pvfs2-cp)
>>
>>  So just wonder anybody has observed the same behavior earlier?
>>
>>  And anybody can help me with some insight here?
>>
>>
>>
>>  When I was IB performance using perf_main, which is a benchmark tool in
>>  IBGD,
>>
>>  The IB performance was about 400MB.
>>
>>
>>
>>  my settings
>>
>>  - AMD Opteron Processor 240 - dual CPU
>>
>>  - kernel 2.6.9-55.0.2.ELsmp
>>
>>  - running pvfs2-2.6.3
>>
>>  - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB)
>>
>>
>>
>>  I compiled pvfs2 with following flags:
>>
>>  ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd
>>
>>  --with-ib-includes=/usr/local/ibgd/driver/infinihost/include
>>
>>  --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64
>>
>>  --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64
>>
>>
>>
>>
>>
>>  configuration
>>
>>  1 server, 1 client
>>
>>
>>
>>  - Client mount command:
>>
>>   mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2
>>
>>
>>
>>  - server pvfs2-fs.conf:
>>
>>  <Aliases>
>>
>>         Alias c0-10-ib ib://c0-10-ib:3335
>>
>>  </Aliases>
>>
>>
>>
>>  - pvfs2-server.conf-c0-10-ib:
>>
>>  StorageSpace /state/pvfs-part/pvfs2-stoarge-space
>>
>>  HostID "ib://c0-10-ib:3335"
>>
>>  LogFile /tmp/pvfs2-server-ib1server.log
>>
>>
>>
>>  Test pvfs2-cp
>>
>>   [root at compute-0-0 ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5
>>
>>   Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds
>>
>>   [root at compute-0-0 ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY
>>  /mnt/pvfs2/aa
>>
>>   Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds
>>
>>  [root at compute-0-0 ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY
>>  /mnt/pvfs2/bb
>>
>>  Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds
>>
>>
>>
>>  Test IB performance(usingperf_main)
>>
>>   perf_main --send -trc -mbw -s10240 -n1000
>>
>>  BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000]
>>
>>
>>
>>  Regards,
>>
>>  Thank you in advance for valuable comments.
>>
>>  Hyeyoung cho
>>
>>
>>
>>
>>  ------------------------------------------------------------------------
>>
>>  _______________________________________________
>>  Pvfs2-users mailing list
>>  Pvfs2-users at beowulf-underground.org 
> <mailto:Pvfs2-users at beowulf-underground.org>
>>  http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> 
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org 
> <mailto:Pvfs2-users at beowulf-underground.org>
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> 
> 
> 
> 
> -- 
> Kyle Schochenmaier
> 