Yes, PCI-X should be a huge bottleneck here, but doesnt explain how you were able to get 400MB/s with the test utilities and not on the fs.<br>You should have a definite and measurable performance increase moving to a pci-e platform of course.
<br><br>Could you let us know if this turns out to be the final solution?<br><br>~Kyle<br><br><div><span class="gmail_quote">On 10/3/07, <b class="gmail_sendername">hyeyoung cho</b> <<a href="mailto:chohy@kisti.re.kr">
chohy@kisti.re.kr</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>Hi Rob.<br>But I already tested with "--without-bmi-tcp" and the flag gave about 10%
<br>improving performance.<br><br>I found my problem.<br>I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz.<br>Maybe I think the PIC-X performance was my bottle neck.<br><br>Thank you for your comments.
<br><br>Regards,<br>Hyeyoung Cho<br><br>-----Original Message-----<br>From: Rob Ross [mailto:<a href="mailto:rross@mcs.anl.gov">rross@mcs.anl.gov</a>]<br>Sent: Thursday, October 04, 2007 12:14 AM<br>To: hyeyoung cho<br>Cc:
<a href="mailto:pvfs2-users@beowulf-underground.org">pvfs2-users@beowulf-underground.org</a><br>Subject: Re: [Pvfs2-users] pvfs performance over IB<br><br>you should disable TCP. -- rob<br><br>hyeyoung cho wrote:<br>> Hello.
<br>><br>><br>><br>> I am working on the performance test of PVFS2 over IB(IBGD-1.8.2).<br>><br>> But the performance was terrible. (70~80MB using pvfs2-cp)<br>><br>> So just wonder anybody has observed the same behavior earlier?
<br>><br>> And anybody can help me with some insight here?<br>><br>><br>><br>> When I was IB performance using perf_main, which is a benchmark tool in<br>> IBGD,<br>><br>> The IB performance was about 400MB.
<br>><br>><br>><br>> my settings<br>><br>> - AMD Opteron Processor 240 - dual CPU<br>><br>> - kernel 2.6.9-55.0.2.ELsmp<br>><br>> - running pvfs2-2.6.3<br>><br>> - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB)
<br>><br>><br>><br>> I compiled pvfs2 with following flags:<br>><br>> ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd<br>><br>> --with-ib-includes=/usr/local/ibgd/driver/infinihost/include
<br>><br>> --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64<br>><br>> --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64<br>><br>><br>><br>><br>><br>> configuration<br>><br>> 1 server, 1 client
<br>><br>><br>><br>> - Client mount command:<br>><br>> mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2<br>><br>><br>><br>> - server pvfs2-fs.conf:<br>><br>> <Aliases><br>>
<br>> Alias c0-10-ib ib://c0-10-ib:3335<br>><br>> </Aliases><br>><br>><br>><br>> - pvfs2-server.conf-c0-10-ib:<br>><br>> StorageSpace /state/pvfs-part/pvfs2-stoarge-space<br>><br>
> HostID "ib://c0-10-ib:3335"<br>><br>> LogFile /tmp/pvfs2-server-ib1server.log<br>><br>><br>><br>> Test pvfs2-cp<br>><br>> [root@compute-0-0 ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5
<br>><br>> Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds<br>><br>> [root@compute-0-0 ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY<br>> /mnt/pvfs2/aa<br>><br>> Wrote 429588480 bytes in
5.449869 seconds. 75.173826 MB/seconds<br>><br>> [root@compute-0-0 ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY<br>> /mnt/pvfs2/bb<br>><br>> Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds
<br>><br>><br>><br>> Test IB performance(usingperf_main)<br>><br>> perf_main --send -trc -mbw -s10240 -n1000<br>><br>> BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000]<br>>
<br>><br>><br>> Regards,<br>><br>> Thank you in advance for valuable comments.<br>><br>> Hyeyoung cho<br>><br>><br>><br>><br>> ------------------------------------------------------------------------
<br>><br>> _______________________________________________<br>> Pvfs2-users mailing list<br>> <a href="mailto:Pvfs2-users@beowulf-underground.org">Pvfs2-users@beowulf-underground.org</a><br>> <a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users">
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users</a><br><br>_______________________________________________<br>Pvfs2-users mailing list<br><a href="mailto:Pvfs2-users@beowulf-underground.org">Pvfs2-users@beowulf-underground.org
</a><br><a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users</a><br></blockquote></div><br><br clear="all"><br>-- <br>Kyle Schochenmaier