<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="date"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="time"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Batang;
panose-1:2 3 6 0 0 1 1 1 1 1;}
@font-face
{font-family:Gulim;
panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
{font-family:GulimChe;
panose-1:2 11 6 9 0 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Gulim;
panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
{font-family:GulimChe;
panose-1:2 11 6 9 0 1 1 1 1 1;}
@font-face
{font-family:Batang;
panose-1:2 3 6 0 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
pre
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:GulimChe;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Gulim;
color:navy;}
@page Section1
{size:595.3pt 841.9pt;
margin:99.25pt 3.0cm 3.0cm 3.0cm;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1413283921;
mso-list-type:hybrid;
mso-list-template-ids:-996096418 -1741389686 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:2;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:38.0pt;
mso-level-number-position:left;
margin-left:38.0pt;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Gulim;
mso-bidi-font-family:Arial;
color:navy;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=KO link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>Hi Kyle.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>I tested PVFS performance on IB with
moving to a PCI-E(x8) platform.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>I got about 850-950MB network
performance. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>(using perf_main in IBGD or using
ib_write_bw, ib_send_bw on OFED - <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>I tested the PVFS performance on
both IBGD and OFDE)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>But </span></font><font size=2
face=굴림><span lang=EN-US style='font-size:10.0pt;font-family:
Gulim'>the performance was also terrible. (50~60MB using pvfs2-cp with
changing buffer size)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>Maybe I got other problems.<o:p></o:p></span></font></p>
<pre><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim'>But now I have no clue why that would happen.<o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>Anybody can give me any clues and
help me? <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>My test platform<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>Hardware<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:38.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 color=navy face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim;color:navy'><span style='mso-list:Ignore'>-<font size=1
face="Times New Roman"><span style='font:7.0pt "Times New Roman"'>
</span></font></span></span></font><![endif]><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'>Intel Xeon®Processor 2.80GHz
- dual CPU<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:38.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 color=navy face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim;color:navy'><span style='mso-list:Ignore'>-<font size=1
face="Times New Roman"><span style='font:7.0pt "Times New Roman"'>
</span></font></span></span></font><![endif]><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'>Memory 1GB</span></font><font
size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim'><o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:38.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 color=navy face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim;color:navy'><span style='mso-list:Ignore'>-<font size=1
face="Times New Roman"><span style='font:7.0pt "Times New Roman"'>
</span></font></span></span></font><![endif]><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'>PCI Express (X8)( using Riser)</span></font><font
size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim'><o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:38.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 color=navy face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim;color:navy'><span style='mso-list:Ignore'>-<font size=1
face="Times New Roman"><span style='font:7.0pt "Times New Roman"'>
</span></font></span></span></font><![endif]><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'>SCSI Controller : 2
Channel LSI Logic U 320</span></font><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'><o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:20.0pt'><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US
style='font-size:10.0pt;font-family:Gulim'>Software<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:38.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 color=navy face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim;color:navy'><span style='mso-list:Ignore'>-<font size=1
face="Times New Roman"><span style='font:7.0pt "Times New Roman"'>
</span></font></span></span></font><![endif]><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'>OS : 2.6.9-5.EL-smp-i686<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:38.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 color=navy face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim;color:navy'><span style='mso-list:Ignore'>-<font size=1
face="Times New Roman"><span style='font:7.0pt "Times New Roman"'>
</span></font></span></span></font><![endif]><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'>Pvfs2 2.6.1<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:38.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 color=navy face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:Gulim;color:navy'><span style='mso-list:Ignore'>-<font size=1
face="Times New Roman"><span style='font:7.0pt "Times New Roman"'>
</span></font></span></span></font><![endif]><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'>IBGD-1.8.2 or OFED-1.0 (
I tested on both )<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:20.0pt'><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='margin-left:20.0pt'><font size=2 face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=굴림><span
lang=EN-US style='font-size:10.0pt;font-family:Gulim;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span lang=EN-US
style='font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</span></font></b><font
size=2 face=Tahoma><span lang=EN-US style='font-size:10.0pt;font-family:Tahoma'>
Kyle Schochenmaier [mailto:kschoche@gmail.com] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Thursday, October 04, 2007 <st1:time
o:ls="trans" Hour="11" Minute="3" w:st="on">11:03 AM</st1:time><br>
<b><span style='font-weight:bold'>To:</span></b> hyeyoung cho<br>
<b><span style='font-weight:bold'>Cc:</span></b> Rob Ross;
pvfs2-users@beowulf-underground.org<br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Pvfs2-users] pvfs
performance over IB</span></font><span lang=EN-US><o:p></o:p></span></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span lang=EN-US
style='font-size:12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt'>Yes, PCI-X
should be a huge bottleneck here, but doesnt explain how you were able to get
400MB/s with the test utilities and not on the fs.<br>
You should have a definite and measurable performance increase moving to a
pci-e platform of course. <br>
<br>
Could you let us know if this turns out to be the final solution?<br>
<br>
~Kyle<o:p></o:p></span></font></p>
<div>
<p class=MsoNormal><span class=gmailquote><font size=3 face="Times New Roman"><span
lang=EN-US style='font-size:12.0pt'>On <st1:date o:ls="trans" Year="2010"
Month="3" Day="7" w:st="on">10/3/07</st1:date>, <b><span style='font-weight:
bold'>hyeyoung cho</span></b> <<a href="mailto:chohy@kisti.re.kr">
chohy@kisti.re.kr</a>> wrote:</span></font></span><span lang=EN-US><o:p></o:p></span></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span lang=EN-US
style='font-size:12.0pt'><br>
Hi Rob.<br>
But I already tested with "--without-bmi-tcp" and the flag gave about
10% <br>
improving performance.<br>
<br>
I found my problem.<br>
I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz.<br>
Maybe I think the PIC-X performance was my bottle neck.<br>
<br>
Thank you for your comments. <br>
<br>
Regards,<br>
Hyeyoung Cho<br>
<br>
-----Original Message-----<br>
From: Rob Ross [mailto:<a href="mailto:rross@mcs.anl.gov">rross@mcs.anl.gov</a>]<br>
Sent: Thursday, October 04, 2007 <st1:time o:ls="trans" Hour="12" Minute="14"
w:st="on">12:14 AM</st1:time><br>
To: hyeyoung cho<br>
Cc: <a href="mailto:pvfs2-users@beowulf-underground.org">pvfs2-users@beowulf-underground.org</a><br>
Subject: Re: [Pvfs2-users] pvfs performance over IB<br>
<br>
you should disable TCP. -- rob<br>
<br>
hyeyoung cho wrote:<br>
> Hello. <br>
><br>
><br>
><br>
> I am working on the performance test of PVFS2 over IB(IBGD-1.8.2).<br>
><br>
> But the performance was terrible. (70~80MB using pvfs2-cp)<br>
><br>
> So just wonder anybody has observed the same behavior earlier? <br>
><br>
> And anybody can help me with some insight here?<br>
><br>
><br>
><br>
> When I was IB performance using perf_main, which is a benchmark tool in<br>
> IBGD,<br>
><br>
> The IB performance was about 400MB. <br>
><br>
><br>
><br>
> my settings<br>
><br>
> - AMD Opteron Processor 240 - dual CPU<br>
><br>
> - kernel 2.6.9-55.0.2.ELsmp<br>
><br>
> - running pvfs2-2.6.3<br>
><br>
> - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB) <br>
><br>
><br>
><br>
> I compiled pvfs2 with following flags:<br>
><br>
> ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd<br>
><br>
> --with-ib-includes=/usr/local/ibgd/driver/infinihost/include <br>
><br>
> --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64<br>
><br>
> --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64<br>
><br>
><br>
><br>
><br>
><br>
> configuration<br>
><br>
> 1 server, 1 client <br>
><br>
><br>
><br>
> - Client mount command:<br>
><br>
> mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2<br>
><br>
><br>
><br>
> - server pvfs2-fs.conf:<br>
><br>
> <Aliases><br>
> <br>
> Alias c0-10-ib
ib://c0-10-ib:3335<br>
><br>
> </Aliases><br>
><br>
><br>
><br>
> - pvfs2-server.conf-c0-10-ib:<br>
><br>
> StorageSpace /state/pvfs-part/pvfs2-stoarge-space<br>
><br>
> HostID "ib://c0-10-ib:3335"<br>
><br>
> LogFile /tmp/pvfs2-server-ib1server.log<br>
><br>
><br>
><br>
> Test pvfs2-cp<br>
><br>
> [root@compute-0-0 ~]# pvfs2-cp -t /state/partition1/testHY
/mnt/pvfs2/5 <br>
><br>
> Wrote 429588480 bytes in 4.854962 seconds. 84.385318
MB/seconds<br>
><br>
> [root@compute-0-0 ~]# pvfs2-cp -t -b 8388608
/state/partition1/testHY<br>
> /mnt/pvfs2/aa<br>
><br>
> Wrote 429588480 bytes in 5.449869 seconds. 75.173826
MB/seconds<br>
><br>
> [root@compute-0-0 ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY<br>
> /mnt/pvfs2/bb<br>
><br>
> Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds <br>
><br>
><br>
><br>
> Test IB performance(usingperf_main)<br>
><br>
> perf_main --send -trc -mbw -s10240 -n1000<br>
><br>
> BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000]<br>
> <br>
><br>
><br>
> Regards,<br>
><br>
> Thank you in advance for valuable comments.<br>
><br>
> Hyeyoung cho<br>
><br>
><br>
><br>
><br>
> ------------------------------------------------------------------------ <br>
><br>
> _______________________________________________<br>
> Pvfs2-users mailing list<br>
> <a href="mailto:Pvfs2-users@beowulf-underground.org">Pvfs2-users@beowulf-underground.org</a><br>
> <a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users</a><br>
<br>
_______________________________________________<br>
Pvfs2-users mailing list<br>
<a href="mailto:Pvfs2-users@beowulf-underground.org">Pvfs2-users@beowulf-underground.org
</a><br>
<a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users</a><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span lang=EN-US
style='font-size:12.0pt'><br>
<br clear=all>
<br>
-- <br>
Kyle Schochenmaier <o:p></o:p></span></font></p>
</div>
<img src="http://mail.kisti.re.kr/cgi-bin/setconfirm.cgi?k=11919777804754615263&u=chohy&d=kisti.re.kr" width=1 height=1>
</body>
</html>