[Pvfs2-users] pvfs performance over IB

[hyeyoung cho]

Hi Rob. 
But I already tested with "--without-bmi-tcp" and the flag gave about 10%
improving performance.  

I found my problem. 
I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz.
Maybe I think the PIC-X performance was my bottle neck.

Thank you for your comments.

Regards,
Hyeyoung Cho

-----Original Message-----
From: Rob Ross [mailto:rross at mcs.anl.gov] 
Sent: Thursday, October 04, 2007 12:14 AM
To: hyeyoung cho
Cc: pvfs2-users at beowulf-underground.org
Subject: Re: [Pvfs2-users] pvfs performance over IB

you should disable TCP. -- rob

hyeyoung cho wrote:
> Hello.
> 
>  
> 
> I am working on the performance test of PVFS2 over IB(IBGD-1.8.2).
> 
> But the performance was terrible.  (70~80MB using pvfs2-cp)
> 
> So just wonder anybody has observed the same behavior earlier?
> 
> And anybody can help me with some insight here?
> 
>  
> 
> When I was IB performance using perf_main, which is a benchmark tool in 
> IBGD,
> 
> The IB performance was about 400MB.
> 
>  
> 
> my settings
> 
> - AMD Opteron Processor 240 - dual CPU  
> 
> - kernel 2.6.9-55.0.2.ELsmp
> 
> - running pvfs2-2.6.3
> 
> - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB)
> 
>  
> 
> I compiled pvfs2 with following flags:
> 
> ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd
> 
> --with-ib-includes=/usr/local/ibgd/driver/infinihost/include
> 
> --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64
> 
> --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64
> 
>  
> 
>  
> 
> configuration
> 
> 1 server, 1 client
> 
>  
> 
> - Client mount command:
> 
>   mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2
> 
>   
> 
> - server pvfs2-fs.conf:
> 
> <Aliases>
> 
>         Alias c0-10-ib ib://c0-10-ib:3335
> 
> </Aliases>
> 
>   
> 
> - pvfs2-server.conf-c0-10-ib:
> 
> StorageSpace /state/pvfs-part/pvfs2-stoarge-space
> 
> HostID "ib://c0-10-ib:3335"
> 
> LogFile /tmp/pvfs2-server-ib1server.log
> 
>  
> 
> Test pvfs2-cp
> 
>   [root at compute-0-0 ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5
> 
>   Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds
> 
>   [root at compute-0-0 ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY 
> /mnt/pvfs2/aa
> 
>   Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds
> 
> [root at compute-0-0 ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY 
> /mnt/pvfs2/bb
> 
> Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds
> 
>  
> 
> Test IB performance(usingperf_main)
> 
>   perf_main --send -trc -mbw -s10240 -n1000
> 
> BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000]
> 
>  
> 
> Regards,
> 
> Thank you in advance for valuable comments.
> 
> Hyeyoung cho
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users