[Pvfs2-users] strange performance: IPoIB outperforms native IB

[Kyle Schochenmaier]

One thing that I've seen in the past is that pvfs2-cp does not often -
especially with IB - use large enough buffers to push the numbers up really
high, by adjusting the buffer settings in pvfs2-cp with `pvfs2 -t -b <4MB+>`
I've seen 1GByte+ bandwidth numbers.

But the mpi-io tests are less easy to fool..
Can you check to see how much cpu usage is going on with the native IB
interfaces?
If something is spinning, or even using more than 50% we should look at
doing some profiling to see what the problem areas are.

Kyle


On 8/15/07, xiangyong ouyang <ouyang.17 at osu.edu> wrote:
>
> I setup pvfs-2.6.3 in a cluster with IB support(ofed-1.2) and linux kernel
> 2.6.20.5. I compiled pvfs2 using the following flags:
>
> ./configure --prefix=/tmp --with-db=/tmp/BerkeleyDB
> --with-openib=/usr/local/ofed --with-openib-li
> bs=/usr/local/ofed/lib64 --with-openib-includes=/usr/local/ofed/include
> --with-mpi=/home/ouyangx/m
> pich2-install --with-kernel=/usr/src/linux-2.6.20.5
>
> make; make install
>
>
> Then I start pvfs2 with IB and IPoIB as transport layer respectively. 4
> machines are IO server, 1 machine serves as metadata server. 1 machine is
> pvfs2-client and mounts up the pvfs2 file system.
>
> After that, I tested pvfs2 performance with "pvfs2-cp -t" to copy a local
> file to pvfs2 file system. At all different file size, pvfs2/IPoIB yields a
> better performance than pvfs2/IB. Here are the numbers (MB/sec).
>
> file-sieze      IpoIB           IB
> 100MB           402                     300
> 400MB           405                     332
> 800MB           411                     334
>
>
>
> I also used MPI-IO(mpich2-1.0) to test pvfs2 performance. Again IPoIB
> outperforms IB.
>
> I believe something must be wrong with my configuration. Any
> comments/suggestions? Thanks!
>
>
>
> xiangyong ouyang
>
>
>
>
>
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>



-- 
Kyle Schochenmaier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20070815/27311f72/attachment.htm