[Pvfs2-users] strange performance: IPoIB outperforms native IB
kschoche at gmail.com
Wed Aug 15 16:52:01 EDT 2007
One thing that I've seen in the past is that pvfs2-cp does not often -
especially with IB - use large enough buffers to push the numbers up really
high, by adjusting the buffer settings in pvfs2-cp with `pvfs2 -t -b <4MB+>`
I've seen 1GByte+ bandwidth numbers.
But the mpi-io tests are less easy to fool..
Can you check to see how much cpu usage is going on with the native IB
If something is spinning, or even using more than 50% we should look at
doing some profiling to see what the problem areas are.
On 8/15/07, xiangyong ouyang <ouyang.17 at osu.edu> wrote:
> I setup pvfs-2.6.3 in a cluster with IB support(ofed-1.2) and linux kernel
> 188.8.131.52. I compiled pvfs2 using the following flags:
> ./configure --prefix=/tmp --with-db=/tmp/BerkeleyDB
> --with-openib=/usr/local/ofed --with-openib-li
> bs=/usr/local/ofed/lib64 --with-openib-includes=/usr/local/ofed/include
> pich2-install --with-kernel=/usr/src/linux-184.108.40.206
> make; make install
> Then I start pvfs2 with IB and IPoIB as transport layer respectively. 4
> machines are IO server, 1 machine serves as metadata server. 1 machine is
> pvfs2-client and mounts up the pvfs2 file system.
> After that, I tested pvfs2 performance with "pvfs2-cp -t" to copy a local
> file to pvfs2 file system. At all different file size, pvfs2/IPoIB yields a
> better performance than pvfs2/IB. Here are the numbers (MB/sec).
> file-sieze IpoIB IB
> 100MB 402 300
> 400MB 405 332
> 800MB 411 334
> I also used MPI-IO(mpich2-1.0) to test pvfs2 performance. Again IPoIB
> outperforms IB.
> I believe something must be wrong with my configuration. Any
> comments/suggestions? Thanks!
> xiangyong ouyang
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Pvfs2-users