[Pvfs2-users] strange performance: IPoIB outperforms native IB

[xiangyong ouyang]

I setup pvfs-2.6.3 in a cluster with IB support(ofed-1.2) and linux kernel 2.6.20.5. I compiled pvfs2 using the following flags:

./configure --prefix=/tmp --with-db=/tmp/BerkeleyDB --with-openib=/usr/local/ofed --with-openib-li
bs=/usr/local/ofed/lib64 --with-openib-includes=/usr/local/ofed/include --with-mpi=/home/ouyangx/m
pich2-install --with-kernel=/usr/src/linux-2.6.20.5

make; make install


Then I start pvfs2 with IB and IPoIB as transport layer respectively. 4 machines are IO server, 1 machine serves as metadata server. 1 machine is pvfs2-client and mounts up the pvfs2 file system.

After that, I tested pvfs2 performance with "pvfs2-cp -t" to copy a local file to pvfs2 file system. At all different file size, pvfs2/IPoIB yields a better performance than pvfs2/IB. Here are the numbers (MB/sec).

file-sieze	IpoIB		IB
100MB	  	402			300
400MB		405			332
800MB		411			334



I also used MPI-IO(mpich2-1.0) to test pvfs2 performance. Again IPoIB outperforms IB. 

I believe something must be wrong with my configuration. Any comments/suggestions? Thanks!



xiangyong ouyang