[Pvfs2-users] Mvapich2 over PVFS2/IB

[Kyle Schochenmaier]

I've spent the last week or two trying to figure out exactly how to 
build MVAPICH2 with ROMIO *and* PVFS2 *and* native Infiniband support, 
and I finally got it working :-)

**Note: These instructions are for configuring with the openfabrics 
stack, it should be conceptually the same for the other IB stacks.

In order to properly configure mvapich2 to use/support all three 
features you need to do the following:

Modify the make.mvapich2.ofa file to include the necessary Libs and 
Cflags, if you specify CFLAGS or LIBS as part of the command line, it 
will overwrite these options and your build will fail:

1. Add -lpvfs2 to LIBS:
export LIBS=${LIBS:--L${OPEN_IB_LIB} ${BLCR_LIB} ${RDMA_CM_LIBS} -libverbs -libumad -lpthread -lpvfs2}
 
2. Add necessary options to CFLAGS, specifically:  -I{PVFS2-HOME/lib}
export CFLAGS=${CFLAGS:--D${ARCH} ${SMP_FLAG} -DUSE_HEADER_CACHING ${PTMALLOC} -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS
 -D_SHMEM_COLL_ ${RDMA_CM_FLAG} ${CR_FLAG} ${BLCR_INC} -I${OPEN_IB_HOME}/include $OPT_FLAG
 -I{PVFS2-HOME/lib}}

3. Modify the configure line to include --with-file-system=pvfs2

 ./configure  --prefix=${PREFIX} ${MULTI_THREAD} \
   --with-device=osu_ch3:mrail --with-rdma=gen2 --with-pm=mpd \
   ${ROMIO} ${SHARED_LIBS} --with-file-system=pvfs2 --without-mpe 2>&1 |tee config-mine.log


Now you can configure and build everything properly :-)

`ROMIO=yes OPEN_IB_HOME=/usr/local/ LDFLAGS="-m64 -L{PVFS2-HOME/lib}" ./make.mvapich2.ofa`


Initial results have been quite impressive, using mpi-io-test, I was 
able to get 15.04Gbps on a 16Gbps link during one test.

Good luck,

Kyle

-- 
Kyle Schochenmaier
kschoche at scl.ameslab.gov
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory