[Pvfs2-users] Problem with Mpich and PVFS2 when using InfiniBand

Becky Ligon ligon at clemson.edu
Tue Mar 22 14:24:01 EST 2011


Can you also send me the pvfs2-fs.conf file that was created?  And, give
me the output of the command "hostname".

Becky
-- 
Becky Ligon
HPC Admin Staff
PVFS Developer
Clemson University
864-650-4065

> If you mean the command line for pvfs2-genconfig, here it is:
>
> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers
> "parapide-7-ib0"
> --metaservers "parapide-7-ib0" --storage /tmp/pvfs2-storage-space
> --logfile
> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf
>
> I'm basically following the "Quick start guide".
>
> If you mean the configuration before compiling, here it is:
>
> ./configure --prefix=/usr --with-kernel=/lib/modules/2.6.26-2-amd64/build
> --with-openib=/usr --without-bmi-tcp
> (then make, make kmod, make install)
>
> Matthieu
>
> 2011/3/22 Becky Ligon <ligon at clemson.edu>
>
>> Can you give us the full command line when you configure PVFS?  Seems
>> that
>> you are doing this in two steps and must be done in one.
>>
>> Becky
>> --
>> Becky Ligon
>> HPC Admin Staff
>> PVFS Developer
>> Clemson University
>> 864-650-4065
>>
>> > Hello,
>> >
>> > I have installed PVFS (version 2.8.2) with InfiniBand support on a set
>> of
>> > nodes running Debian (linux 2.6). The configuration line being the
>> > following:
>> > ./configure --with-openib=/usr --without-bmi-tcp --prefix=/usr
>> > The filesystem works fine with the kernel module, I'm able to write
>> and
>> > read
>> > to/from the filesystem.
>> > Then I compiled Mpich2 (1.2.1p1) with PVFS support using
>> --enable-romio
>> > --with-pvfs2=/usr (among other options which doesn't matter).
>> > Running the IOR benchmark with POSIX, it works fine, but when using
>> MPIIO
>> > I
>> > got the following errors (this is from process 0, but all the
>> processes
>> > are
>> > printing the same)
>> >
>> > 0: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>> > 0:     This will severely limit memory registrations.
>> > 0: [E 14:12:05.314874] Warning: openib_ib_initialize: ibv_open_device.
>> > 0: [E 14:12:05.314908] PVFS_isys_fs_add: Failed to initialize any
>> > appropriate BMI methods for addresses:
>> > 0: [E 14:12:05.314921]     ib://parapide-7-ib0.rennes.grid5000.fr:3335
>> > 0: [E 14:12:05.314952] PVFS_isys_fs_add call: No such device
>> > 0: [E 14:12:05.314969] ERROR: could not initialize any file systems in
>> > /etc/mtab.
>> > 0: PVFS_util_init_defaults: No such device (error class: 0)
>> >
>> > Do you have any idea where it comes from? I'm unsuccefully trying to
>> > reinstall OFED-1.5, recompiling PVFS, Mpich, but I always get the same
>> > thing.
>> > Note: previsouly everything worked well with Ethernet, I was able to
>> have
>> > kernel module AND MPI-IO working.
>> >
>> > Thank you,
>> >
>> > Regards
>> >
>> > Matthieu Dorier
>> >
>> >
>> >
>> > --
>> > Matthieu Dorier
>> > ENS Cachan, Brittany (Computer Science dpt.)
>> > IRISA Rennes, Office E324
>> > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>> > _______________________________________________
>> > Pvfs2-users mailing list
>> > Pvfs2-users at beowulf-underground.org
>> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> >
>>
>>
>>
>
>
> --
> Matthieu Dorier
> ENS Cachan, Brittany (Computer Science dpt.)
> IRISA Rennes, Office E324
> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>




More information about the Pvfs2-users mailing list