[Pvfs2-users] Problem with Mpich and PVFS2 when using InfiniBand

Matthieu Dorier Matthieu.Dorier at eleves.bretagne.ens-cachan.fr
Thu Mar 24 08:09:15 EST 2011


Here is what I have:

ldd pvfs2-ping
    linux-vdso.so.1 =>  (0x00007fff4e106000)
    libpthread.so.0 => /lib/libpthread.so.0 (0x00007fc94f3dd000)
    libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00007fc94f1d1000)
    libc.so.6 => /lib/libc.so.6 (0x00007fc94ee7e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fc94f5f9000)
    libdl.so.2 => /lib/libdl.so.2 (0x00007fc94ec7a000)

Matthieu

2011/3/24 Michael Moore <mtmoore at omnibond.com>

> Hi Matthieu,
>
> To me it looks like the version of pvfs2 commands you're using doesn't know
> about IB. Can you double check that the version of the pvfs2-ping command
> used below is linked against OpenIB? You can do that with 'ldd <path to
> pvfs2-ping>'
>
> On my system, amongst other libraries, I see a reference to libibverbs.so
> like so:
> "libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002b8ba74c0000)"
>
> Is that the same for you? If not, post what you get and confirm that 'which
> pvfs2-ping' points you to the version of pvfs2 that you built against
> OpenIB.
>
> Thanks,
> Michael
>
>
> On Thu, Mar 24, 2011 at 8:27 AM, Matthieu Dorier <
> Matthieu.Dorier at eleves.bretagne.ens-cachan.fr> wrote:
>
>> Hello,
>>
>> Additional informations regarding the problem: doing a pvfs2-ping doesn't
>> work (neither do all the pvfs2-* commands, such as pvfs2-ls).
>> I've the following error with pvfs2-ping:
>>
>> mdorier at parapluie-31:~$ pvfs2-ping -m /mnt/pvfs2
>>
>> (1) Parsing tab file...
>>
>> (2) Initializing system interface...
>>
>> (3) Initializing each file system found in tab file: /etc/mtab...
>>
>>    PVFS2 servers: ib://parapide-5-ib0:3335
>>    Storage name: pvfs2
>>    Local mount point: /mnt/pvfs2
>> [E 13:01:41.415907] PVFS_isys_fs_add: Failed to initialize any appropriate
>> BMI methods for addresses:
>> [E 13:01:41.415970]     ib://parapluie-32-ib0:3335
>> [E 13:01:41.415992] PVFS_isys_fs_add call: Protocol not available
>>    /mnt/pvfs2: FAILURE!
>>
>> Failure: could not initialze at least one of the target file systems.
>>
>> (4) Searching for /mnt/pvfs2 in pvfstab...
>> [E 13:01:41.416022] Error: /mnt/pvfs2/ resides on a PVFS2 file system that
>> has not yet been initialized.
>> Failure: could not find filesystem for /mnt/pvfs2 in pvfs2tab /etc/mtab
>> Entry 0: /mnt/pvfs2
>>
>> The line in /etc/mtab is the following:
>>
>> ib://parapide-5-ib0:3335/pvfs2 /mnt/pvfs2 pvfs2 defaults,noauto 0 0
>>
>> What I don't understand is why the kernel module works, but not the pvfs2
>> tools or any MPI application?
>> Thank you for your help,
>>
>> Regards
>>
>>
>> Matthieu
>>
>>
>> 2011/3/23 Matthieu Dorier <Matthieu.Dorier at eleves.bretagne.ens-cachan.fr>
>>
>>> Actually when I use the standard names (parapide-X or
>>> parapide-X.rennes.grid5000.fr), the kernel module uses the eth0
>>> interface, so I switched again to using the names ending with "-ib0". I
>>> attach the configuration file.
>>>
>>> Matthieu
>>>
>>>
>>> 2011/3/23 Matthieu Dorier <Matthieu.Dorier at eleves.bretagne.ens-cachan.fr
>>> >
>>>
>>>> Ok actually my nodes are connected to 2 switches: one 1G Eth and one IB,
>>>> so the names "parapide-X-ib0" were to be used for IP over IB (the real
>>>> hostnames being "parapide-X", or even "parapide-X.rennes.grid5000.fr",
>>>> which is the output of the hostname command). I realized it was useless
>>>> working with the names ending in "-ib0" because I'm not using IP over IB in
>>>> this case.
>>>>
>>>> So I deployed again PVFS using the real hostnames, but I still get the
>>>> same errors (and again, the kernel module works fine), the configuration
>>>> line is now:
>>>> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers "parapide-5"
>>>> --metaservers "parapide-5" --storage /tmp/pvfs2-storage-space --logfile
>>>> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf
>>>> And the configuration file is attached.
>>>>
>>>> I also changed the limits.conf as advised by Kevin Harms, by adding the
>>>> following line:
>>>> *                hard    memlock         unlimited
>>>> But it does change the warning (when I'm logged as user the ulimit -l
>>>> unlimited says I don't have the permissions). Also tried
>>>> *                hard    memlock         1073741824
>>>> Doesn't work either.
>>>>
>>>>
>>>> Matthieu
>>>>
>>>>
>>>> 2011/3/22 Becky Ligon <ligon at clemson.edu>
>>>>
>>>>> The kernel module is not required for MPI.
>>>>>
>>>>> Becky
>>>>> --
>>>>> Becky Ligon
>>>>> HPC Admin Staff
>>>>> PVFS Developer
>>>>> Clemson University
>>>>> 864-650-4065
>>>>>
>>>>> > Is make kmod_install also required?
>>>>> >
>>>>> > -crispy
>>>>> >
>>>>> >
>>>>> > On 03/22/2011 03:04 PM, Matthieu Dorier wrote:
>>>>> >> If you mean the command line for pvfs2-genconfig, here it is:
>>>>> >>
>>>>> >> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers
>>>>> >> "parapide-7-ib0"
>>>>> >> --metaservers "parapide-7-ib0" --storage /tmp/pvfs2-storage-space
>>>>> >> --logfile
>>>>> >> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf
>>>>> >>
>>>>> >> I'm basically following the "Quick start guide".
>>>>> >>
>>>>> >> If you mean the configuration before compiling, here it is:
>>>>> >>
>>>>> >> ./configure --prefix=/usr
>>>>> >> --with-kernel=/lib/modules/2.6.26-2-amd64/build
>>>>> >> --with-openib=/usr --without-bmi-tcp
>>>>> >> (then make, make kmod, make install)
>>>>> >>
>>>>> >> Matthieu
>>>>> >>
>>>>> >> 2011/3/22 Becky Ligon<ligon at clemson.edu>
>>>>> >>
>>>>> >>> Can you give us the full command line when you configure PVFS?
>>>>>  Seems
>>>>> >>> that
>>>>> >>> you are doing this in two steps and must be done in one.
>>>>> >>>
>>>>> >>> Becky
>>>>> >>> --
>>>>> >>> Becky Ligon
>>>>> >>> HPC Admin Staff
>>>>> >>> PVFS Developer
>>>>> >>> Clemson University
>>>>> >>> 864-650-4065
>>>>> >>>
>>>>> >>>> Hello,
>>>>> >>>>
>>>>> >>>> I have installed PVFS (version 2.8.2) with InfiniBand support on a
>>>>> set
>>>>> >>>> of
>>>>> >>>> nodes running Debian (linux 2.6). The configuration line being the
>>>>> >>>> following:
>>>>> >>>> ./configure --with-openib=/usr --without-bmi-tcp --prefix=/usr
>>>>> >>>> The filesystem works fine with the kernel module, I'm able to
>>>>> write
>>>>> >>>> and
>>>>> >>>> read
>>>>> >>>> to/from the filesystem.
>>>>> >>>> Then I compiled Mpich2 (1.2.1p1) with PVFS support using
>>>>> >>>> --enable-romio
>>>>> >>>> --with-pvfs2=/usr (among other options which doesn't matter).
>>>>> >>>> Running the IOR benchmark with POSIX, it works fine, but when
>>>>> using
>>>>> >>>> MPIIO
>>>>> >>>> I
>>>>> >>>> got the following errors (this is from process 0, but all the
>>>>> >>>> processes
>>>>> >>>> are
>>>>> >>>> printing the same)
>>>>> >>>>
>>>>> >>>> 0: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>> >>>> 0:     This will severely limit memory registrations.
>>>>> >>>> 0: [E 14:12:05.314874] Warning: openib_ib_initialize:
>>>>> ibv_open_device.
>>>>> >>>> 0: [E 14:12:05.314908] PVFS_isys_fs_add: Failed to initialize any
>>>>> >>>> appropriate BMI methods for addresses:
>>>>> >>>> 0: [E 14:12:05.314921]     ib://
>>>>> parapide-7-ib0.rennes.grid5000.fr:3335
>>>>> >>>> 0: [E 14:12:05.314952] PVFS_isys_fs_add call: No such device
>>>>> >>>> 0: [E 14:12:05.314969] ERROR: could not initialize any file
>>>>> systems in
>>>>> >>>> /etc/mtab.
>>>>> >>>> 0: PVFS_util_init_defaults: No such device (error class: 0)
>>>>> >>>>
>>>>> >>>> Do you have any idea where it comes from? I'm unsuccefully trying
>>>>> to
>>>>> >>>> reinstall OFED-1.5, recompiling PVFS, Mpich, but I always get the
>>>>> same
>>>>> >>>> thing.
>>>>> >>>> Note: previsouly everything worked well with Ethernet, I was able
>>>>> to
>>>>> >>>> have
>>>>> >>>> kernel module AND MPI-IO working.
>>>>> >>>>
>>>>> >>>> Thank you,
>>>>> >>>>
>>>>> >>>> Regards
>>>>> >>>>
>>>>> >>>> Matthieu Dorier
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Matthieu Dorier
>>>>> >>>> ENS Cachan, Brittany (Computer Science dpt.)
>>>>> >>>> IRISA Rennes, Office E324
>>>>> >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>>>>> >>>> _______________________________________________
>>>>> >>>> Pvfs2-users mailing list
>>>>> >>>> Pvfs2-users at beowulf-underground.org
>>>>> >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> Pvfs2-users mailing list
>>>>> >> Pvfs2-users at beowulf-underground.org
>>>>> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Matthieu Dorier
>>>> ENS Cachan, Brittany (Computer Science dpt.)
>>>> IRISA Rennes, Office E324
>>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>>>>
>>>
>>>
>>>
>>> --
>>> Matthieu Dorier
>>> ENS Cachan, Brittany (Computer Science dpt.)
>>> IRISA Rennes, Office E324
>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>>>
>>
>>
>>
>> --
>> Matthieu Dorier
>> ENS Cachan, Brittany (Computer Science dpt.)
>> IRISA Rennes, Office E324
>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>>
>


-- 
Matthieu Dorier
ENS Cachan, Brittany (Computer Science dpt.)
IRISA Rennes, Office E324
http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20110324/069cf2ff/attachment.htm


More information about the Pvfs2-users mailing list