[Pvfs2-users] error starting server compiled for infiniband
(topspin)
Carlson, Timothy S
Timothy.Carlson at pnl.gov
Tue Mar 20 16:42:08 EST 2007
Attached.
Thanks!
Tim
-----Original Message-----
From: Sam Lang [mailto:slang at mcs.anl.gov]
Sent: Tuesday, March 20, 2007 2:35 PM
To: Carlson, Timothy S
Cc: pvfs2-users at beowulf-underground.org
Subject: Re: [Pvfs2-users] error starting server compiled for infiniband
(topspin)
Hi Tim,
It looks like ib support isn't getting compiled in to pvfs, even though
you've specified paths and the server appears to link with the vapi lib.
Could you send the config.log and pvfs2-config.h files generated from
the build?
Thanks,
-sam
On Mar 20, 2007, at 3:59 PM, Carlson, Timothy S wrote:
> I've built and installed pvfs-2.6.2 on x86_64 (RHEL 4) with Cisco/
> Topspin cards and software stack.
>
> topspin-ib-rhel4-3.2.0-118
> topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118
>
> ./configure --with-kernel=/usr/src/kernels/2.6.9-42.EL-smp-x86_64 --
> with-ib=/usr/local/topspin --with-ib-includes=/usr/local/topspin/
> include/vapi
>
> Everything builds fine and I install with make install make
> kmod_install
>
> However after configuring the metadata and IO nodes, I try to start
> the server and get this in the server log.
>
> [D 03/20 13:25] PVFS2 Server version 2.6.2 starting.
> [E 03/20 13:25] src/io/bmi/bmi.c line 1841: Error: no method available
> for bmi_ib.
> [E 03/20 13:25] [bt] /usr/local/sbin/pvfs2-server [0x433a50]
> [E 03/20 13:25] [bt] /usr/local/sbin/pvfs2-server
> (BMI_initialize+0x13e) [0x433f1e]
> [E 03/20 13:25] [bt] /usr/local/sbin/pvfs2-server [0x40de0d]
> [E 03/20 13:25] [bt] /usr/local/sbin/pvfs2-server(main
> +0xaa1) [0x40f6a1]
> [E 03/20 13:25] [bt] /lib64/tls/libc.so.6(__libc_start_main
> +0xdb) [0x39ce01c3fb]
> [E 03/20 13:25] [bt] /usr/local/sbin/pvfs2-server
> (aio_cancel64+0x16a) [0x40d4aa]
> [E 03/20 13:25] Error: BMI_initialize: Protocol not available [E 03/20
> 13:25] Error: Could not initialize server interfaces; aborting.
> [E 03/20 13:25] Error: Could not initialize server; aborting.
>
> I saw a similar post in the February archives, but that was for
> Infinipath and I didn't see a final answer to the question that was
> posted. Any ideas here? The pvfs2-server binary is built against the
> topspin libraries.
>
> [root at compute-0-0 tmp]# ldd /usr/local/sbin/pvfs2-server
> libdb-4.2.so => /lib64/tls/libdb-4.2.so (0x00000039cf900000)
> libpthread.so.0 => /lib64/tls/libpthread.so.0
> (0x00000039ce900000)
> librt.so.1 => /lib64/tls/librt.so.1 (0x00000039cf100000)
> libvapi.so => /usr/local/topspin/lib64/libvapi.so
> (0x0000002a95573000)
> libmtl_common.so => /usr/local/topspin/lib64/ libmtl_common.so
> (0x0000002a95693000)
> libmosal.so => /usr/local/topspin/lib64/libmosal.so
> (0x0000002a95795000)
> libmpga.so => /usr/local/topspin/lib64/libmpga.so
> (0x0000002a958a4000)
> libdl.so.2 => /lib64/libdl.so.2 (0x00000039ce300000)
> libc.so.6 => /lib64/tls/libc.so.6 (0x00000039ce000000)
> /lib64/ld-linux-x86-64.so.2 (0x00000039cde00000)
> libg2c.so.0 => /usr/lib64/libg2c.so.0 (0x0000002a959a6000)
> libm.so.6 => /lib64/tls/libm.so.6 (0x00000039ce500000)
>
> The following ib modules are loaded into the running kernel.
>
> [root at compute-0-0 tmp]# lsmod | grep ib
>
> ts_ib_dm_client 31776 1 ts_srp_host
> ts_ib_sa_client 44688 2 ts_srp_host,ts_ib_dm_client
> ts_ib_client_query 23072 3
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client
> ts_ib_cm 59128 1 ts_srp_host
> ts_kernel_poll 28496 3
> ts_ib_dm_client,ts_ib_client_query,ts_ib_cm
> ts_ib_useraccess 18728 0
> ts_ib_tavor 34752 3
> mod_vip 461096 3 ts_ib_tavor,mod_rhh,mod_thh
> mlxsys 166380 4
> ts_ib_useraccess,mod_rhh,mod_thh,mod_vip
> ts_ib_mad 37144 4
> ts_ib_client_query,ts_ib_cm,ts_ib_useraccess,ts_ib_tavor
> ts_ib_core 65688 7
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client,ts_ib_cm,ts_ib_useraccess,
> ts_ib_tavor,ts_ib_mad
>
> ts_ib_packet_lib 212992 3 ts_ib_tavor,ts_ib_mad,ts_ib_core
> ts_kernel_services 25664 11
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client,ts_ib_client_query,ts_ib_c
> m,ts_kernel_poll,ts_ib_useraccess,ts_ib_tavor,ts_ib_mad,ts_ib_core,ts_
> ib_packet_lib
>
>
> Tim
>
> Tim Carlson
> Voice: (509) 376 3423
> Email: Tim.Carlson at pnl.gov
> Pacific Northwest National Laboratory
> HPCaNS: High Performance Computing and Networking Services
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2-config.h
Type: application/octet-stream
Size: 10388 bytes
Desc: pvfs2-config.h
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20070320/187599c2/pvfs2-config-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: application/octet-stream
Size: 140227 bytes
Desc: config.log
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20070320/187599c2/config-0001.obj
More information about the Pvfs2-users
mailing list