[Pvfs2-users] error starting server compiled for infiniband (topspin)

[Sam Lang]

Hi Tim,

It looks like ib support isn't getting compiled in to pvfs, even  
though you've specified paths and the server appears to link with the  
vapi lib.  Could you send the config.log and pvfs2-config.h files  
generated from the build?

Thanks,

-sam

On Mar 20, 2007, at 3:59 PM, Carlson, Timothy S wrote:

> I've built and installed pvfs-2.6.2 on x86_64 (RHEL 4) with Cisco/ 
> Topspin cards and software stack.
>
> topspin-ib-rhel4-3.2.0-118
> topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118
>
> ./configure --with-kernel=/usr/src/kernels/2.6.9-42.EL-smp-x86_64 -- 
> with-ib=/usr/local/topspin --with-ib-includes=/usr/local/topspin/ 
> include/vapi
>
> Everything builds fine and I install with
> make install
> make kmod_install
>
> However after configuring the metadata and IO nodes, I try to start  
> the server and get this in the server log.
>
> [D 03/20 13:25] PVFS2 Server version 2.6.2 starting.
> [E 03/20 13:25] src/io/bmi/bmi.c line 1841: Error: no method  
> available for bmi_ib.
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server [0x433a50]
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server 
> (BMI_initialize+0x13e) [0x433f1e]
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server [0x40de0d]
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server(main 
> +0xaa1) [0x40f6a1]
> [E 03/20 13:25]         [bt] /lib64/tls/libc.so.6(__libc_start_main 
> +0xdb) [0x39ce01c3fb]
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server 
> (aio_cancel64+0x16a) [0x40d4aa]
> [E 03/20 13:25] Error: BMI_initialize: Protocol not available
> [E 03/20 13:25] Error: Could not initialize server interfaces;  
> aborting.
> [E 03/20 13:25] Error: Could not initialize server; aborting.
>
> I saw a similar post in the February archives, but that was for  
> Infinipath and I didn't see a final answer to the question that was  
> posted. Any ideas here? The pvfs2-server binary is built against  
> the topspin libraries.
>
> [root at compute-0-0 tmp]# ldd /usr/local/sbin/pvfs2-server
>         libdb-4.2.so => /lib64/tls/libdb-4.2.so (0x00000039cf900000)
>         libpthread.so.0 => /lib64/tls/libpthread.so.0  
> (0x00000039ce900000)
>         librt.so.1 => /lib64/tls/librt.so.1 (0x00000039cf100000)
>         libvapi.so => /usr/local/topspin/lib64/libvapi.so  
> (0x0000002a95573000)
>         libmtl_common.so => /usr/local/topspin/lib64/ 
> libmtl_common.so (0x0000002a95693000)
>         libmosal.so => /usr/local/topspin/lib64/libmosal.so  
> (0x0000002a95795000)
>         libmpga.so => /usr/local/topspin/lib64/libmpga.so  
> (0x0000002a958a4000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00000039ce300000)
>         libc.so.6 => /lib64/tls/libc.so.6 (0x00000039ce000000)
>         /lib64/ld-linux-x86-64.so.2 (0x00000039cde00000)
>         libg2c.so.0 => /usr/lib64/libg2c.so.0 (0x0000002a959a6000)
>         libm.so.6 => /lib64/tls/libm.so.6 (0x00000039ce500000)
>
> The following ib modules are loaded into the running kernel.
>
> [root at compute-0-0 tmp]# lsmod | grep ib
>
> ts_ib_dm_client        31776  1 ts_srp_host
> ts_ib_sa_client        44688  2 ts_srp_host,ts_ib_dm_client
> ts_ib_client_query     23072  3  
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client
> ts_ib_cm               59128  1 ts_srp_host
> ts_kernel_poll         28496  3  
> ts_ib_dm_client,ts_ib_client_query,ts_ib_cm
> ts_ib_useraccess       18728  0
> ts_ib_tavor            34752  3
> mod_vip               461096  3 ts_ib_tavor,mod_rhh,mod_thh
> mlxsys                166380  4  
> ts_ib_useraccess,mod_rhh,mod_thh,mod_vip
> ts_ib_mad              37144  4  
> ts_ib_client_query,ts_ib_cm,ts_ib_useraccess,ts_ib_tavor
> ts_ib_core             65688  7  
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client,ts_ib_cm,ts_ib_useraccess, 
> ts_ib_tavor,ts_ib_mad
>
> ts_ib_packet_lib      212992  3 ts_ib_tavor,ts_ib_mad,ts_ib_core
> ts_kernel_services     25664  11  
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client,ts_ib_client_query,ts_ib_c 
> m,ts_kernel_poll,ts_ib_useraccess,ts_ib_tavor,ts_ib_mad,ts_ib_core,ts_ 
> ib_packet_lib
>
>
> Tim
>
> Tim Carlson
> Voice: (509) 376 3423
> Email: Tim.Carlson at pnl.gov
> Pacific Northwest National Laboratory
> HPCaNS: High Performance Computing and Networking Services
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users