[Pvfs2-users] error starting server compiled for infiniband (topspin)

[Kyle Schochenmaier]

To my knowledge there has been no pvfs2 support written for the topspin 
library?

The error there is saying the server is confused on how to startup all 
the IB stuff.

I've also seen the following error when the config files are misconfigured.
[E 03/20 13:25] src/io/bmi/bmi.c line 1841: Error: no method available 
for bmi_ib.

I could be wrong though on this, as I'm not familiar with the 
topspin/vapi stack.  I dont know if anyone else has used the topspin 
stack yet here.. It seems to be building..  Maybe pete can shed some 
light on this.

Kyle

Carlson, Timothy S wrote:
>
> I've built and installed pvfs-2.6.2 on x86_64 (RHEL 4) with 
> Cisco/Topspin cards and software stack.
>
> topspin-ib-rhel4-3.2.0-118
> topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118
>
> ./configure --with-kernel=/usr/src/kernels/2.6.9-42.EL-smp-x86_64 
> --with-ib=/usr/local/topspin 
> --with-ib-includes=/usr/local/topspin/include/vapi
>
> Everything builds fine and I install with
> make install
> make kmod_install
>
> However after configuring the metadata and IO nodes, I try to start 
> the server and get this in the server log.
>
> [D 03/20 13:25] PVFS2 Server version 2.6.2 starting.
> [E 03/20 13:25] src/io/bmi/bmi.c line 1841: Error: no method available 
> for bmi_ib.
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server [0x433a50]
> [E 03/20 13:25]         [bt] 
> /usr/local/sbin/pvfs2-server(BMI_initialize+0x13e) [0x433f1e]
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server [0x40de0d]
> [E 03/20 13:25]         [bt] /usr/local/sbin/pvfs2-server(main+0xaa1) 
> [0x40f6a1]
> [E 03/20 13:25]         [bt] 
> /lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x39ce01c3fb]
> [E 03/20 13:25]         [bt] 
> /usr/local/sbin/pvfs2-server(aio_cancel64+0x16a) [0x40d4aa]
> [E 03/20 13:25] Error: BMI_initialize: Protocol not available
> [E 03/20 13:25] Error: Could not initialize server interfaces; aborting.
> [E 03/20 13:25] Error: Could not initialize server; aborting.
>
> I saw a similar post in the February archives, but that was for 
> Infinipath and I didn't see a final answer to the question that was 
> posted. Any ideas here? The pvfs2-server binary is built against the 
> topspin libraries.
>
> [root at compute-0-0 tmp]# ldd /usr/local/sbin/pvfs2-server
>         libdb-4.2.so => /lib64/tls/libdb-4.2.so (0x00000039cf900000)
>         libpthread.so.0 => /lib64/tls/libpthread.so.0 
> (0x00000039ce900000)
>         librt.so.1 => /lib64/tls/librt.so.1 (0x00000039cf100000)
>         libvapi.so => /usr/local/topspin/lib64/libvapi.so 
> (0x0000002a95573000)
>         libmtl_common.so => /usr/local/topspin/lib64/libmtl_common.so 
> (0x0000002a95693000)
>         libmosal.so => /usr/local/topspin/lib64/libmosal.so 
> (0x0000002a95795000)
>         libmpga.so => /usr/local/topspin/lib64/libmpga.so 
> (0x0000002a958a4000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00000039ce300000)
>         libc.so.6 => /lib64/tls/libc.so.6 (0x00000039ce000000)
>         /lib64/ld-linux-x86-64.so.2 (0x00000039cde00000)
>         libg2c.so.0 => /usr/lib64/libg2c.so.0 (0x0000002a959a6000)
>         libm.so.6 => /lib64/tls/libm.so.6 (0x00000039ce500000)
>
> The following ib modules are loaded into the running kernel.
>
> [root at compute-0-0 tmp]# lsmod | grep ib
>
> ts_ib_dm_client        31776  1 ts_srp_host
> ts_ib_sa_client        44688  2 ts_srp_host,ts_ib_dm_client
> ts_ib_client_query     23072  3 
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client
> ts_ib_cm               59128  1 ts_srp_host
> ts_kernel_poll         28496  3 
> ts_ib_dm_client,ts_ib_client_query,ts_ib_cm
> ts_ib_useraccess       18728  0
> ts_ib_tavor            34752  3
> mod_vip               461096  3 ts_ib_tavor,mod_rhh,mod_thh
> mlxsys                166380  4 ts_ib_useraccess,mod_rhh,mod_thh,mod_vip
> ts_ib_mad              37144  4 
> ts_ib_client_query,ts_ib_cm,ts_ib_useraccess,ts_ib_tavor
> ts_ib_core             65688  7 
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client,ts_ib_cm,ts_ib_useraccess,ts_ib_tavor,ts_ib_mad
>
> ts_ib_packet_lib      212992  3 ts_ib_tavor,ts_ib_mad,ts_ib_core
> ts_kernel_services     25664  11 
> ts_srp_host,ts_ib_dm_client,ts_ib_sa_client,ts_ib_client_query,ts_ib_cm,ts_kernel_poll,ts_ib_useraccess,ts_ib_tavor,ts_ib_mad,ts_ib_core,ts_ib_packet_lib
>
>
> Tim
>
> Tim Carlson
> Voice: (509) 376 3423
> Email: Tim.Carlson at pnl.gov
> Pacific Northwest National Laboratory
> HPCaNS: High Performance Computing and Networking Services
>
> !DSPAM:46004c76230776908510162!
> ------------------------------------------------------------------------
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>
>
> !DSPAM:46004c76230776908510162!
>   


-- 
Kyle Schochenmaier
kschoche at scl.ameslab.gov
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory