[Pvfs2-users] PVFS2 over infiniband: pvfs2-client-core uses up 100% CPU

[Pete Wyckoff]

pal at di.fct.unl.pt wrote on Mon, 28 Aug 2006 17:24 +0100:
> I'm going to raise a different, although slightly related issue; please
> advise me if it should be sent off-list or create another "thread".
> 
> I am also using Topspin IB PCI-X cards and TS120 switch, with Scientific
> Linux EL4.2 (2.6.9-11.ELsmp) on Opteron x86_64 - HP DL145G2 nodes -
> (Vikrant is SLES9 a typo? is it Scientific Linux EL?).
> 
> IPoIB is running fine (I get around 680MB/s from netperf at 70% CPU usage)
> but as sson as I try to run anything that uses IB SDP (I think it uses
> RDMA), such as tests in topspin/bin/ or topspin/sbin/hca_stress, the
> machines where the test is run crash with an IB module oops

Personally I've been afraid to run SDP.  It has been fraught with
problems in the past.

> So I am running PVFS but on the Gb boards; I don't want to use PVFS-IB
> while the IB stack is "unreliable"...

Note that PVFS-IB does not use IPoIB.  It does not use SDP.  It uses
native IB calls, much like an MPI implementation or native
InfiniBand program would use it.  Thus I wouldn't be too worried
about stability problems.  The kernel module part is completely
optional, so the worst that would happen if you tried PVFS-IB is
that your userspace program might crash.

> I have been advised to replace the Topspin/Cisco with the openIB stack,
> but I had yet no window to halt the nodes...

That seems to be the trend.  We use both Mellanox IBGD-1.8.2 and
OpenIB here locally, for testing, and they work fine.  The
programming interface for OpenIB is much nicer to work with though.

		-- Pete