[Pvfs2-developers] Re: memory allocations in BMI

[kyle schochenmaier]

On Dec 25, 2006, at 9:59 AM, Pete Wyckoff wrote:

> kschoche at scl.ameslab.gov wrote on Fri, 22 Dec 2006 15:45 -0600:
>> We stumbled across some hardware issues/limitations with our IB
>> hardware, and have 'devised' a way of working around these
>> limitations by increasing the sizes of memory region allocations
>> inside BMI, currently we're seeing chunks or blocks of 64KB being
>> allocated on the hardware level (this is what the firmware people
>> show us is allocated physically)
>> Is there a way we can tune these allocation calls to the IB drivers,
>> it should increase our ability to make large pinning requests if we
>> can use larger than 64KB, they suggest 2MB.
>>
>> Are you aware of any specific memory region sizes that are allocated
>> in the BMI-IB layers?
>
> Take a look at
>     http://www.beowulf-underground.org/pipermail/pvfs2-developers/ 
> 2006-October/002785.html
> and the surrounding thread.
>
> The patch that's discussed in there did get checked in, so recent
> CVS HEAD will pin the entire buffer once if you try to send the
> whole thing, rather than pinning multiple 64kB chunks of the buffer.
>
> 		-- Pete
>
>
Excellent, thanks Pete!  For some reason i thought that patch was  
only for caching.
I built the latest pvfs-CVS, and am having problems with openib.  I  
originally thought it was a problem with having my client using CVS  
and the server using the latest release, but rebuilt the server to be  
on CVS head, and got this on the client:

p5l8:~# pvfs2-ls
[E 15:02:10.201683] Error: openib_new_connection: asked for 70 send  
WRs on QP, got 0.
[E 15:02:10.201951]     [bt] pvfs2-ls [0x100b659c]
[E 15:02:10.201982]     [bt] pvfs2-ls [0x100b8698]
[E 15:02:10.202018]     [bt] pvfs2-ls [0x100b5098]
[E 15:02:10.202054]     [bt] pvfs2-ls [0x100b558c]
[E 15:02:10.202090]     [bt] pvfs2-ls [0x100b22a8]
[E 15:02:10.202124]     [bt] pvfs2-ls [0x100b2998]
[E 15:02:10.202161]     [bt] pvfs2-ls [0x100b2ff0]
[E 15:02:10.202197]     [bt] pvfs2-ls [0x10063cf8]
[E 15:02:10.202232]     [bt] pvfs2-ls [0x10073d3c]
[E 15:02:10.202268]     [bt] pvfs2-ls [0x1005d00c]
[E 15:02:10.202303]     [bt] pvfs2-ls [0x10012ec0]
p5l8:~# pvfs2-ls --version
2.6.1pre1-2006-12-27-201055

Any thoughts on debugging this? I dont think I have problems with the  
network unless something was sabotaged over the weekend. heh.

+=Kyle
>