[Pvfs2-developers] Re: memory allocations in BMI

[Kyle Schochenmaier]

Pete Wyckoff wrote:
> kschoche at scl.ameslab.gov wrote on Fri, 29 Dec 2006 12:57 -0600:
>   
>> Just a sanity check for everyone, I rebuilt release 2.5.1, and 
>> everything works correctly.
>> I looked back at the 2.5.1 code, and noticed that we were just 
>> hardcoding the max_wr value to 512, which is much less than what I'm 
>> seeing here in 2.6.1 - (70).
>> I also verified that we have a 32768 max_wr value from 2.5.1's 
>> ibv_device_query() function.
>> However, 2.6.1 is reporting 0?
>>     
>
>   
Printed out hca_cap->max_qp_wr immediately after querying the device 
ibv_query_device(), and got 32k, which is what I got w/ 2.5.1.
This is in the ib_init() function.
Then we goto:
od->nic_max_wr = hca_cap->max_qp_wr   (sorry, summarising the code, 
these are lines from my head)

Then in new_connection()
we do a check:
max_nr > od->nic_max_wr
at which point od->nic_max_wr is 0,
I havent searched through the flow of the code to see where its getting 
overwritten yet, but will try this afternoon.

I think that is where we're getting off, havent applied the patches yet, 
but, again I'll get to that this afternoon.

thx
Kyle
> 512 is greater that 70.  We're only asking for 70 now, so fewer WRs
> requested should be "easier" for the driver.
>
> Can you step through the device setup code and make sure that all
> the numbers are reasonable?  In particular, print out "att" both
> before and after the call to ibv_create_qp.
>
> The main difference from the old pvfs version is that now we only
> need one QP, not two.  Not sure how this could cause the sort of
> regression you are seeing (yet).
>
> 		-- Pete
>
>   
> !DSPAM:4595716f197131497551287!
>
>   


-- 
Kyle Schochenmaier
kschoche at scl.ameslab.gov
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory