[Pvfs2-developers] pvfs2-cp performance with single client and single server

[Kyle Schochenmaier]

Pete Wyckoff wrote:
> atchley at myri.com wrote on Fri, 29 Dec 2006 13:35 -0500:
>   
>> Do these require the kernel module? I have not tried using that yet.
>>     
>
> Nope, but you need to figure out MPI.
>
> There are some BMI tests in test/io/bmi that can be used just to
> look at that layer.  They do work, but may need some tweaking for
> your new device.  I haven't taken numbers with them for a long time
> though.
>
>   
>> I do not see any improvement using larger than 1 MB with  
>> FlowBufferSizeBytes.
>>     
>
> Odd.  I figured most of the overhead here was server request
> processing and that you would also see an improvement.
>
>   
We've seen similar lack of improvement above 1MB flowbuffers on IB here, 
at least for tmpfs.
>> I was not swapping (I have 8 GB available). Using ramfs instead of  
>> tmpfs, I can get 1,200 MB/s. I have switched to ramfs but the numbers  
>> are roughly the same.
>>     
>
> Much better.  Wonder how your kernel managed to have a bad tmpfs.
>
>   
>>> BMI_OPTIMISTIC_BUFFER_REG.
>>>       
>>  
>> What does this do? Internally, MX can cache some registrations (the  
>> API does not expose it). It is in my best interest to try to reuse  
>> buffers.
>>     
>
> Just chunks up the memory registrations.  If you don't have an MX API
> knob for it, you'll never be able to use it.  Very handy for IB to
> amortize registration overhead though.
>
> Let us know if you manage to figure out why you're not getting good
> MX performance.  Perhaps these other tests can shed some light.  You
> might look at detailed server or client logs with usec timestamps to
> figure out where the time is going.  And oprofile may be useful if
> you think the time is going to the MX libs or driver, or pthread
> overhead.
>
> 		-- Pete
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>
> !DSPAM:45956664175553366512726!
>
>   


-- 
Kyle Schochenmaier
kschoche at scl.ameslab.gov
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory