[Pvfs2-developers] Suggested testing for bmi_mx?

[Scott Atchley]

Hi all,

For the sendunexpected test, the client calls sendunexpected, the  
server spins on testunexpected, then sends a matching size, expected  
message back to the client.

Here are some sendunexpected numbers:

Length		Lat (us)	BW (MB/s)
1		19.93		0.05
64		42.03		1.52
256		21.06		12.15
512		47.24		10.84
1024		32.19		31.82
4096		74.55		54.94

Performance is far less since there is a malloc() on the server for  
every received unexpected message. I do not know why the numbers vary  
between sizes the way that they do. I will look into it more next week.

Scott

On Jan 26, 2007, at 4:35 PM, Scott Atchley wrote:

> Hi Murali,
>
> Ok, I will check them out.
>
> In the meantime, I have written a test similar to IMB PingPong that  
> uses BMI directly. It should work with TCP, GM, MX, and IB. Below  
> are some various results for some old Xeons with Myrinet-2000 cards  
> (250 MB/s link rate).
>
> The latency is one-way and throughput is bi-directional.
>
> I will also write a version that tests unexpected messages up to  
> unexpected max size.
>
> Scott
>
> bmi_mx results:
>
> Length		Lat (us)	BW (MB/s)
> 1		7.97		0.13
> 64		8.85		7.23
> 256		11.99		21.35
> 512		13.76		37.20
> 1024		17.76		57.67
> 4096		32.49		126.07
> 8192		54.38		150.65
> 32768		158.41		206.85
> 1048576		4583.91		228.75
>
> With the registration cache:
> 1048576		4305.72		243.53
>
>
> For comparison, these are mx_pingpong (raw MX) results for the same  
> message sizes:
>
>    Length   Latency(us)    Bandwidth(MB/s)
>         1       3.466          0.288
>        64       4.587         13.954
>       256       7.141         35.852
>       512       9.089         56.329
>      1024      12.937         79.153
>      4096      26.696        153.431
>      8192      46.815        174.987
>     32768     154.087        212.659
>   1048576    4271.931        245.457
>
> BMI and/or bmi_mx adds about 4.5 us additional latency. It can get  
> close to line rate with or without the MX registration cache.
>
> Scott
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers