[Pvfs2-developers] pvfs2-cp profile

[Scott Atchley]

Hi all,

I used MPE to measure send, sendunexpected and recv on the client  
during pvfs2-cp (from the client to the server) of a (just less than)  
256 MB file. This is using older 2.4 GHz Xeons with MX-2G (2 Gb/s  
line rate). I am using FlowBufferSizeBytes of 1 MB. I am using the  
default FlowBuffersPerFlow. The server did not generate a log and I  
am looking into why.

This first image shows the entire copy of the file from client to  
server. Recv are blue, send are red and sendunexpected are orange.  
There 25 transfer of 10 MB each and the 26th transfer is ~5.5 MB.  
Note that there are distinct gaps between transfers in the range of  
17-18 msecs.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2-cp-3.png
Type: image/png
Size: 29812 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20070110/60cd010a/pvfs2-cp-3-0001.png
-------------- next part --------------

The second image shows one of the 25 bulk IO transfers. There is a 11  
msec delay from the last send completing to the recv completing (the  
large blue box). On the very far left is the initial sendunexpected  
that starts this operation.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2-cp-2.png
Type: image/png
Size: 8500 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20070110/60cd010a/pvfs2-cp-2-0001.png
-------------- next part --------------

The third image is of the start of the operation and includes the  
first two 1 MB transfers. At the far left, you see the client post a  
recv, then post the sendunexpected that contacts the server to start  
the operation. Once the first recv completes (the server has  
replied), the client sends the first 1 MB. Jumpshot did not draw this  
in a way that is intuitive. The small red column on the left actually  
completes just before the large red box on the right.

It is interesting that pvfs2-cp posts the first 1 MB send before the  
recv. In actuality, it does not matter because the recv is posted  
before the last IO transfer per operation.

The individual 1 MB transfers take about 4.5 msec, which works out to  
about 1.8 Gb/s (out of the possible 2 Gb/s). The overall throughput  
that pvfs2-cp reports is 1.0 Gb/s.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2-cp-1.png
Type: image/png
Size: 8931 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20070110/60cd010a/pvfs2-cp-1-0001.png
-------------- next part --------------

I wonder if the gaps between operations is due to reading in the file  
from disk or something else. Are the delays at the end of the  
operation (after the last send completes until the recv completes)  
due to the server processing the received data before replying to the  
client?

I will try this next on perf.

Scott