[Pvfs2-developers] pvfs2-cp profile

[Sam Lang]

On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:

> Hi all,
>
> I used MPE to measure send, sendunexpected and recv on the client  
> during pvfs2-cp (from the client to the server) of a (just less  
> than) 256 MB file. This is using older 2.4 GHz Xeons with MX-2G (2  
> Gb/s line rate). I am using FlowBufferSizeBytes of 1 MB. I am using  
> the default FlowBuffersPerFlow. The server did not generate a log  
> and I am looking into why.
>
> This first image shows the entire copy of the file from client to  
> server. Recv are blue, send are red and sendunexpected are orange.  
> There 25 transfer of 10 MB each and the 26th transfer is ~5.5 MB.  
> Note that there are distinct gaps between transfers in the range of  
> 17-18 msecs.
>
> <pvfs2-cp-3.png>
>
> The second image shows one of the 25 bulk IO transfers. There is a  
> 11 msec delay from the last send completing to the recv completing  
> (the large blue box). On the very far left is the initial  
> sendunexpected that starts this operation.
>
> <pvfs2-cp-2.png>
>
> The third image is of the start of the operation and includes the  
> first two 1 MB transfers. At the far left, you see the client post  
> a recv, then post the sendunexpected that contacts the server to  
> start the operation. Once the first recv completes (the server has  
> replied), the client sends the first 1 MB. Jumpshot did not draw  
> this in a way that is intuitive. The small red column on the left  
> actually completes just before the large red box on the right.
>
> It is interesting that pvfs2-cp posts the first 1 MB send before  
> the recv. In actuality, it does not matter because the recv is  
> posted before the last IO transfer per operation.

Hi Scott,  Nice images.  Rob and I had discussed this behavior you're  
seeing of receiving expected messages before the receive has been  
posted on the server.  We came up with a solution for this (or at  
least, something for you to try), and I started implementing it, but  
got pulled away by other things.  I'll try to finish that up and get  
you a patch to try.

>
> The individual 1 MB transfers take about 4.5 msec, which works out  
> to about 1.8 Gb/s (out of the possible 2 Gb/s). The overall  
> throughput that pvfs2-cp reports is 1.0 Gb/s.
>
> <pvfs2-cp-1.png>
>
> I wonder if the gaps between operations is due to reading in the  
> file from disk or something else.

If I understand correctly, this test was a write.  The gaps are  
likely writing the data to disk.  You could verify this by watching  
the write acks that the server sends back to the client once all the  
data has been written to disk for each operation.  My guess would be  
that the ack would come write before the next recv (blue block)  
started, covering the gap between operations.

To give more detail, for write operations, the client posts a receive  
for the write ack before posting the expected sends.  Once the server  
completes all the IO (pushes it to disk), it sends an expected 'write- 
completed' message back to the client.  You could probably track this  
similar to how you're tracking the sendunexpected and response ack at  
the beginning of the IO operation.

-sam

> Are the delays at the end of the operation (after the last send  
> completes until the recv completes) due to the server processing  
> the received data before replying to the client?
>
> I will try this next on perf.
>
> Scott_______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers