[Pvfs2-developers] pvfs2-cp profile

[Scott Atchley]

On Jan 11, 2007, at 2:51 PM, Julian Martin Kunkel wrote:

> Hi,
>
> These results are quite interesting and the possible throughput you  
> get with
> IB is amazing.

Do you mean Pete's IB 4x numbers or my MX-10G numbers? Or both?

> However, I think you get into trouble to find the bottleneck of the  
> operation and the reason for the observed gaps due to the  
> complexity of the software
> stack.
>
> Maybe you could benefit from using TAS instead of DBPF so that you  
> could avoid
> network activity at all. (Also you might use only one buffer to  
> already fill
> the network).

I am not sure that I follow you here. Ideally, I only want to measure  
network activity and PVFS2 overhead. I would prefer to avoid  
measuring disk activity but these old nodes do not have enough memory  
to use ramfs well.

My MX-10G results are from newer nodes that have enough memory to use  
ramfs effectively. I can't keep them tied up for bmi_mx development  
as long as I can these older nodes. :-)

> On the other hand you could use the pvfs2-hint-branch which  
> provides you with
> better MPE logging on the server side, we have some tools to  
> convert and
> merge client and pvfs2-server logs and show the results. I could  
> upgrade the
> current pvfs2-hint-branch with your patches (which is currently  
> somewhere at
> release 2.5). The reason we need parts of the advanced logging is  
> that logs
> have a problem on the server side if multiple start events occur  
> before the
> end events happen, for example if you use multiple flow streams.
>
> @Scott
> I have seen you solved it by using events and wonder which tool you  
> have used
> to create states out of the events.
> You said you have problems with the MPE log on the server, maybe we  
> could help
> you if you give details ?
>
> Best regards,
> Julian

I am using MPE. Since I am not using MPI, I compile mpich2 with  
CLFAGS="-DCLOG_NOMPI". I then add "-lmpe_nompi" to my LIBS and the  
path to mpich2 to my LDFLAGS.

In my initialization function, I have:

#if BMX_LOGGING
         MPE_Init_log();
         send_start              = MPE_Log_get_event_number();
         send_finish             = MPE_Log_get_event_number();
         recv_start              = MPE_Log_get_event_number();
         recv_finish             = MPE_Log_get_event_number();
         sendunex_start          = MPE_Log_get_event_number();
         sendunex_finish         = MPE_Log_get_event_number();
         recvunex_start          = MPE_Log_get_event_number();
         recvunex_finish         = MPE_Log_get_event_number();
         MPE_Describe_state(send_start, send_finish, "Send", "red");
         MPE_Describe_state(recv_start, recv_finish, "Recv", "blue");
         MPE_Describe_state(sendunex_start, sendunex_finish,  
"SendUnex", "orange");
         MPE_Describe_state(recvunex_start, recvunex_finish,  
"RecvUnex", "green");
#endif

I then insert MPE_Log_event() in my functions that send, recv and  
complete messages.

I can now get server logs. My SERVER_LDFLAGS were wrong. Also, on the  
server, I had to specify an absolute path (I did not on the client).

I would be interested in merging the logs if you can provide some  
tools or insight.

Thanks,

Scott