[Pvfs2-developers] pvfs2-cp profile

[Sam Lang]

On Jan 11, 2007, at 2:39 PM, Scott Atchley wrote:

> On Jan 11, 2007, at 2:51 PM, Julian Martin Kunkel wrote:
>
>> Hi,
>>
>> These results are quite interesting and the possible throughput  
>> you get with
>> IB is amazing.
>
> Do you mean Pete's IB 4x numbers or my MX-10G numbers? Or both?
>
>> However, I think you get into trouble to find the bottleneck of  
>> the operation and the reason for the observed gaps due to the  
>> complexity of the software
>> stack.
>>
>> Maybe you could benefit from using TAS instead of DBPF so that you  
>> could avoid
>> network activity at all. (Also you might use only one buffer to  
>> already fill
>> the network).
>
> I am not sure that I follow you here. Ideally, I only want to  
> measure network activity and PVFS2 overhead. I would prefer to  
> avoid measuring disk activity but these old nodes do not have  
> enough memory to use ramfs well.

I think Julian meant 'avoid _disk_ activity'.  TAS is a replacement  
he wrote for our disk layer, and would be similar to using tmpfs.  I  
should try to merge the TAS stuff into trunk (sorry Julian!).  Its  
been something I've been meaning to do for a while now.

-sam

>
> My MX-10G results are from newer nodes that have enough memory to  
> use ramfs effectively. I can't keep them tied up for bmi_mx  
> development as long as I can these older nodes. :-)
>
>> On the other hand you could use the pvfs2-hint-branch which  
>> provides you with
>> better MPE logging on the server side, we have some tools to  
>> convert and
>> merge client and pvfs2-server logs and show the results. I could  
>> upgrade the
>> current pvfs2-hint-branch with your patches (which is currently  
>> somewhere at
>> release 2.5). The reason we need parts of the advanced logging is  
>> that logs
>> have a problem on the server side if multiple start events occur  
>> before the
>> end events happen, for example if you use multiple flow streams.
>>
>> @Scott
>> I have seen you solved it by using events and wonder which tool  
>> you have used
>> to create states out of the events.
>> You said you have problems with the MPE log on the server, maybe  
>> we could help
>> you if you give details ?
>>
>> Best regards,
>> Julian
>
> I am using MPE. Since I am not using MPI, I compile mpich2 with  
> CLFAGS="-DCLOG_NOMPI". I then add "-lmpe_nompi" to my LIBS and the  
> path to mpich2 to my LDFLAGS.
>
> In my initialization function, I have:
>
> #if BMX_LOGGING
>         MPE_Init_log();
>         send_start              = MPE_Log_get_event_number();
>         send_finish             = MPE_Log_get_event_number();
>         recv_start              = MPE_Log_get_event_number();
>         recv_finish             = MPE_Log_get_event_number();
>         sendunex_start          = MPE_Log_get_event_number();
>         sendunex_finish         = MPE_Log_get_event_number();
>         recvunex_start          = MPE_Log_get_event_number();
>         recvunex_finish         = MPE_Log_get_event_number();
>         MPE_Describe_state(send_start, send_finish, "Send", "red");
>         MPE_Describe_state(recv_start, recv_finish, "Recv", "blue");
>         MPE_Describe_state(sendunex_start, sendunex_finish,  
> "SendUnex", "orange");
>         MPE_Describe_state(recvunex_start, recvunex_finish,  
> "RecvUnex", "green");
> #endif
>
> I then insert MPE_Log_event() in my functions that send, recv and  
> complete messages.
>
> I can now get server logs. My SERVER_LDFLAGS were wrong. Also, on  
> the server, I had to specify an absolute path (I did not on the  
> client).
>
> I would be interested in merging the logs if you can provide some  
> tools or insight.
>
> Thanks,
>
> Scott
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>