[Pvfs2-users] MX help

[Scott Atchley]

On Mar 5, 2009, at 8:46 AM, Robert Latham wrote:

> On Wed, Mar 04, 2009 at 07:15:24PM -0500, Bradley Settlemyer wrote:
>> Hello
>>
>>  I am trying to use PAV to run pvfs with the MX protocol.  I've
>> updated pav so that servers start and ping correctly.  But when I try
>> and run an mpi code, I'm getting client timeouts like the client
>> cannot contact the servers:
>>
>> Lots of this stuff:
>>
>> [E 19:11:02.573509] job_time_mgr_expire: job time out: cancelling bmi
>> operation, job_id: 3.
>> [E 19:11:02.583659] msgpair failed, will retry: Operation cancelled
>> (possibly due to timeout)

Brad, which version of MX and PVFS2?

> OK, so pvfs utilities are all hunky-dory? not just pvfs2-ping but
> pvfs2-cp and pvfs2-ls?
>
> On Jazz, I usually configure MPICH2 to communicate over TCP and have
> the PVFS system interface communicate over MX.  This keeps the
> situation fairly simple, but of course you get awful MPI performance.
>
> Does MX still have the "ports" restriction that GM has?  I wonder if
> MPI communication is getting in the way of PVFS communication...
>
> In short, I don't exactly know what's wrong myself.  just tossing out
> some theories.
>
> ==rob

Rob, MX is limited to 8 endpoints per NIC. One can use mx_info to get  
the number:

8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host

This can be increased to 16 with a module parameter.

Generally, you want no more than one endpoint per process and one  
process per core for MPI. When you want to use MPI-IO over PVFS2, each  
process will need two endpoints (one for MPI and one for PVFS2). If  
you have eight cores, you should increase the max endpoints to 16 (if  
you have eight cores).

Generally, I would not want to limit my MPI to TCP and IO to MX  
especially if my TCP is over gigabit Ethernet. Unless your IO can  
exceed the link rate, there will be plenty of bandwidth left over for  
MPI and your latency will stay much lower than TCP.

What is PAV?

Scott