[Pvfs2-users] myrinet MX and Ethernet

[Scott Atchley]

On Jul 5, 2007, at 3:47 PM, Robert Latham wrote:

> On Thu, Jul 05, 2007 at 02:37:02PM -0500, Glenn Johnson wrote:
>> The compute nodes will have the Myri-10G cards. The disks on the  
>> compute
>> nodes will make up a PVFS2 file system. The compute nodes will  
>> also be
>> PVFS2 clients. Data on this file system will consist of raw output  
>> from
>> calculations. The data files will not be immediately deleted but they
>> will not be backed up either. Users are expected to save post- 
>> processing
>> files in the home directory on the head node. The head node does not
>> have a Myri-10G card. I would like to be able to mount the PVFS2 file
>> system on the head node via Ethernet (Gb) if that is possible. I  
>> guess
>> what I am not sure about is if I can use the MX bmi for the I/O  
>> servers
>> and clients on the compute nodes and TCP for the client on the head
>> node.
>
> Since the head node won't have MX, one approach might be to
> "multi-home" the pvfs servers so they are listening on both the MX
> interface (for the compute nodes) and on TCP (for the head node).
> Pete does something like this for his Infiniband cluster.
>
> Another approach, though one that is not likely to give great
> performance, is to have the PVFS clients export the file system via
> NFS to the head node.
>
> ==rob

Glenn,

For the nodes with Myri-10G cards that will be using bmi_mx, there is  
a known issue if clients and servers are run on the same node _and_  
the client and server are run by different users. By default, MX  
tries to use shared memory for traffic between processes on the same  
node. This requires the processes to have the same owner. To avoid  
this issue, add MX_DISABLE_SHMEM=1 to your environment until we have  
a longer-term solution. Setting this variable will turn off MX shared  
memory and it will cause MX to send local messages to the switch and  
back.

Scott