[Pvfs2-users] myrinet MX and Ethernet

[Rob Ross]

Hi Scott,

Would it be easy to patch bmi_mx to disable shared memory use until a 
work-around is created?

Thanks,

Rob

Scott Atchley wrote:
> On Jul 5, 2007, at 3:47 PM, Robert Latham wrote:
> 
>> On Thu, Jul 05, 2007 at 02:37:02PM -0500, Glenn Johnson wrote:
>>> The compute nodes will have the Myri-10G cards. The disks on the compute
>>> nodes will make up a PVFS2 file system. The compute nodes will also be
>>> PVFS2 clients. Data on this file system will consist of raw output from
>>> calculations. The data files will not be immediately deleted but they
>>> will not be backed up either. Users are expected to save post-processing
>>> files in the home directory on the head node. The head node does not
>>> have a Myri-10G card. I would like to be able to mount the PVFS2 file
>>> system on the head node via Ethernet (Gb) if that is possible. I guess
>>> what I am not sure about is if I can use the MX bmi for the I/O servers
>>> and clients on the compute nodes and TCP for the client on the head
>>> node.
>>
>> Since the head node won't have MX, one approach might be to
>> "multi-home" the pvfs servers so they are listening on both the MX
>> interface (for the compute nodes) and on TCP (for the head node).
>> Pete does something like this for his Infiniband cluster.
>>
>> Another approach, though one that is not likely to give great
>> performance, is to have the PVFS clients export the file system via
>> NFS to the head node.
>>
>> ==rob
> 
> Glenn,
> 
> For the nodes with Myri-10G cards that will be using bmi_mx, there is a 
> known issue if clients and servers are run on the same node _and_ the 
> client and server are run by different users. By default, MX tries to 
> use shared memory for traffic between processes on the same node. This 
> requires the processes to have the same owner. To avoid this issue, add 
> MX_DISABLE_SHMEM=1 to your environment until we have a longer-term 
> solution. Setting this variable will turn off MX shared memory and it 
> will cause MX to send local messages to the switch and back.
> 
> Scott
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>