[Pvfs2-users] PVFS2 on a large cluster

[Dean Hildebrand]

Here are two *very* basic ways that I use to start pvfs2 on my 8 node 
fedora cluster.  Not fancy, but quick.

MPI
====
mpiexec -n 8 insmod /usr/local/bin/pvfs2-server/pvfs2.ko
mpiexec -n 8 /usr/local/bin/pvfs2-server/sbin/pvfs2-client -p 
/usr/local/bin/pvfs2-server/sbin/pvfs2-client-core
mpiexec -n 8 mount -t pvfs2 tcp://iota1:3334/pvfs2-fs /mnt/pvfs2

XARGS
======
cat clients | xargs -i ssh {} 'insmod /usr/local/bin/pvfs2-server/pvfs2.ko'
cat clients | xargs -i ssh {} 
'/usr/local/bin/pvfs2-server/sbin/pvfs2-client -p 
/usr/local/bin/pvfs2-server/sbin/pvfs2-client-core'
cat clients | xargs -i ssh {} 'mount -t pvfs2 tcp://iota1:3334/pvfs2-fs 
/mnt/pvfs2'


To shutdown:
cat clients | xargs -i ssh {} 'umount /mnt/pvfs2'
cat clients | xargs -i ssh {} 'killall pvfs2-client'
sleep 2
cat clients | xargs -i ssh {} 'rmmod pvfs2'

Robert Latham wrote:
> On Wed, Nov 15, 2006 at 05:41:45PM +0530, Vikrant Kumar wrote:
>   
>> 1. I understand that a 1.6 release is around the corner. Can anyone
>> provide a more definite time? Would be really helpful to plan things
>> on my side.
>>     
>
> You probably already saw this, but we released PVFS 2.6.0 today.
>
>   
>> 2. I have a cluster with 11 nodes connected with Infiniband . Each
>> node has four Opteron cores. My application uses MPI IO and I would
>> like to test it up to 32 way. Given the cluster and the application
>> requirement, what would be the ideal number of servers for this
>> set-up? I was thinking either four or eight PVFS2 servers, being
>> shared with the computing-cores on some nodes.  I have read the FAQ
>> entry and I guess this is a very general question.  But I just
>> wanted to get a quick review of the layout from you guys.
>>     
>
> 8 is a good number for a lot of workloads.  You can use fewer
> datafiles if you find your workload uses lots of small files, while
> still retaining the ablity to have all 8 servers working together for
> larger file accesses. 
>
>   
>> 3. Another thing is when a node is rebooted. The provided script
>> file "pvfs2-server.rc" can take care of the server but what about
>> client application, the kernel module and the mount point? Is there
>> a way to automate the startup of all these as well. I think this
>> would be a critical thing for this bigger installation.
>>     
>
> The problem with automating the kernel moudle and mounting process is
> that every distribution does it a little differently.  This is an area
> where we really can't make too many assumptions.  The best way to
> automate it is to create a script that does the appropriate things
> (load kernel module, start up pvfs2-client and pvfs2-client-core,
> mount the file system) for your system.   Perhaps admins from other
> sites might have ideas for you.
>
> Have fun
> ==rob
>
>   

-- 
Dean Hildebrand
Ph.D. Candidate
University of Michigan