[PVFS2-CVS] commit by robl in pvfs2-1/doc: pvfs2-ha.tex

CVS commit program cvs at parl.clemson.edu
Fri Jul 2 11:50:01 EDT 2004


Update of /projects/cvsroot/pvfs2-1/doc
In directory parlweb:/tmp/cvs-serv15542

Modified Files:
	pvfs2-ha.tex 
Log Message:
another round of revisions.  thanks for the suggesions, rross


Index: pvfs2-ha.tex
===================================================================
RCS file: /projects/cvsroot/pvfs2-1/doc/pvfs2-ha.tex,v
diff -u -w -p -u -r1.1 -r1.2
--- pvfs2-ha.tex	29 Jun 2004 20:18:45 -0000	1.1
+++ pvfs2-ha.tex	2 Jul 2004 14:50:01 -0000	1.2
@@ -16,7 +16,6 @@
 \headsep 0.0in
 \headheight 0.0in
 
-
 \title{PVFS2 High-Availibility Clustering}
 \author{PVFS2 Development Team}
 \date{June, 2004}
@@ -31,19 +30,29 @@
 %\setlength{\parindent}{0pt}
 %\setlength{\parskip}{11pt}
 
-\section{Preliminaries}
-Jasmina Janic and the Dell High Performance Computing group (\emph{is that
-the proper atribution}) loaned us (the PVFS2 development team) some Dell
-hardware and some tips on how to configure it in a failover mode.  With
-this hardware, we were able to evaluate several high availability
-solutions and verify PVFS2's performance in that environment.  This
-document would not be possible without their assistance.
+\section{Introduction}
+We designed PVFS2 with performance in mind.  Software redundancy, while
+appealing for its cost and reliability, has a substantial impact on
+performance.  While we are thinking how best to design software
+redundancy, there will always be a performance cost.  Hardware-based
+failover is one way to achieve resistance to failures while maintaining
+high performance.  This document outlines how we set up a PVFS2
+high-availiablity cluster using free software.  First we will walk
+through setting up an active-passive system, then show what needs to be
+changed for a full active-active failover configuration.
 
 Please send updates, suggestions, corrections, and especially any
-caveats with other Linux distributions to
+notes for other Linux distributions to
 \texttt{pvfs2-developers at beowulf-underground.org}
 
 \section{Hardware}
+
+The whole point of failover is for one computer to take over the job of
+another if and when it dies (component failure, power cord unplugged,
+etc.).   The easiest way to achieve that is to have two
+identical machines with some sort of shared storage between them.
+Here's the hardware we used:
+
 \begin{itemize}
 \item 2 Dell PowerEdge 2650s, 
 	each with a PERC (PowerEdge Raid Controller) card
@@ -52,6 +61,10 @@ caveats with other Linux distributions t
 	with 7 160 GB disks configured in RAID-5
 \end{itemize}
 
+It's concievable that a FibreChannel or Firewire drive would suffice for
+the shared storage device.  Reports of success or failure using such
+devices would be most welcome.
+
 \section{Software}
 \subsection{Installing the Operating System}
 Some preliminary notes about installing Linux (Debian) on this hardware:
@@ -62,32 +75,39 @@ Some preliminary notes about installing 
   on other systems and had to do anything differently, please send
   updates.
 
-\item woody boot floppies don't recognize megaraid (PERC) hardware raid, so
-  use the new debian-installer.  NOTE: debian-installer test candidate 1
-  had a bug in base-system, so use debian-installer beta 4
+\item Debian's ``woody'' boot floppies don't recognize megaraid (PERC)
+  hardware raid, so we used the new debian-installer.  NOTE:
+  debian-installer test candidate 1 had a bug in base-system, so use
+  debian-installer beta 4 instead.  By the time you read this,
+  debian-installer will probably be fixed, but beta 4 is known to work
+  on this hardware.
 
-\item once Debian is installed, build a 2.4 kernel, making sure to compile
+\item Once Debian is installed, build a 2.4 kernel, making sure to compile
   in support for \texttt{AIC\_7XXX} and \texttt{MEGARAID2} scsi drivers.
-  There is a \texttt{MEGARAID} and a \texttt{MEGARAID2}.  We need
+  There is a \texttt{MEGARAID} and a \texttt{MEGARAID2}; we need
   megaraid2.  Yes, I said 2.4 -- the megaraid2 driver has not been
-  forward-ported to 2.6 as of this writing (2.6.7-rc3)
+  forward-ported to 2.6 as of Linux 2.6.7-rc3, though it looks like
+  there is some movement on that front. 
 
-\item put the PowerVault controller in \emph{cluster mode}.  To do so,
+\item Put the PowerVault enclosure in \emph{cluster mode}.  To do so,
   flip the little switch on the back of the PowerVault to the posistion
-  with the linked SCSI symbols.  
+  with the linked SCSI symbols.  This is different from putting the
+  controller into cluster mode, which you must also do and is described
+  later on.
 
 \item Turn on the storage and the servers at about the same time or
   weird delays happen
 
 \item There were two SCSI cards in the back of the PowerEdge. I plugged the
-  PowerVault into the 2nd card, channel 1.
+  PowerVault into the 2nd (top) card, channel 1.
 
-\item There are some command-line tools you can use to configure storage
-  volumes, but they are unable to do things like enabling "cluster mode"
-  and changing SCSI id numbers.  Also, they don't work so hot at
-  configuring storage volumes, but that could have just been because the
-  PowerVault was in a weird state.  Still, it's probably best to avoid
-  the command-line tools.
+\item There are some command-line tools you can download from Dell's
+  site to configure storage volumes under Linux, but they are unable to
+  do things like enabling "cluster mode" and changing SCSI id numbers.
+  Also, they don't work so hot at configuring storage volumes, but that
+  could have just been because the PowerVault was in a weird state.
+  Still, it's probably best to set up the PowerVault from the BIOS as
+  outlined below and avoid the command-line tools if possible.
 
 \item Instead of using the command-line tools, configure through the
   bios: hit Ctrl-M when on bootup when prompted to do so.  Once in the
@@ -99,27 +119,34 @@ Some preliminary notes about installing 
   bottom of the screen.  Not exactly intuitive, but at least they are
   documented. For more information, see
   http://docs.us.dell.com/docs/storage/perc3dc/ug/en/index.htm ,
-  particularly the "BIOS Configuration Utility" chapter.
+  particularly the ``BIOS Configuration Utility'' chapter.
 
 \item If toggle on the back of the PowerVault is in cluster mode, and you
-  haven't disabled the PERC bios and put the NVRAM into cluster mode,
-  "weird stuff" happens.   I'm not really sure what i did to make it go
+  haven't disabled the PERC BIOS and put the NVRAM into cluster mode,
+  ``weird stuff'' happens.   I'm not really sure what i did to make it go
   away, but it involved a lot of futzing around with cables and that
   toggle on the back and rebooting nodes. 
 
 \item The GigE chips in the PowerEdge machines don't need a crossover cable:
   they'll figure out how to talk to each other if you plug a
   straight-through or crossover cable between them.  I'm going to say
-  'crossover cable' a lot in this document out of habit.  When i say
-  'crossover' i mean 'either crossover or straight-through'.
+  ``crossover cable'' a lot in this document out of habit.  When I say
+  ``crossover'' I mean ``either crossover or straight-through''.
 
-\item A 100\% legitimate failover configuration would have network power
-  supplies or some other way to STONITH, but we're going to cut some
-  corners here.  One corner case could really mess things up.  If one
-  node thinks the other died, it will start taking over it's operations.
-  If the other node didn't actually die, but just got hung up for a
-  time, it will continue as if everything is OK.  Then you have
-  simultaneous writes and a corrupted file system.
+\item Node failover has one particularly sticky corner case that can
+  really mess things up.  If one node (A) thinks the other (B) died, A
+  will start taking over B's operations.  If B didn't actually die, but
+  just got hung up for a time, it will continue as if everything is OK.
+  Then you have both A and B thinking they controll the file system,
+  both will write to it, and the result is a corrupted file system. A
+  100\% legitimate failover configuration would take measures so that
+  one node can ``fence'' a node -- ensure that it will not attempt to
+  access the storage until forgetting all state.  The most common way to
+  do so is to Shoot The Other Node In The Head (STONITH), and the most
+  common way to STONITH is via network-addressable power supplies.  You
+  can get away without a STONITH mechanisim, and we're going to outline
+  just such a configuration, but just because you { \em can} do something
+  doesn't mean you nescesarily {\em should} do it.  
 
 \item NOTE: the heartbeat software will set up IP addresses and mount file
   systems.  The nodes will have a private (192.168.1.x) address for
@@ -132,12 +159,13 @@ Some preliminary notes about installing 
 
 \subsection{PVFS2}
 
-partition and make a file system on the PowerVault.  If you're going to
+Partition and make a file system on the PowerVault.  If you're going to
 set up Active-Active, make two partitions, else make one.  Mount the
 filesystem somewhere, but don't add an entry to /etc/fstab: heartbeat
-will take care of mounting it once you have things set up.  Reboot the
-other node to make sure it sees the new partition information on the
-enclosure.
+will take care of mounting it once you have things set up, and we are
+mounting the file system just long enough to put a PVFS2 storage space
+on it.  Reboot the other node to make sure it sees the new partition
+information on the enclosure.
 
 Download, build, install, and configure PVFS2.  Create a storage space
 on the PowerVault filesystem.  Now shutdown PVFS2 and unmount the file
@@ -146,7 +174,7 @@ system.  
 \subsection{Failover Software}
 
 There are two main failover packages.  I went with heartbeat from
-linux-ha.org.  There is another package called 'kimberlite', but it
+linux-ha.org.  There is another package called ``kimberlite'', but it
 seems to have bitrotted.  While it has excellent documentation, it
 requires a 'quorum' partition, which the two nodes will write to using
 raw devices.  At some point, something scrambled the main (not raw)
@@ -158,7 +186,7 @@ the config file.
 
 \subsubsection{ACTIVE-PASSIVE (A-P)}
 
-The two nodes are configured as in figure\ref{fig:nodes}.  They have a
+The two nodes are configured as in Figure~\ref{fig:nodes}.  They have a
 private internal network for heartbeat, and a public IP address so
 people can log into them and perform maintenance tasks.
 
@@ -173,22 +201,25 @@ people can log into them and perform mai
 There is a shared "cluster" IP address which is assigned to whichever
 node is active. 
 
-follow GettingStarted.{txt,html} to set up haresources and ha.cf
-Heartbeat ships with a heavily commented ha.cf, haresources, and
-authkeys file.  ha.cf configures the heartbeat infrastructure itself.
-authkeys sets up an authentication mechanism between two nodes.
-haresources describes the actual resources which will migrate from node
-to node.  'Resources' includes IP address, file system partition, and
-service. 
+Follow GettingStarted.\{txt,html\} to set up haresources and ha.cf.
+Heartbeat ships with a heavily commented set of config files:
+\begin{itemize}
+\item ha.cf: configures the heartbeat infrastructure itself.
+\item haresources: describes the actual resources which will migrate
+  from node to node.  'Resources' includes IP address, file system
+  partition, and service. 
+\item authkeys: sets up an authentication mechanism between two nodes.
+\end{itemize}
 
 Copy the ha.cf, haresources, and authkeys files shipped with heartbeat
-to /etc/ha.d and edit them. The defaults are pretty reasonable to get
-started.  For a simple active-passive system
+to the /etc/ha.d directory and edit them. The defaults are pretty
+reasonable to get started.  For a simple active-passive system
 there are only a few settings you need to adjust: see
 Figure~\ref{fig:haconfig}, Figure~\ref{fig:haresources}, and
 Figure~\ref{fig:authkeys} for examples.
 
 \begin{figure}
+\begin{scriptsize}
 \begin{verbatim}
 # pretty self explanatory: send heartbeat logging to the /var/log/ha-log
 # file and also syslog with the 'local0' level
@@ -214,22 +245,24 @@ node pvfs2-ha2
 # heartbeat needs to know the difference between it's partner node
 # dieing and the entire network failing up, so give it the IP address of
 # a stable machine (e.g. a router) in your network.
-ping 140.221.67.253
+ping 10.0.67.253
 
 # the 'ipfail' program keeps an eye on the network
 respawn hacluster /usr/lib/heartbeat/ipfail
 \end{verbatim}
+\end{scriptsize}
 \caption{Minimal \texttt{/etc/heartbeat/ha.cf} file}
 \label{fig:haconfig}
 \end{figure}
 
 
 \begin{figure}
+\begin{scriptsize}
 \begin{verbatim}
 # this line describes resources managed by heartbeat.  
 #   pvfs2-ha1: the primary host for this service.  Heartbeat will start
 #              these resources on pvfs2-ha1 if that node is up
-#   140.221.67.104: the 'cluster' (or shared) IP address for these
+#   10.0.67.104: the 'cluster' (or shared) IP address for these
 #              nodes. refer to the comments in haresources for the many
 #              many options you can use to express network settings. 
 #   Filesystem::/dev/sdb3::/shared::ext3
@@ -242,13 +275,15 @@ respawn hacluster /usr/lib/heartbeat/ipf
 #              When nicely shutting down, will call 'pvfs2 stop'
 #              so make sure the script understands those arguments.
 #              Typical service init scripts work great.
-pvfs2-ha1 140.221.67.104 Filesystem::/dev/sdb3::/shared::ext3 pvfs2
+pvfs2-ha1 10.0.67.104 Filesystem::/dev/sdb3::/shared::ext3 pvfs2
 \end{verbatim}
+\end{scriptsize}
 \caption{Minimal \texttt{/etc/heartbeat/haresources} file}
 \label{fig:haresources}
 \end{figure}
 
 \begin{figure}
+\begin{scriptsize}
 \begin{verbatim}
 # you can specify multiple authentication methods, each prefixed with a
 # 'method-id'. 'auth 1' means use method-id 1
@@ -261,15 +296,50 @@ auth 1
 # to prevent man-in-the-middle attacks on the heartbeat nodes.
 1 crc
 \end{verbatim}
+\end{scriptsize}
 \caption{Example \texttt{/etc/heartbeat/authkeys} file}
 \label{fig:authkeys}
 \end{figure}
 
+\begin{figure}
+\begin{scriptsize}
+\begin{verbatim}
+eth0      Link encap:Ethernet  HWaddr 00:0F:1F:6A:6F:DC  
+          inet addr:10.0.67.105  Bcast:140.221.67.255  Mask:255.255.254.0
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          RX packets:2893591 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:1637691 errors:0 dropped:0 overruns:0 carrier:0
+          collisions:0 txqueuelen:1000 
+          RX bytes:1304753410 (1.2 GiB)  TX bytes:189439176 (180.6 MiB)
+          Interrupt:28 
+
+eth0:0    Link encap:Ethernet  HWaddr 00:0F:1F:6A:6F:DC  
+          inet addr:10.0.67.104  Bcast:140.221.67.255  Mask:255.255.254.0
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          Interrupt:28 
+
+eth1      Link encap:Ethernet  HWaddr 00:0F:1F:6A:6F:DD  
+          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          RX packets:1188003 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:944704 errors:0 dropped:0 overruns:0 carrier:0
+          collisions:0 txqueuelen:1000 
+          RX bytes:197055953 (187.9 MiB)  TX bytes:156677942 (149.4 MiB)
+          Interrupt:29 
+\end{verbatim}
+\end{scriptsize}
+\caption{\texttt{ifconfig} output with an aliased interface. \texttt{eth0:0} is
+an aliased interface for \texttt{eth0}.  \texttt{eth1} is a heartbeat channel,
+over which both nodes in the cluster can communicate their status to each
+other}
+\label{fig:alias}
+\end{figure}
 
 Now you've got heartbeat configured and you've described the resources.
-Fire up heartbeat (/etc/init.d/heartbeat start) on one node and see if
-all the resources start up (you should see an aliased interface bound to
-the cluster ip, the file system mounted, and the pvfs2 servers running).
+Fire up the ``heartbeat'' daemon (/etc/init.d/heartbeat start) on one
+node and see if all the resources start up (you should see an aliased
+interface bound to the cluster ip (see Figure~\ref{fig:alias}), the file system mounted, and the
+pvfs2 servers running).
 Ping the cluster IP from another machine.  If something is broken,
 consult the /var/log/ha-log file or /var/log/syslog and see if any of
 the scripts in /etc/ha.d/resource.d failed.
@@ -279,32 +349,55 @@ Availability (PVFS2 running on one node)
 or pvfs2-cp or mounting the PVFS2 file system from a client (not the
 servers: we're going to reboot them soon to test).  Now start heartbeat
 on the standby server.  Make sure that the IP address, the file system,
-and pvfs2 did not migrate to the standby node.  
+and pvfs2 did not migrate to the standby node --  if you were to use the
+haresources file in Figure~\ref{fig:haresources}, the output of
+\texttt{ifconfig} should still look like Figure~\ref{fig:alias}, you
+would still have /dev/sdb3 mounted on /shared, and pvfs2-server would
+still be running. 
 
 Ok, the moment of truth.  Everything is in place: node A serving PVFS2
 requests, node B ready to step in.  Start a long-running process on the
 client (pvfs2-cp of a large file will work, as will unpacking a tarball
-onto a pvfs2 file system).  Kill node A somehow:  you could be as brutal
+onto a PVFS2 file system).  Kill node A somehow:  you could be as brutal
 as pulling the power cable, or as gentle as /etc/init.d/heartbeat stop.
-(As the heartbeat docs note, don't just pull the network cables out: the
-heartbeat process will become confused when the cables are reconnected.
-yes this is different from stopping heartbeat and starting it up later)
-
-heartbeat will migrate everything to node B, and the client won't notice
-a thing.   Congratulations, you've got High Availability.  To finish,
-bring A back up.  The resources which were on node B will migrate back
-to node A (if you set \texttt{auto\_failback} to 'on' in ha.cf), and the client
-remains oblivious.
+As the heartbeat docs note, don't just pull the network cables out: the
+heartbeat process on both nodes will assume the other process died and
+will attempt to recover.  Rembmer that ``takeover'' means taking over
+the IP address, file system, and programs, so you will have two nodes
+writing to the same file system and trying to share the same ip address.
+When you plug the network cables back in, you will have network
+collisions and simulatneous writes to the filesystem.  Yes this is
+different from stopping heartbeat and starting it up later: when
+heartbeat starts, it checks to see the state of its partner node, and
+will do the right thing. 
+
+If the failover works correctly, heartbeat will migrate everything to
+node B, and the client won't notice a thing.   Congratulations, you've
+got High Availability.  To finish, bring A back up.  The resources which
+were on node B will migrate back to node A (if you set
+\texttt{auto\_failback} to 'on' in ha.cf), and the client remains
+oblivious.
 
 \subsubsection{Active-Active (A-A)}
 
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.75]{pvfs2-failover-AA.eps}
+\end{center}
+\caption{Simplified wiring diagram of a PVFS2 HA cluster, Active-Active
+confgiuration}
+\label{fig:nodes-aa}
+\end{figure}
+
 If that wasn't exciting enough, we can do active-active, too.  It's
 pretty much like active-passive, except both nodes are pvfs2 servers.
 Instead of sharing one cluster IP, there will be two -- one for each
 server.  Instead of sharing one file system, there will be two.  If A
 dies, B will serve it's data and A's data, and vice versa.  You get all
 the benefits of Active-Passive, but you don't have a server waiting idly
-for a (hopefully rare) failure.
+for a (hopefully rare) failure.   Figure~\ref{fig:nodes-aa} depicts an
+Active-Active cluster.
 
 As mentioned above, you'll need two partitions on the shared storage and
 two shared IP addresses.  configure PVFS2 on the two servers as you
@@ -333,34 +426,122 @@ resources.d directory.  PVFS2 has an exa
 examples/pvfs2-server.rc you can use to start.  Make sure PVFS2\_FS\_CONF
 and PVFS2\_SERVER\_CONF point to the proper config files (it will guess
 the wrong ones if you don't specify them) and PVFS2\_PIDFILE is different
-in both scripts.
+in both scripts.  See Figure~\ref{fig:init} and
+Figure~\ref{fig:init-other}.
+
+\begin{figure}
+\begin{scriptsize}
+\begin{verbatim}
+PVFS2_FS_CONF=/etc/pvfs2/fs.conf
+PVFS2_SERVER_CONF=/etc/pvfs2/server.conf-140.221.67.104
+        
+# override this if your server binary resides elsewhere
+PVFS2SERVER=/usr/local/sbin/pvfs2-server
+# override this if you want servers to automatically pick a conf file,
+#   but you just need to specify what directory they are in
+PVFS2_CONF_PATH=/etc/pvfs2
+PVFS2_PIDFILE=/var/run/pvfs2-1.pid
+...  # remainder of init script omitted
+\end{verbatim}
+\end{scriptsize}
+\caption{Exerpt from PVFS2 init script on one A-A node}
+\label{fig:init}
+\end{figure}
 
 \begin{figure}
+\begin{scriptsize}
+\begin{verbatim}
+PVFS2_FS_CONF=/etc/pvfs2/fs.conf
+PVFS2_SERVER_CONF=/etc/pvfs2/server.conf-140.221.67.107
+        
+# override this if your server binary resides elsewhere
+PVFS2SERVER=/usr/local/sbin/pvfs2-server
+# override this if you want servers to automatically pick a conf file,
+#   but you just need to specify what directory they are in
+PVFS2_CONF_PATH=/etc/pvfs2
+PVFS2_PIDFILE=/var/run/pvfs2-2.pid
+...  # remainder of init script omitted
+\end{verbatim}
+\end{scriptsize}
+\caption{Exerpt from PVFS2 init script on the other A-A node}
+\label{fig:init-other}
+\end{figure}
+
+\begin{figure}
+\begin{scriptsize}
 \begin{verbatim}
 # Each server has its associated IP address and file system.  pvfs2-1,
-# for example,  is associated with 140.221.67.104 and has its data on
+# for example,  is associated with 10.0.67.104 and has its data on
 # sdb1.  
 # 
 # note that each server has its own file system.  You must have a
 # dedicated partition for each service you run via heartbeat.
 
-pvfs2-ha1 140.221.67.104 Filesystem::/dev/sdb1::/mnt/shared1::ext3 pvfs2-1
-pvfs2-ha2 140.221.67.107 Filesystem::/dev/sdb2::/mnt/shared2::ext3 pvfs2-2
+pvfs2-ha1 10.0.67.104 Filesystem::/dev/sdb1::/mnt/shared1::ext3 pvfs2-1
+pvfs2-ha2 10.0.67.107 Filesystem::/dev/sdb2::/mnt/shared2::ext3 pvfs2-2
 \end{verbatim}
+\end{scriptsize}
 \caption{haresources file, Active-Active configuration}
 \label{fig:haresources-aa}
 \end{figure}
 
 The ha.cf file looks the same in A-A as it does in A-P, as does the
-authkeys.  We only have to add an entry to haresources.  See
+authkeys.  We only have to add an entry to haresources indicating that
+heartbeat needs to manage two separate resources.  See
 Figure~\ref{fig:haresources-aa}.
 
-
 Start heartbeat on both machines.  See if a client can reach the servers
 (e.g. pvfs2-ping).  Kill a machine.  The resources that were on that
 machine (IP address, file system, pvfs2-servers) will migrate to the
 machine that is still up.  Clients won't notice a thing.
+Figure~\ref{fig:ifconfig-aa} shows node A after node B goes down.  Node A
+now has both of the two cluster IP addresses bound to two aliased
+interfaces B while continuing to manage it's default resource.
+
+\begin{figure}
+\begin{scriptsize}
+\begin{verbatim}
+eth0      Link encap:Ethernet  HWaddr 00:0F:1F:6A:6F:DC  
+          inet addr:140.221.67.105  Bcast:140.221.67.255  Mask:255.255.254.0
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          RX packets:2911950 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:1647984 errors:0 dropped:0 overruns:0 carrier:0
+          collisions:0 txqueuelen:1000 
+          RX bytes:1306604241 (1.2 GiB)  TX bytes:190743053 (181.9 MiB)
+          Interrupt:28 
+
+eth0:0    Link encap:Ethernet  HWaddr 00:0F:1F:6A:6F:DC  
+          inet addr:140.221.67.104  Bcast:140.221.67.255  Mask:255.255.254.0
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          Interrupt:28 
+
+eth0:1    Link encap:Ethernet  HWaddr 00:0F:1F:6A:6F:DC  
+          inet addr:140.221.67.107  Bcast:140.221.67.255  Mask:255.255.254.0
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          Interrupt:28 
+
+eth1      Link encap:Ethernet  HWaddr 00:0F:1F:6A:6F:DD  
+          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
+          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+          RX packets:1197984 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:954689 errors:0 dropped:0 overruns:0 carrier:0
+          collisions:0 txqueuelen:1000 
+          RX bytes:198722216 (189.5 MiB)  TX bytes:158334802 (150.9 MiB)
+          Interrupt:29 
+\end{verbatim}
+\end{scriptsize}
+\caption{\texttt{ifconfig} output.  This node now has both cluster IP
+addresses .}
+\label{fig:ifconfig-aa}
+\end{figure}
 
+\section{Acknowledgements}
+We would like to thank Jasmina Janic for notes and technical support.
+The Dell Scalable Systems Group loaned the PVFS2 development team Dell
+hardware.  With this hardware, we were able to evaluate several high
+availability solutions and verify PVFS2's performance in that
+environment.  This document would not be possible without their
+assistance.
 \end{document}
 
 % vim: tw=72



More information about the PVFS2-CVS mailing list