[Pvfs2-cvs] commit by kunkel in pvfs2/doc: pvfs2-tuning.tex
pvfs2-guide.tex pvfs2-quickstart.tex pvfs2-faq.tex
module.mk.in tuning.tex
CVS commit program
cvs at parl.clemson.edu
Sat Dec 2 06:12:40 EST 2006
Update of /projects/cvsroot/pvfs2/doc
In directory parlweb1:/tmp/cvs-serv17506/doc
Modified Files:
Tag: kunkel-hint-branch
pvfs2-guide.tex pvfs2-quickstart.tex pvfs2-faq.tex
module.mk.in
Added Files:
Tag: kunkel-hint-branch
pvfs2-tuning.tex
Removed Files:
Tag: kunkel-hint-branch
tuning.tex
Log Message:
Upgrade to current CVS version
--- /dev/null 2004-06-24 14:04:38.000000000 -0400
+++ pvfs2-tuning.tex 2006-12-02 06:12:40.000000000 -0500
@@ -0,0 +1,154 @@
+\documentclass[11pt,letterpaper]{article}
+\usepackage{html}
+\usepackage{charter}
+\pagestyle{empty}
+
+\topmargin 0.0in
+\textwidth 6.5in
+\textheight 9.0in
+\columnsep 0.25in
+\oddsidemargin 0.0in
+\evensidemargin 0.0in
+\headsep 0.0in
+\headheight 0.0in
+
+\title{PVFS Tuning}
+\author{ PVFS Development Team }
+
+\begin{document}
+\maketitle
+
+\tableofcontents
+\thispagestyle{empty}
+
+\section{Introduction}
+
+The default settings for PVFS (those provided and in the source code
+and added to the config files by \texttt{pvfs2-genconfig})
+provide good performance on most systems and for a wide variety of workloads.
+This document describes system level and PVFS specific parameters
+that can be tuned to improve
+performance on specific hardware, or for specific workloads and usage
+scenarios.
+
+In general performance tuning should begin with the available hardware
+and system software, to maximize the bandwidth of the network and
+transfer rates of the storage hardware. From there, PVFS server
+parameters can be tuned to improve performance of the
+entire system, especially
+if specific usage scenarios are targetted. Finally, file system
+extended attributes and hints can be tweaked by different users to
+improve individual performance within a system with varying workloads.
+
+Some (especially system level) parameters can be tuned to provide better
+performance without sacrificing another property of the system.
+Tuning some parameters though, may have a direct effect on the
+performance of other usage scenarios, or some other property of the
+system (such as durability).
+Our discussion of performance tuning will include the tradeoffs
+that must be made during the tuning process, but the final decisions are best
+made by the administrators to determine the optimal
+setup that meets the needs of their users.
+
+\section{Cluster Partitioning}
+
+For users that have one use case, and a generic cluster, what's the best
+partition of compute/IO nodes? Is this section needed?
+
+\section{Storage}
+
+\subsection{Server Configuration}
+
+How many IO servers? \footnote{The FAQ already answers this to some
+degree}
+How many MD servers?
+Should IO and MD servers be shared?
+
+\subsection{Local File System}
+
+\begin{itemize}
+\item ext3
+\item xfs
+\end{itemize}
+
+\subsection{Disk Synchronization}
+
+The easiest way to see an improvement in performance is to set the
+\texttt{TroveSyncMeta} and \texttt{TroveSyncData} attributes to ``no''
+in the \texttt{<StorageHints>} section. If those attributes are set to
+``no'' then Trove will read and write data from a cache and not the
+underlying file. Performance will increase greatly, but if the server
+dies at some point, you could lose data. At this point in PVFS2
+development, server crashes are rare outside of hardware failures.
+PVFS2 developers should probably leave these settings to ``yes''. If
+PVFS2 hosts the only copy of your data, leave these settings to ``yes''.
+Otherwise, give ``no'' a shot.
+
+Sync or not, metadata, data
+ coalescing
+
+ distributed metadata
+
+\subsection{Metadata}
+\subsubsection{Coalescing}
+
+\subsection{Data}
+
+\section{Networks}
+
+\subsection{Network Independent}
+
+\begin{enumerate}
+\item Unexpected message size
+\item Flow Parameters
+\end{enumerate}
+
+\begin{itemize}
+\item buffer size
+\item count
+\end{itemize}
+
+\subsection{TCP}
+
+\subsubsection{Kernel Parameters}
+\subsubsection{Socket Buffer Sizes}
+\subsubsection{Listening Backlog (?)}
+
+\subsection{Infiniband}
+\subsection{Myrinet Express}
+
+\section{VFS Layer}
+
+\subsection{Maximum I/O Size}
+
+\subsection{Workload Specifics}
+
+\subsection{Extended Attributes}
+
+\subsubsection{Directory Hints}
+
+\begin{itemize}
+\item Number of Datafiles
+\item Stripe Size
+\item Distribution
+\end{itemize}
+
+\section{Workloads}
+
+\subsection{Small files}
+\subsection{Large Files}
+\subsection{Concurrent IO}
+
+\section{Benchmarking}
+
+\begin{itemize}
+\item mpi-io-test
+\item mpi-md-test
+\end{itemize}
+
+\section{References}
+
+\end{document}
+
+
+% vim: tw=72
Index: pvfs2-guide.tex
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/pvfs2-guide.tex,v
diff -p -u -r1.4 -r1.4.40.1
--- pvfs2-guide.tex 16 Nov 2004 17:12:48 -0000 1.4
+++ pvfs2-guide.tex 2 Dec 2006 11:12:40 -0000 1.4.40.1
@@ -61,8 +61,5 @@
\newpage
\input{code-tree.tex}
-\newpage
-\input{tuning.tex}
-
\end{document}
Index: pvfs2-quickstart.tex
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/pvfs2-quickstart.tex,v
diff -p -u -r1.39 -r1.39.2.1
--- pvfs2-quickstart.tex 10 Aug 2006 19:02:20 -0000 1.39
+++ pvfs2-quickstart.tex 2 Dec 2006 11:12:40 -0000 1.39.2.1
@@ -86,6 +86,8 @@ as part of the following MPI implementat
\begin{itemize}
\item MPICH2-0.96p2 or newer, though we suggest using the most recent MPICH2
release
+\item OpenMPI-1.0 or newer, though it may not have some of the bug fixes or
+features of the MPICH2 version
\end{itemize}
\subsection{Untarring the packages}
@@ -950,33 +952,40 @@ The diagnostic level will be a bitwise O
\section{ROMIO Support}
\label{sec:romio}
-Building ROMIO with PVFS2 support can be a bit tricky, and is certainly
-not well documented. While ROMIO has been updated with PVFS2 support,
-only MPICH2 has included a recent snapshot of ROMIO.
+Building ROMIO with PVFS2 support can be a bit tricky, and is certainly not
+well documented. Reports of the correct way to build for OpenMPI would be
+appreciated. This document will cover building for MPICH2.
First, get the software. Download MPICH2 from
-http://www.mcs.anl.gov/mpi/mpich2/. While MPICH2 contains a ROMIO
-with PVFS2 support, the PVFS2 API has continued to stabilize since
-MPICH2 was last released. Patches to syncronize the ROMIO in MPICH2
-with PVFS2 can be found at
-http://www.mcs.anl.gov/romio/pvfs2-patches.html. There is also a
-patch in the PVFS2 source in {\tt doc/coding}. For example's sake,
-assume all software was downloaded to {\tt \$\{HOME\}/src}.
+http://www.mcs.anl.gov/mpi/mpich2/. We may have found bug fixes since the last MPICH2 release. If there are bug fixes, they can be found at
+http://www.mcs.anl.gov/romio/pvfs2-patches.html.
-Unpack mpich2, then change to the src/mpi/romio directory. Apply the
-ROMIO patch. The patch makes changes to the ROMIO configure scripts, so you'll have to re-run {\tt autoconf} to generate a new configure file.
+Unpack mpich2. Apply the ROMIO patch in the src/mpi/romio directory if one is
+needed.
\begin{verbatim}
-prompt% tar xzf ~/src/mpich2-0.96p2.tar.gz # unpack mpich2 source
-prompt% cd mpich2-0.96p2/src/mpi/romio # change to ROMIO dir
+prompt% tar xzf ~/src/mpich2-1.4.0p1.tar.gz # unpack mpich2 source
+prompt% cd mpich2-1.4.0p1/src/mpi/romio # change to ROMIO dir
prompt% patch -p1 < ~/src/romio-<CORRECT_VERSION>.diff #apply patch
-prompt% autoconf # create a new 'configure'
prompt% cd ../../.. # return to top of src
prompt%
\end{verbatim}
-In order to build MPICH2 with a ROMIO that speaks PVFS2, a few changes
-have to be made to the normal configure process. MPICH2 will need to
+In order to build MPICH2 with a ROMIO that speaks PVFS2, pass the
+\texttt{--with-pvfs2=PVFS\_PREFIX} option to configure. 'PVFS2\_PREFIX' is the
+prefix you gave to the PVFS configure script (e.g. /usr/local or
+/opt/packages/pvfs-2.6.0).
+
+\begin{verbatim}
+configure --with-pvfs2=PVFS_PREFIX [other flags]
+\end{verbatim}
+
+Now compile and install MPICH2 as you normally would. Applications accessing
+PVFS2 through MPI-IO will bypass the kernel interface and talk to PVFS2 servers
+directly.
+
+Please note: older versions of MPICH2 need a few changes
+to the normal configure process. MPICH2-1.0.4p1 and older will need to
know the path to the PVFS2 installation. Modify the {\tt CFLAGS},
{\tt LDFLAGS} and {\tt LIBS} environment variables.
@@ -984,18 +993,7 @@ know the path to the PVFS2 installation.
prompt% export CFLAGS="<other desired flags> -I/usr/local/pvfs2/include"
prompt% export LDFLAGS="-L/usr/local/pvfs2/lib"
prompt% export LIBS="-lpvfs2 -lpthread"
-prompt%
+prompt% configure --with-file-system=ufs+nfs+testfs+pvfs2 [other flags]
\end{verbatim}
-
-The MPICH2 configure script needs a some additional arguments to build
-ROMIO correctly. The {\tt enable-romio} flag builds ROMIO and the
-{\tt with-file-system} flag tells ROMIO which file systems to support.
-
-\begin{verbatim}
-configure --enable-romio --with-file-system=ufs+nfs+pvfs2 [other flags]
-\end{verbatim}
-
-Now compile and install MPICH2 as you normally would. Applications accessing
-PVFS2 through MPI-IO will bypass the kernel interface and talk to PVFS2 servers directly.
\end{document}
Index: pvfs2-faq.tex
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/pvfs2-faq.tex,v
diff -p -u -r1.48.4.1 -r1.48.4.2
--- pvfs2-faq.tex 25 Sep 2006 12:40:20 -0000 1.48.4.1
+++ pvfs2-faq.tex 2 Dec 2006 11:12:40 -0000 1.48.4.2
@@ -65,6 +65,27 @@ The PVFS2 Guide (\url{http://www.pvfs.or
information on all of these components, plus a discussion of the system as a
whole, the code tree, and more.
+\subsection{What is the format of the PVFS version string?}
+\label{sec:version-string}
+PVFS uses a three-number version string: X.Y.Z. The first number (X)
+represents the high level design version of PVFS. The current design
+version is 2, and will likely remain there. The second number (Y) refers
+to the major version of the release. Major versions are incremented with
+new features, protocol changes, public API changes, and storage format
+changes. The third number (Z) refers to the minor version of the release,
+and is incremented primarily for bug fix releases.
+
+With our 2.6.0 release,
+we changed the release version and name from PVFS2 1.x.x, to PVFS 2.x.x.
+Users familiar with 'PVFS2' and had been using PVFS2 1.5.1
+will find the same software in PVFS version 2.6.0 or
+later (with updates and new features of course).
+
+Users of PVFS version 1 can still go to:
+\url{http://www.parl.clemson.edu/pvfs}, although we highly
+encourage you to upgrade to PVFS version 2, if you are still using
+version 1.
+
%
% SUPPORTED ARCHITECTURES
%
@@ -628,6 +649,23 @@ inlined kernel routines. PVFS2-1.3.2 or
\texttt{--disable-kernel-aio}. Passing this option to configure results in a pvfs2
kernel module that uses only exported symbols.
+\subsection{Everything built fine, but when I try to compile programs that use PVFS, I get undefined references}
+\label{sec:undefined_references}
+
+The \texttt{libpvfs2} library requires a few additional libraries. Usually
+"-lpthread -lcrypto -lssl" are required. Further, Myrinet and Infiniband have
+their own libraries. If you do not link the required libraries, you will
+probably get errors such as \texttt{undefined reference to `BIO\_f\_base64'}.
+
+The easiest and most portable way to ensure that you link in all required
+libraries when you link \texttt{libpvfs2} is to use the \texttt{pvfs2-config}
+utility. \texttt{pvfs2-config --libs} will give you the full set of linker
+flags needed. Here's an example of how one might use this tool:
+\begin{verbatim}
+$ gcc -c $(pvfs2-config --cflags) example.c
+$ gcc example.o -o example $(pvfs2-config --libs)
+\end{verbatim}
+
\subsection{Can we run the Apache webserver to serve files off a PVFS2 volume?}
Sure you can! However, we recommend that you turn off the EnableSendfile option in
@@ -645,6 +683,7 @@ the page-cache and thus there is a real
Therefore users of the {\tt sendfile} system call are warned to be wary of this detail.
\subsection{Trove-dbpf metadata format version mismatch!}
+\label{sec:trove-migration}
In PVFS2-1.5.0 or newer the format of the metadata storage has change from
previous versions (1.4.0 or earlier). This affects users that have created
@@ -777,6 +816,7 @@ reasonable choice.
\subsection{Is there any way to tune particular directories for different
workloads?}
+\label{sec:dir_tuning}
Yes. This can be done by using extended attributes to set directory
hints. Three hints are currently supported, and they allow you to specify
@@ -985,6 +1025,7 @@ Not at this time, no.
\subsection{Why does df show less free space than I think it should? What
can I do about that?}
+\label{sec:df-free-space}
PVFS2 uses a particular algorithm for calculating the free space on a file
system that takes the minimum amount of space free on a single server and
Index: module.mk.in
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/module.mk.in,v
diff -p -u -r1.5 -r1.5.40.1
--- module.mk.in 29 Jun 2004 20:18:45 -0000 1.5
+++ module.mk.in 2 Dec 2006 11:12:40 -0000 1.5.40.1
@@ -4,7 +4,8 @@ DOCSRC +=\
$(DIR)/pvfs2-guide.tex \
$(DIR)/pvfs2-faq.tex \
$(DIR)/pvfs2-ha.tex \
- $(DIR)/pvfs2-status.tex
+ $(DIR)/pvfs2-status.tex \
+ $(DIR)/pvfs2-tuning.tex
# some extra dependency information
doc/pvfs2-guide.pdf: doc/terms.tex doc/intro.tex doc/basics.tex doc/code-tree.tex doc/io-api-terms.tex
--- tuning.tex 2006-12-02 06:12:40.000000000 -0500
+++ /dev/null 2004-06-24 14:04:38.000000000 -0400
@@ -1,20 +0,0 @@
-\section{Tuning PVFS2}
-\label{sec:tuning}
-
-The default settings -- those provided by \texttt{pvfs2-genconfig} --
-will give correct behavior and decent performance. You can use more
-aggressive settings that will yield better performance, but you should
-understand the consequences of altering the defaults.
-
-\subsection{Sync Mode}
-
-The easiest way to see an improvement in performance is to set the
-\texttt{TroveSyncMeta} and \texttt{TroveSyncData} attributes to ``no''
-in the \texttt{<StorageHints>} section. If those attributes are set to
-``no'' then Trove will read and write data from a cache and not the
-underlying file. Performance will increase greatly, but if the server
-dies at some point, you could lose data. At this point in PVFS2
-development, server crashes are rare outside of hardware failures.
-PVFS2 developers should probably leave these settings to ``yes''. If
-PVFS2 hosts the only copy of your data, leave these settings to ``yes''.
-Otherwise, give ``no'' a shot.
More information about the Pvfs2-cvs
mailing list