[Pvfs2-cvs] commit by mtmoore in pvfs2/doc: db-recovery.txt
pvfs2-logging.txt REFERENCES.bib add-server-req pvfs2-faq.tex
pvfs2-quickstart.tex
CVS commit program
cvs at parl.clemson.edu
Thu Jul 8 10:02:38 EDT 2010
Update of /projects/cvsroot/pvfs2/doc
In directory parlweb1:/tmp/cvs-serv12057/pvfs2/doc
Modified Files:
Tag: mtmoore-meta-mirror
REFERENCES.bib add-server-req pvfs2-faq.tex
pvfs2-quickstart.tex
Added Files:
Tag: mtmoore-meta-mirror
db-recovery.txt pvfs2-logging.txt
Log Message:
merging Orange Branch changes in
--- /dev/null 2004-06-24 14:04:38.000000000 -0400
+++ db-recovery.txt 2010-07-08 10:02:38.000000000 -0400
@@ -0,0 +1,126 @@
+# 2009-11-5
+# Notes on recoverying corrupted Berkeley DB files in PVFS
+====================================================================
+
+The pvfs2-server daemon uses Berkeley DB as the mechanism for storing file
+system metadata. There are 5 database files in total that can be found in
+the following locations in the storage space:
+
+./storage_attributes.db
+./50a6d673/collection_attributes.db
+./50a6d673/dataspace_attributes.db
+./50a6d673/keyval.db
+./collections.db
+
+The dataspace_attributes.db and keyval.db are most frequently used by
+the file system. If one of these database files is corrupted for some
+reason, then it may prevent the file system from operating correctly.
+
+One common technique for repairing a Berkeley DB database is to dump its
+contents using db_dump (possibly found in the db4-utils package) and then
+reload it into a new .db file with db_load. However, both the keyval.db and
+dataspace_attributes.db use a custom function for sorting entries in order
+to improve PVFS performance. db_load must therefore be modified to use the
+correct key order.
+
+Here are the steps needed to build a db_load utility that will work on the
+keyval.db or dataspace_attributes.db file:
+
+- download the source code for Berkeley DB
+- edit db_load/db_load.c
+- find the section marked by #if 0 that indicates where to insert
+ application specific btree comparison or hash functions
+- insert the code listed at the end of this file (NOTE: there is different
+ code depending on which .db file you are trying to recover)
+- build Berkeley DB
+- rename db_load binary to db_load_pvfs_keyval to avoid confusion
+
+For keyval.db:
+====================================================================
+#include <stdint.h>
+
+typedef uint64_t PVFS_handle;
+typedef PVFS_handle TROVE_handle;
+
+#define PVFS_NAME_MAX 256
+#define DBPF_MAX_KEY_LENGTH PVFS_NAME_MAX
+
+struct dbpf_keyval_db_entry
+{
+ TROVE_handle handle;
+ char key[DBPF_MAX_KEY_LENGTH];
+};
+
+#define DBPF_KEYVAL_DB_ENTRY_TOTAL_SIZE(_size) \
+ (sizeof(TROVE_handle) + _size)
+
+#define DBPF_KEYVAL_DB_ENTRY_KEY_SIZE(_size) \
+ (_size - sizeof(TROVE_handle))
+
+int PINT_trove_dbpf_keyval_compare(
+ DB * dbp, const DBT * a, const DBT * b)
+{
+ const struct dbpf_keyval_db_entry * db_entry_a;
+ const struct dbpf_keyval_db_entry * db_entry_b;
+
+ db_entry_a = (const struct dbpf_keyval_db_entry *) a->data;
+ db_entry_b = (const struct dbpf_keyval_db_entry *) b->data;
+
+ if(db_entry_a->handle != db_entry_b->handle)
+ {
+ return (db_entry_a->handle < db_entry_b->handle) ? -1 : 1;
+ }
+
+ if(a->size > b->size)
+ {
+ return 1;
+ }
+
+ if(a->size < b->size)
+ {
+ return -1;
+ }
+
+ /* must be equal */
+ return (memcmp(db_entry_a->key, db_entry_b->key,
+ DBPF_KEYVAL_DB_ENTRY_KEY_SIZE(a->size)));
+}
+
+if ((ret = dbp->set_bt_compare(dbp, PINT_trove_dbpf_keyval_compare)) != 0)
+
+ dbp->err(dbp, ret, "DB->set_bt_compare");
+ goto err;
+}
+
+====================================================================
+
+For dataspace_attributes.db:
+====================================================================
+
+#include <stdint.h>
+
+typedef uint64_t PVFS_handle;
+typedef PVFS_handle TROVE_handle;
+
+int PINT_trove_dbpf_ds_attr_compare(DB * dbp, const DBT * a, const DBT * b)
+{
+ const TROVE_handle * handle_a;
+ const TROVE_handle * handle_b;
+
+ handle_a = (const TROVE_handle *) a->data;
+ handle_b = (const TROVE_handle *) b->data;
+
+ if(*handle_a == *handle_b)
+ {
+ return 0;
+ }
+
+ return (*handle_a > *handle_b) ? -1 : 1;
+}
+
+if ((ret = dbp->set_bt_compare(dbp, PINT_trove_dbpf_ds_attr_compare)) != 0) {
+ dbp->err(dbp, ret, "DB->set_bt_compare");
+ goto err;
+}
+
+====================================================================
--- /dev/null 2004-06-24 14:04:38.000000000 -0400
+++ pvfs2-logging.txt 2010-07-08 10:02:38.000000000 -0400
@@ -0,0 +1,191 @@
+PVFS Logging
+============
+
+This document describes log files produced by PVFS and how to control
+what messages are included in them.
+
+PVFS Log Format
+---------------
+
+The log messages from all PVFS components except for the kernel module
+are in the following format:
+
+ [<type> <timestamp>] LOG MESSAGE
+
+The type will be one of 4 different letters depending on what type of
+log message it is:
+
+ D - DEBUG
+ E - ERROR
+ A - ACCESS LOGGING
+ P - PERFORMANCE COUNTER
+
+The timestamp defaults to showing the date, hour, and minute that the
+log message was generated. The timestamp format can be modified to one
+of the following styles by using the --logstamp argument to pvfs2-client
+or the LogStamp field in the pvfs2-server config file:
+
+ datetime: (default, as described above)
+ usec: shows time with microsecond granularity (but no date)
+ none: no time stamp
+ thread: includes thread ID with each message
+
+PVFS Log Locations
+------------------
+
+The pvfs2-server daemon writes log messages to /tmp/pvfs2-server.log by
+default. A different output file can be specified using the LogFile
+parameter in the configuration file. The logs can also be sent to syslog
+instead by adding "LogType syslog" to the configuration file.
+
+The pvfs2-client daemon writes log messages to /tmp/pvfs2-client.log by
+default. This can be overridden using the --logfile or --logtype command
+line arguments to pvfs2-client.
+
+The PVFS kernel module (pvfs2.ko) generates log messages to dmesg and/or
+/var/log/messages and/or /var/log/kern depending on your Linux distribution.
+
+The PVFS client library (libpvfs2) and command line utilities generate log
+messages to stderr if enabled.
+
+Logging Levels
+--------------
+
+All PVFS components log critical error messages automatically. However,
+you can also turn on additional logging for debugging purposes. This is
+controlled by specifying which logging "masks" should be enabled.
+
+You can see a list of available pvfs2-server, client library, and
+pvfs2-client logging levels by running the pvfs2-set-debugmask utility
+with no arguments.
+
+You can see a list of available kernel module logging masks and client
+logging masks by running "cat /proc/sys/pvfs2/debug-help".
+
+The "verbose" mask is commonly used to turn on most of the logging
+levels that are useful for debugging problems.
+
+Changing the log mask for pvfs2-server
+--------------------------------------
+
+Use the EventLogging field in the configuration file to specify
+a comma separated list of masks to enable. You can also use the
+pvfs2-set-debugmask command line utility to change the mask dynamically
+without restarting the server.
+
+Changing the log mask for libpvfs and command line utilities
+------------------------------------------------------------
+
+Set the PVFS2_DEBUGMASK environment variable to a comma separated list of
+client-appropriate masks prior to launching the application.
+
+
+Changing the log mask for the kernel module
+-------------------------------------------
+
+There are three ways to set the debugging level for the kernel module:
+
+1. Set module_parm_debug_mask parameter when the kernel module is
+loaded.
+
+2. Set the environment variable PVFS2_KMODMASK before starting the
+pvfs2-client. NOTE: the kernel module must be loaded before starting
+the client-core.
+
+3. Write a debug string to /proc/sys/pvfs2/kernel-debug after the kernel
+module is loaded.
+
+
+Options 1 and 2 allow the kernel debug mask to be set ONLY when PVFS is started,
+while option 3 allows the kernel debug mask to be modified while PVFS is
+running. Thus, option 3 dynamically updates the kernel debug mask,
+immediately turning on the debugging options specified, and REPLACES the
+existing debug mask. Whenever you modify the kernel debug mask using
+option 3, an informational message is printed to the system log file,
+giving both its numerical value and a comma-separated list of keywords representing
+the areas of debugging just turned on.
+
+Options 1 and 2 require the user to specify a numerical value that is
+an OR'd list of gossip debug values. These values can be found in
+include/pvfs2-debug.h. For example, to load the kernel module with
+"file" debugging turned on, issue the following command:
+
+insmod pvfs2.ko module_parm_debug_mask=4
+
+The 4 is the value of GOSSIP_FILE_DEBUG, and module_parm_debug_mask is the kernel
+module's input parameter for the kernel debug mask. To turn on multiple areas,for
+example, file and dcache, set module_parm_debug_mask = (GOSSIP_FILE_DEBUG | GOSSIP_DCACHE_DEBUG) =
+(4 | 128) = 132. Its string equivalent would be "file,dcache".
+
+An informational message is displayed in the system log whenever you load the kernel
+module, giving you the kernel debug mask's numerical value and its string
+equivalent. Be aware that you can modify this value later using option 3.
+
+To set the kernel debug mask using PVFS2_KMODMASK, create a global environment
+variable and set it equal to the desired numerical value. When the pvfs2-client
+is started, the kernel debug mask and its string equivalent will be modified. Note that
+PVFS2_KMODMASK will override any value set by the kernel module load process. Again,
+option 3 allows you to change the debug mask at any time.
+
+To set the kernel debug mask using the /proc variable, write a debug string to
+/proc/sys/pvfs2/kernel-debug. Example: echo "file,dcache" > /proc/sys/pvfs2/kernel-debug.
+An informational message will be written to the system log file displaying the new
+kernel debug mask and its string equivalent.
+
+To see the kernel debug mask without looking in the system log, issue a "cat" on
+/proc/sys/pvfs2/kernel-debug and you will see the string equivalent of the kernel
+debug mask.
+
+A helper /proc variable, /proc/sys/pvfs2/debug-help, will display a list of valid
+keywords for both the kernel and client debug masks, when you issue a "cat" on it. These
+keywords are used to build the string that represent the areas of debugging that you
+want turned on.
+
+
+
+Changing the log mask for the client module
+-------------------------------------------
+
+There are three ways to set the debugging level for the pvfs2-client:
+
+1. Set --gossip-mask=MASK_LIST on the command line when starting the client. This
+list can be overridden by PVFS2_DEBUGMASK or by setting the /proc variable client-debug.
+
+2. Write a debug string to /proc/sys/pvfs2/client-debug after starting the client. This
+will override any value set by --gossip-mask on the command line and any value set by
+PVFS2_DEBUGMASK.
+
+3. Set the environment variable PVFS2_DEBUGMASK before starting the client. This will
+override any value set by --gossip-mask on the command line.
+
+
+Options 1 and 2 require a string of comma-separated keywords to set the client debug mask.
+For example:
+
+./pvfs2-client --gossip-mask="server,trove" -p ./pvfs2-client-core
+NOTE: after kernel module is loaded and during client startup.
+
+echo "server,trove" > /proc/sys/pvfs2/client-debug
+NOTE: after kernel module is loaded and client is started.
+
+A list of client debug keywords can be found in include/pvfs2-debug.h or by accessing
+the /proc/sys/pvfs2/debug-help variable after the kernel module is loaded. Example:
+
+cat /proc/sys/pvfs2/debug-help
+
+
+When the client starts, the client debug mask information is sent to the kernel module
+where a local version of the mask and its string equivalent is maintained. This process
+writes an informational message to the system log file displaying the numerical value of
+the client debug mask and its string equivalent. You can also see the mask's current string
+equivalent by issuing the following:
+
+cat /proc/sys/pvfs2/client-debug.
+
+
+Whenever you modify the client debug mask after the client has started, an informational message will
+be written to the system log file displaying the new numerical value and string equivalent. At any
+time, once PVFS is running, you can view the client debug mask using the "cat" statement above without
+having to look in the system log file for the last modification.
+
+
Index: REFERENCES.bib
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/REFERENCES.bib,v
diff -p -u -r1.16 -r1.16.32.1
--- REFERENCES.bib 5 Feb 2009 13:04:21 -0000 1.16
+++ REFERENCES.bib 8 Jul 2010 14:02:38 -0000 1.16.32.1
@@ -1,3 +1,72 @@
+ at PhdThesis{bradles-diss,
+ author = {Bradley W. Settlemyer},
+ title = {A Study of Client-based Caching for Parallel {I/O}},
+ school = {Clemson University},
+ address = {Clemson, SC},
+ year = {2009},
+ month = {August},
+}
+
+ at conference{hadoop-pvfs,
+ title={In Search of an {API} for Scalable File Systems: Under the table or
+ above it?},
+ author={Swapnil Patil and Garth A. Gibson and Gregory R. Ganger and
+ Julio Lopez and Milo Polte and Wittawat Tantisiroj and Lin Xiao},
+ booktitle={USENIX HotCloud Workshop 2009},
+ month={June},
+ year={2009}
+}
+
+ at conference{syscall-diag,
+ title={System-Call Based Problem Diagnosis for {PVFS}},
+ author={Michael P. Kasick and Keith A. Bare and Eugene E. Marinelli III
+ and Jiaqi Tan and Rajeev Gandhi and Priya Narasimhan},
+ booktitle={Proceedings of the 5th Workshop on Hot Topics in System
+ Dependability (HotDep '09)},
+ month={June},
+ year={2009}
+}
+
+ at conference{coord-access-toappear-iasds09,
+ title={Interfaces for Coordinated Access in the File System},
+ author={Sam Lang and Robert Latham and Dries Kimpe and Robert Ross},
+ booktitle={Proceedings of 2009 Workshop on Interfaces and
+ Architectures for Scientific Data Storage},
+ month={September},
+ year={2009},
+ note={(\textbf{To Appear})}
+}
+
+ at article{dyn-fs-semantics,
+ author = {Michael Kuhn and Julian Martin Kunkel and Thomas Ludwig},
+ title = {Dynamic file system semantics to enable metadata optimizations
+ in {PVFS}},
+ journal ={Concurrency and Computation: Practice and Experience},
+ year = {2009},
+ publisher = {John Wiley and Sons, Ltd},
+}
+
+ at conference{pvfs-bgp-toappear-sc09,
+ title={{I/O} Performance Challenges at Leadership Scale},
+ author={Samuel Lang and Philip Carns and Robert Latham and Robert Ross and Kevin Harms and William Allcock},
+ booktitle={Proceedings of Supercomputing},
+ month={November},
+ year={2009},
+ note={(\textbf{To Appear})}
+}
+
+ at article{IPDPS.2009.5161070,
+ author = {Xuechen Zhang and Song Jiang and Kei Davis},
+ title = {Making resonance a common case: A high-performance implementation
+ of collective I/O on parallel file systems},
+ journal ={Parallel and Distributed Processing Symposium, International},
+ volume = {0},
+ year = {2009},
+ pages = {1-12},
+ doi = {http://doi.ieeecomputersociety.org/10.1109/IPDPS.2009.5161070},
+ publisher = {IEEE Computer Society},
+ address = {Los Alamitos, CA, USA},
+}
@inproceedings{SmallFilesIPDPS09,
author = {Philip Carns and Sam Lang and Robert Ross Murali Vilayannur Julian Kunkel and Thomas Ludwig},
@@ -9,7 +78,7 @@
@inproceedings{PosixExtTReport,
author = {M. Vilayannur and S. Lang and R. Ross and R. Klundt and L. Ward},
- title = {Extending the POSIX I/O Interface: A Parallel File System Perspective},
+ title = {Extending the {POSIX I/O} Interface: A Parallel File System Perspective},
booktitle = {Technical Memorandum ANL/MCS-TM-302},
year = {2008},
month = {October},
@@ -36,7 +105,7 @@
@inproceedings{OSDCluster08,
author = {N. Ali and A. Devulapalli and D. Dalessandro and P. Wyckoff and P. Sadayappan},
- title = {An OSD-based Approach to Managing Directory Operations in Parallel File Systems},
+ title = {An {OSD-based} Approach to Managing Directory Operations in Parallel File Systems},
booktitle = {IEEE International Conference on Cluster Computing},
year = {2008},
month = {September},
@@ -72,6 +141,23 @@
address = {Berlin, Heidelberg},
}
+ at inproceedings{kunkel:bottlenecks,
+ author={Julian Kunkel and Thomas Ludwig},
+ title={Bottleneck Detection in Parallel File Systems with Trace-Based Performance Monitoring},
+ booktitle={Euro-Par 2008: Proceedings of the 14th International Euro-Par Conference on Parallel Processing},
+ year = {2008},
+ pages={212--221}
+}
+
+ at inproceedings{ludwig:pioviz,
+ author={Thomas Ludwig and Stephan Krempel and Michael Kuhn and Julian Kunkel and Christian Lohse},
+ title={Analysis of the {MPI-IO} Optimization Levels with the {PIOViz} Jumpshot Enhancement},
+ booktitle={Proceedings of EuroPVM/MPI 2007},
+ year={2007},
+ month={September},
+ pages={213--222}
+}
+
@article{BradThesis,
author = {Bradley W. Settlemyer},
title = {A Mechanism for Scalable Redundancy in Parallel File Systems},
@@ -90,7 +176,6 @@
year = {2007},
pages = {26-29},
ee = {http://doi.acm.org/10.1145/1374596.1374604},
- crossref = {DBLP:conf/sc/2007pdsw},
bibsource = {DBLP, http://dblp.uni-trier.de},
url = {http://www.pdl.cmu.edu/PDL-FTP/HECStorage/sc07-patil.pdf}
}
@@ -620,7 +705,6 @@
title = {Implementation of an {MPI-I/O} Mechanism Using {PVFS} in Remote I/O to a PC Cluster},
booktitle = {HPCASIA '04: Proceedings of the High Performance Computing and Grid in Asia Pacific Region, Seventh International Conference},
year = {2004},
- isbn = {0-7695-2138-X},
pages = {136--139},
publisher = {IEEE Computer Society},
address = {Washington, DC, USA},
Index: add-server-req
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/add-server-req,v
diff -p -u -r1.3 -r1.3.94.1
--- add-server-req 6 Oct 2005 13:43:03 -0000 1.3
+++ add-server-req 8 Jul 2010 14:02:38 -0000 1.3.94.1
@@ -37,24 +37,14 @@ Steps in adding a request to the server:
src/server/pvfs2-server.h
PINT_server_op -- only if required
-5) write state machine -- se details below
+5) write state machine -- see details below
src/server/<reqname>.sm
each state machine added must have a delcaration in
src/server/pvfs2-server.h
-6) update request scheduler
- src/server/request-scheduler/request-scheduler.c
- PINT_req_sched_target_handle()
-
-7) add entry to server operation parameters table
- src/server/pvfs2-server.c
- init_req_table -- see src/server/prelude.sm
- prelude_perm_check()
-
-8) add entry in final response state machine
- src/server/final-response.sm
- s_req_resp_type_map
-
+6) update src/server/pvfs2-server-req.c
+ add new entry to PINT_server_req_table[]
+ reference the params structure from the state machine
Writing State Machines
Index: pvfs2-faq.tex
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/pvfs2-faq.tex,v
diff -p -u -r1.58 -r1.58.32.1
--- pvfs2-faq.tex 26 Feb 2009 19:42:48 -0000 1.58
+++ pvfs2-faq.tex 8 Jul 2010 14:02:38 -0000 1.58.32.1
@@ -318,6 +318,9 @@ has happened when you have a problem.
The best place to look for documentation on PVFS is the PVFS web site at
\url{http://www.pvfs.org/}. Documentation (including this FAQ) is also
available in the \texttt{doc} subdirectory of the PVFS source distribution.
+Please reference \texttt{pvfs2-logging.txt} to understand more about PVFS'
+informational messages, where the logs exist, and how to turn logging
+on and off.
\subsection{What should I do if I have a problem?}
@@ -712,6 +715,17 @@ If you are using a pre-release kernel (a
stand a good chance of running into problems. We are unable to track every
pre-release kernel, but do make an effort to publish necessary patches once a
kernel is officially released.
+
+\subsection{Does PVFS work with Open-MX?}
+\label{sec:open-mx}
+
+Yes, PVFS does work with Open-MX. To use Open-MX, configure PVFS with
+the the same arguments that you would use for a normal MX installation:
+``--disable-bmi-tcp'' and ``--with-mx=PATH''. In addition, however, you
+must set the ``MX\_IMM\_ACK'' environment variable to ``1'' before starting
+the pvfs2-server or pvfs2-client daemons. This is necessary in order to
+account for differences in how MX and Open-MX handle message progression by
+default.
%
% PERFORMANCE
Index: pvfs2-quickstart.tex
===================================================================
RCS file: /projects/cvsroot/pvfs2/doc/pvfs2-quickstart.tex,v
diff -p -u -r1.46 -r1.46.44.1
--- pvfs2-quickstart.tex 25 Feb 2008 14:14:57 -0000 1.46
+++ pvfs2-quickstart.tex 8 Jul 2010 14:02:38 -0000 1.46.44.1
@@ -938,8 +938,10 @@ to ``file, inode'' prior to starting pvf
verbose kmod subsystem error diagnostics are written to the system ring buffer
and eventually to the kernel logs.
One could also set the kmod diagnostic level when the kernel module is loaded
-like so, insmod pvfs2.ko gossip\_debug\_mask=<diagnostic level>.
+like so, insmod pvfs2.ko module\_parm\_debug\_mask=<diagnostic level>.
The diagnostic level will be a bitwise OR of values specified in pvfs2-debug.h.
+For more information on setting the kernel or client debug mask, see
+\texttt{doc/pvfs2-logging.txt} in the PVFS source tree.
\section{ROMIO Support}
\label{sec:romio}
More information about the Pvfs2-cvs
mailing list