<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="City"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=blue style='word-wrap: break-word;-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space'>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>I ran the db_recovery tool, and by itself,
that didn’t fix enough to get the server running.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>I was going to make a backup copy of the
data directory before trying other things, but during the copy, got a bunch of
Input/Output errors. That triggered me to try an FSCK on the EXT3 file system I
was mounting. After that fsck (which found many problems), I tried starting the
server again, and it came back up. I in the process of seeing if all is there,
but it’s running now, and files still appear to be in place. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Thanks,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Sam Lang
[mailto:slang@mcs.anl.gov] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Wednesday, May 27, 2009 4:39
PM<br>
<b><span style='font-weight:bold'>To:</span></b> david.metheny@gmail.com<br>
<b><span style='font-weight:bold'>Cc:</span></b>
pvfs2-developers@beowulf-underground.org<br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Pvfs2-developers]
PVFS2 Error on startup</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>On May 27, 2009, at 4:28 PM, David Metheny wrote:<o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><br>
<br>
<o:p></o:p></span></font></p>
<span style='orphans: 2;text-align:auto;widows: 2;-webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0;word-spacing:0px'><u1:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="City"><u1:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place">
<div link=blue vlink=blue style='word-wrap: break-word;-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space'>
<div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>We’ve got db4-4.2.52-7.3.el4 installed.<u1:p></u1:p></span></font><font
color=black><span style='color:black'><o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><u1:p> </u1:p></span></font><font
color=black><span style='color:black'><o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>The value of ret after the line 650 call
is ‘-30988’.</span></font><font color=black><span style='color:black'><o:p></o:p></span></font></p>
</div>
</div>
</div>
</span></u1:smarttagtype></u1:smarttagtype>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>That's DB_PAGE_NOTFOUND. You should try to recover the dspace.db
file using db_recover.<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><a
href="http://www.oracle.com/technology/documentation/berkeley-db/db/utility/db_recover.html">http://www.oracle.com/technology/documentation/berkeley-db/db/utility/db_recover.html</a><o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>-sam<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><br>
<br>
<o:p></o:p></span></font></p>
<span style='orphans: 2;text-align:auto;widows: 2;-webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0;word-spacing:0px'><u1:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="City"><u1:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"><u1:p></u1:p><u1:p>
<div link=blue vlink=blue style='word-wrap: break-word;-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space'>
<div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> </u1:p></span></font><font
color=black><span style='color:black'><o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><u1:p> </u1:p></span></font><font
color=black><span style='color:black'><o:p></o:p></span></font></p>
</div>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
color=black face="Times New Roman"><span style='font-size:12.0pt;color:black'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<div>
<p class=MsoNormal><b><font size=2 color=black face=Tahoma><span
style='font-size:10.0pt;font-family:Tahoma;color:black;font-weight:bold'>From:</span></font></b><span
class=apple-converted-space><font size=2 color=black face=Tahoma><span
style='font-size:10.0pt;font-family:Tahoma;color:black'> </span></font></span><font
size=2 color=black face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma;
color:black'>Sam Lang [<a href="mailto:slang@mcs.anl.gov">mailto:slang@mcs.anl.gov</a>]<span
class=apple-converted-space> </span><br>
<b><span style='font-weight:bold'>Sent:</span></b><span
class=apple-converted-space> </span>Wednesday, May 27, 2009 11:27 AM<br>
<b><span style='font-weight:bold'>To:</span></b><span
class=apple-converted-space> </span><a
href="mailto:david.metheny@gmail.com">david.metheny@gmail.com</a><br>
<b><span style='font-weight:bold'>Cc:</span></b><span
class=apple-converted-space> </span><a
href="mailto:pvfs2-developers@beowulf-underground.org">pvfs2-developers@beowulf-underground.org</a><br>
<b><span style='font-weight:bold'>Subject:</span></b><span
class=apple-converted-space> </span>Re: [Pvfs2-developers] PVFS2 Error on
startup</span></font><font color=black><span style='color:black'><o:p></o:p></span></font></p>
<u1:p></u1:p></div>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u1:p> </u1:p><o:p></o:p></span></font></p>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u1:p> </u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>That's not a very useful error, but it
looks like there might be something wrong with your dspace db. Can you
run it in gdb, set a breakpoint at dbpf-dspace.c:650, step to the next line and
print the result of "ret"? Line 650 should have:<u1:p></u1:p><o:p></o:p></span></font></p>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>b dbpf-dspace.c:<u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'> ret = dbc_p->c_get(dbc_p,
&key, &data, DB_MULTIPLE_KEY|DB_NEXT);<u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u1:p> </u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>The ret value is getting translated to a
trove error code, except the BDB error value isn't one that trove recognizes,
so it just spits out an error code that says it doesn't know anything about the
error code.<u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u1:p> </u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>What version of<span
class=apple-converted-space> <st1:city u2:st="on"><st1:place u2:st="on"></span><st1:City
w:st="on"><st1:place w:st="on">berkeley</st1:place></st1:city></st1:place></st1:City><span
class=apple-converted-space> </span>DB are you using?<u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u1:p> </u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>-sam<u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u1:p> </u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>On May 27, 2009, at 9:45 AM, David Metheny
wrote:<u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><br>
<br>
<br>
<o:p></o:p></span></font></p>
</div>
<u1:p></u1:p><span style='orphans: 2;widows: 2;-webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:
0px'>
<div link=blue vlink=purple>
<div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>I’m getting the following error starting
up a PVFS2 server, running both IO and metadata<u3:p></u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'><u3:p> </u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>PVFS2: [D] PVFS2 Server version 2.6.2pre1-2009-05-12-185735
starting.<u3:p></u3:p></span></font><font color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>PVFS2: [E] c_get failed on iteration 3059<u3:p></u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>PVFS2: [E] Warning: non PVFS2 error code
(4243):<u3:p></u3:p></span></font><font color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>PVFS2: [E]
dbpf_dspace_iterate_handles_op_svc: Unknown error 4243<u3:p></u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>PVFS2: [E] Error adding handle range
805306369-1073741823,2952790009-3221225463 to filesystem pvfs2-fs<u3:p></u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>PVFS2: [E] Error: Could not initialize
server interfaces; aborting.<u3:p></u3:p></span></font><font color=black><span
style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>PVFS2: [E] Error: Could not initialize
server; aborting.<u3:p></u3:p></span></font><font color=black><span
style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'><u3:p> </u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>The server is configured to sync both
metadata and data. The server’s connected storage experienced a connection
problem, causing a ‘read-only file system’ error. After stopping and starting
the server, it throws the above errors each time.<u3:p></u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'><u3:p> </u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>I’m using the base 2.6 release, with many
patches applied. Any ideas on how to debug this, or recover from it?<u3:p></u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'><u3:p> </u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color=black face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:black'>Thanks,<u3:p></u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=2 color="#666666" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#666666'><u3:p> </u3:p></span></font><font
color=black><span style='color:black'><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u3:p> </u3:p><u1:p></u1:p><o:p></o:p></span></font></p>
</div>
</div>
</div>
<div>
<p class=MsoNormal><font size=1 color=black face=Helvetica><span
style='font-size:9.0pt;font-family:Helvetica;color:black'>_______________________________________________<br>
Pvfs2-developers mailing list<br>
<a href="mailto:Pvfs2-developers@beowulf-underground.org">Pvfs2-developers@beowulf-underground.org</a><br>
<a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><u1:p></u1:p></span></font><font
color=black><span style='color:black'><o:p></o:p></span></font></p>
</div>
</div>
</div>
</span>
<div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><u1:p> </u1:p><o:p></o:p></span></font></p>
</div>
</div>
</div>
</div>
</div>
</div>
</span></u1:smarttagtype></u1:smarttagtype>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>