<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1420977795;
mso-list-type:hybrid;
mso-list-template-ids:-53993238 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor=white lang=EN-US link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>I’ve run a couple tests just to get
an idea of what it might take to delete large data files (bstreams). <o:p></o:p></span></font></p>
<ul style='margin-top:0in' type=disc>
<li class=MsoNormal style='color:navy;mso-list:l0 level1 lfo1'><font size=2
color=navy face=Arial><span style='font-size:10.0pt;font-family:Arial'>From
an idle RHEL4 U6 i386 server, with local EXT3, it seems to take around 120
seconds to complete a delete of a 100GB file. <o:p></o:p></span></font></li>
<li class=MsoNormal style='color:navy;mso-list:l0 level1 lfo1'><font size=2
color=navy face=Arial><span style='font-size:10.0pt;font-family:Arial'>From
an idle RHEL4 U6 i386 server, with SAN attached storage, it seems to take
around [180] seconds to complete a removal of a 200GB file. <o:p></o:p></span></font></li>
</ul>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Some other things that I’ve noticed,
is that during a remove of large files like this, other metadata operations
(i.e ‘ls’), as well as I/O operations. These operations are also
starting to timeout and retry operations, and depending on the timeframe,
cancel as well. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>What are you thinking along the lines of
tuning the client side? I’m assuming this would require the remove to
complete at least once, either with a valid object delete, or recognizing the
object isn’t there anymore. Would this tuning also take into account that
the delete might take longer than the client timeout/retry settings?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>We’ve bounced ideas around in the
past about lazy deletes and such, allowing the delete to occur, but not hold up
other trove operations. Any suggestions here?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Rob Ross
[mailto:rross@mcs.anl.gov] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Thursday, October 02, 2008
3:51 PM<br>
<b><span style='font-weight:bold'>To:</span></b> david.metheny@gmail.com<br>
<b><span style='font-weight:bold'>Cc:</span></b> <<st1:PersonName w:st="on">pvfs2-developers@beowulf-underground.org</st1:PersonName>><br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Pvfs2-developers]
PVFS2 Removal of large files</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>Maybe removing the 2TByte
file takes longer than 30 seconds on ext3, so client times out. It would be
useful to know when the server first succeeds. Maybe some tuning on client side
to catch the case where on retry the objects aren't there?<o:p></o:p></span></font></p>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>-- Rob<o:p></o:p></span></font></p>
</div>
</div>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'><br>
On Oct 2, 2008, at 3:21 PM, "David Metheny" <<a
href="mailto:david.metheny@gmail.com">david.metheny@gmail.com</a>> wrote:<o:p></o:p></span></font></p>
</div>
<blockquote style='margin-top:5.0pt;margin-bottom:5.0pt' type=cite>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>I’m
seeing an issue when removing large files from a PVFS2 file system. My example
setup is a 12 node PVFS2 file system with 2.2TB EXT3 SAN mounts to each pvfs2
server. The server is configured for 30 second timeouts and 5 retries. We
really don’t want to change the timeout values and retries if possible.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>There is a
2TB file that exists. When the client tries to ‘rm’ the 2TB file,
the client basically goes through the 30 second timeout and exhausts the
retries and then reports back to the command line “Invalid
Argument”. From everything I can tell, the file *<b><span
style='font-weight:bold'>really</span></b>* gets deleted and doesn’t show
up in a directory listing. <u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>I’ve
included the client command line results and the log messages from the delete
below<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>bash-2.05b$
rm cmsdb_silo_mstr_20080606a<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>rm: cannot
remove `cmsdb_silo_mstr_20080606a': Invalid argument<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955100.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955103.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955106.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955109.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955112.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955115.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955118.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955121.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955124.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955127.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955130.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955133.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:35 clientNode1 PVFS2: [E] msgpair failed, will retry: Operation cancelled
(possibly due to timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:29:36 clientNode1 last message repeated 11 times.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><SKIPPING
REPEAT OF THE ABOVE 5 MORE TIMES><u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server1HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2 10:32:10
clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server2HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server3HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server4HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server5HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server6HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server7HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server8HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server9HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server10HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server11HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** msgpairarray_completion_fn: msgpair to
server tcp://server12HA:3334 failed: Operation cancelled (possibly due to
timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] *** Out of retries.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] Error: failed removing one or more datafiles
associated with the meta handle 1610612708<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] WARNING: PVFS_sys_remove() encountered an error
which may lead to inconsistent state: Operation cancelled (possibly due to timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 PVFS2: [E] WARNING: PVFS2 fsck (if available) may be
needed.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:10 clientNode1 kernel: pvfs2: warning: got error code without errno
equivalent: -1610612865.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:59 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955696.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:32:59 clientNode1 PVFS2: [E] msgpair failed, will retry: Operation cancelled
(possibly due to timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:33:29 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955732.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:33:29 clientNode1 PVFS2: [E] msgpair failed, will retry: Operation cancelled
(possibly due to timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:33:59 clientNode1 PVFS2: [E] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 192955766.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:33:59 clientNode1 PVFS2: [E] msgpair failed, will retry: Operation cancelled
(possibly due to timeout)<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:34:20 clientNode1 PVFS2: [E] Error: failed removing one or more datafiles
associated with the meta handle 1252698765<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:34:20 clientNode1 PVFS2: [E] WARNING: PVFS_sys_remove() encountered an error
which may lead to inconsistent state: No such file or directory<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Oct 2
10:34:20 clientNode1 PVFS2: [E] WARNING: PVFS2 fsck (if available) may be
needed.<u1:p></u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><u1:p> </u1:p></span></font><o:p></o:p></p>
</div>
</div>
</blockquote>
<blockquote style='margin-top:5.0pt;margin-bottom:5.0pt' type=cite>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>_______________________________________________<br>
Pvfs2-developers mailing list<br>
<a href="mailto:Pvfs2-developers@beowulf-underground.org">Pvfs2-developers@beowulf-underground.org</a><br>
<a href="http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers">http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers</a><o:p></o:p></span></font></p>
</div>
</blockquote>
</div>
</body>
</html>