[Pvfs2-cvs] commit by slang in pvfs2/src/client/sysint: sys-io.sm
CVS commit program
cvs at parl.clemson.edu
Fri Aug 11 15:18:04 EDT 2006
Update of /projects/cvsroot/pvfs2/src/client/sysint
In directory parlweb1:/tmp/cvs-serv22361/src/client/sysint
Modified Files:
sys-io.sm
Log Message:
[phil]: alternative lio_listio implementation that spawns a thread and does pwrite/pread.
[phil]: bmi-socket-close: This fixes a bug in the new BMI_set_info(... BMI_TCP_CLOSE_SOCKET ...) mechanism, which is used to reconnect the socket to the initial configuration server if new socket buffer sizes are specified in the config file. I didn't follow the code path find the exact problem, but at a high level it wasn't being thorough enough in cleaning out the old socket. This showed up when using epoll and specifying socket buffer sizes in the server configuration- in this case the client will often fail to mount with a cryptic "not a directory" error and leave some epoll() errors in the pvfs2-client.log file. I think a stale (or possibly reused) file descriptor was being left in the epoll fd set. At any rate, the fix is to use a different set of functions for tearing down the entire address etc. so that it is reconnected on the next BMI addr lookup. This path is already used by the server to discard old BMI addresses after critical errors on addresses that cannot be reconnected. It is triggered from bmi.c without entering the bmi_tcp module, so this patch also adds a check to make sure we don't bother for non-tcp methods.
[phil]: bmi-test-overflow: One of the bmi bandwidth test programs was using types that might overflow if testing large enough transfers. The fix is to convert to doubles and drop in several type casts to be cautious when performing the computation that was causing trouble.
[phil]: cancel-bugs: The biggest fix here is a change to the job timer code. It was performing some pointer operations in the wrong order, which could lead to job timers failing to trigger in some cases. This would prevent some operations from ever timing out. A secondary fix is a minor cleanup in BMI to catch potential race conditions in cancellation where a lock wasn't being held while checking to see if the target operation is complete.
[phil]: flow-post-error: This patch adds checks in the client side I/O state machine to test for failure at post time for flow operations. This type of error is uncommon unless the flow parameters are faulty, but it should have checked anyway to be safe.
[phil]: dfile-config-check: This is a safety test. The problem here is that there was no bounds checking for the DefaultNumDFiles option in the config file. This made it possible to select -1 (which in PVFS1 meant "use the default number"). In PVFS2 this number gets passed verbatim to the client and would cause malloc failures and various other odd results when used. The patch just checks at parse time to make sure the value isn't negative.
[phil]: bmi-flow-logging: This patch just adds a little bit more gossip logging output to BMI and the flow protocol. In particular, it adds tag information to messages about BMI operations, and adds pointer information to the flow protocol to help match posts and completions of flows in log files.
[phil]: server-freopen: This patch just removes duplicate calls to fdreopen(stdin/stdout/stderr) that were being performed on server startup when the server is run in the background. The first set should be sufficient.
Index: sys-io.sm
===================================================================
RCS file: /projects/cvsroot/pvfs2/src/client/sysint/sys-io.sm,v
diff -p -u -r1.146 -r1.147
--- sys-io.sm 16 Jun 2006 17:22:47 -0000 1.146
+++ sys-io.sm 11 Aug 2006 19:18:04 -0000 1.147
@@ -1028,12 +1028,30 @@ static int io_datafile_complete_operatio
sm_p->u.io.flow_completion_count--;
assert(sm_p->u.io.flow_completion_count > -1);
- /* if error, restart; but if this is a write, let write ack catch */
+ /* look for flow error when no write ack is in progress (usually a
+ * read case)
+ */
if (js_p->error_code < 0 && !cur_ctx->write_ack_in_progress) {
- gossip_debug(GOSSIP_IO_DEBUG,
- "%s: flow failed, retrying from msgpair\n", __func__);
- cur_ctx->msg_send_has_been_posted = 0;
- cur_ctx->msg_recv_has_been_posted = 0;
+ if ((PVFS_ERROR_CLASS(-js_p->error_code) == PVFS_ERROR_BMI) ||
+ (PVFS_ERROR_CLASS(-js_p->error_code) == PVFS_ERROR_FLOW) ||
+ (js_p->error_code == -ECONNRESET) ||
+ (js_p->error_code == -PVFS_EPROTO))
+ {
+ /* if this is a an error that we can retry */
+ gossip_debug(GOSSIP_IO_DEBUG,
+ "%s: flow failed, retrying from msgpair\n", __func__);
+ cur_ctx->msg_send_has_been_posted = 0;
+ cur_ctx->msg_recv_has_been_posted = 0;
+ }
+ else
+ {
+ /* do not retry on remaining error codes */
+ gossip_debug(GOSSIP_IO_DEBUG,
+ "%s: flow failed, not retrying\n", __func__);
+
+ /* forcing the count high insures that the sm won't restart */
+ sm_p->u.io.retry_count = sm_p->msgarray_params.retry_limit;
+ }
}
}
else if (STATUS_USER_TAG_TYPE(status_user_tag, IO_SM_PHASE_FINAL_ACK))
@@ -1711,26 +1729,46 @@ static inline int io_post_flow(
server_config->client_job_flow_timeout);
PINT_put_server_config_struct(server_config);
- if (ret < 0)
- {
- gossip_err("job_flow failed\n");
- return ret;
- }
- else if (ret == 1)
+ /* if the flow fails immediately, then we have to do some special
+ * handling. This function is not equiped to handle the failure
+ * directly, so we instead post a null job that will propigate the error
+ * to the normal state where we interpret flow errors
+ */
+ if((ret < 0) || (ret == 1 && cur_ctx->flow_status.error_code != 0))
{
- gossip_debug(GOSSIP_IO_DEBUG, " flow for context %p "
- "completed immediately\n", cur_ctx);
- assert(cur_ctx->flow_status.error_code == 0);
+ /* make sure the error code is stored in the flow descriptor */
+ if(ret == 1)
+ {
+ cur_ctx->flow_desc.error_code = cur_ctx->flow_status.error_code;
+ }
+ else
+ {
+ cur_ctx->flow_desc.error_code = ret;
+ }
+
+ gossip_debug(GOSSIP_IO_DEBUG, " immediate flow failure for "
+ "context %p, error code: %d\n", cur_ctx,
+ cur_ctx->flow_desc.error_code);
+ gossip_debug(GOSSIP_IO_DEBUG, " posting job_null() to propigate.\n");
+
+ /* post a fake job to propigate the flow failure to a later state */
+ ret = job_null(cur_ctx->flow_desc.error_code, sm_p,
+ status_user_tag, &cur_ctx->flow_status,
+ &cur_ctx->flow_job_id, pint_client_sm_context);
+ if(ret !=0)
+ {
+ return(ret);
+ }
}
else
{
gossip_debug(GOSSIP_IO_DEBUG, " posted flow for "
"context %p\n", cur_ctx);
-
- cur_ctx->flow_has_been_posted = 1;
- cur_ctx->flow_in_progress = 1;
- sm_p->u.io.flow_completion_count++;
}
+
+ cur_ctx->flow_has_been_posted = 1;
+ cur_ctx->flow_in_progress = 1;
+ sm_p->u.io.flow_completion_count++;
return 0;
}
More information about the Pvfs2-cvs
mailing list