[Pvfs2-cvs] commit by slang in pvfs2/src/io/bmi: bmi.c

[CVS commit program]

Update of /projects/cvsroot/pvfs2/src/io/bmi
In directory parlweb1:/tmp/cvs-serv22361/src/io/bmi

Modified Files:
	bmi.c 
Log Message:
[phil]: alternative lio_listio implementation that spawns a thread and does pwrite/pread.

[phil]: bmi-socket-close: This fixes a bug in the new BMI_set_info(... BMI_TCP_CLOSE_SOCKET ...) mechanism, which is used to reconnect the socket to the initial configuration server if new socket buffer sizes are specified in the config file.  I didn't follow the code path find the exact problem, but at a high level it wasn't being thorough enough in cleaning out the old socket.  This showed up when using epoll and specifying socket buffer sizes in the server configuration- in this case the client will often fail to mount with a cryptic "not a directory" error and leave some epoll() errors in the pvfs2-client.log file.  I think a stale (or possibly reused) file descriptor was being left in the epoll fd set.  At any rate, the fix is to use a different set of functions for tearing down the entire address etc. so that it is reconnected on the next BMI addr lookup.  This path is already used by the server to discard old BMI addresses after critical errors on addresses that cannot be reconnected.  It is triggered from bmi.c without entering the bmi_tcp module, so this patch also adds a check to make sure we don't bother for non-tcp methods.

[phil]: bmi-test-overflow: One of the bmi bandwidth test programs was using types that might overflow if testing large enough transfers.  The fix is to convert to doubles and drop in several type casts to be cautious when performing the computation that was causing trouble.

[phil]: cancel-bugs: The biggest fix here is a change to the job timer code.  It was performing some pointer operations in the wrong order, which could lead to job timers failing to trigger in some cases. This would prevent some operations from ever timing out.  A secondary fix is a minor cleanup in BMI to catch potential race conditions in cancellation where a lock wasn't being held while checking to see if the target operation is complete.

[phil]: flow-post-error: This patch adds checks in the client side I/O state machine to test for failure at post time for flow operations.  This type of error is uncommon unless the flow parameters are faulty, but it should have checked anyway to be safe.

[phil]: dfile-config-check: This is a safety test.  The problem here is that there was no bounds checking for the DefaultNumDFiles option in the config file.  This made it possible to select -1 (which in PVFS1 meant "use the default number").  In PVFS2 this number gets passed verbatim to the client and would cause malloc failures and various other odd results when used. The patch just checks at parse time to make sure the value isn't negative.

[phil]: bmi-flow-logging: This patch just adds a little bit more gossip logging output to BMI and the flow protocol.  In particular, it adds tag information to messages about BMI operations, and adds pointer information to the flow protocol to help match posts and completions of flows in log files.

[phil]: server-freopen: This patch just removes duplicate calls to fdreopen(stdin/stdout/stderr) that were being performed on server startup when the server is run in the background.  The first set should be sufficient.



Index: bmi.c
===================================================================
RCS file: /projects/cvsroot/pvfs2/src/io/bmi/bmi.c,v
diff -p -u -r1.71 -r1.72
--- bmi.c	14 Jul 2006 20:46:00 -0000	1.71
+++ bmi.c	11 Aug 2006 19:18:05 -0000	1.72
@@ -516,8 +516,8 @@ int BMI_post_recv(bmi_op_id_t * id,
     int ret = -1;
 
     gossip_debug(GOSSIP_BMI_DEBUG_OFFSETS,
-                 "BMI_post_recv: addr: %ld, offset: 0x%lx, size: %ld\n",
-                 (long)src, (long)buffer, (long)expected_size);
+                 "BMI_post_recv: addr: %ld, offset: 0x%lx, size: %ld, tag: %d\n",
+                 (long)src, (long)buffer, (long)expected_size, (int)tag);
 
     *id = 0;
 
@@ -554,8 +554,8 @@ int BMI_post_send(bmi_op_id_t * id,
     int ret = -1;
 
     gossip_debug(GOSSIP_BMI_DEBUG_OFFSETS,
-                 "BMI_post_send: addr: %ld, offset: 0x%lx, size: %ld\n",
-                 (long)dest, (long)buffer, (long)size);
+                 "BMI_post_send: addr: %ld, offset: 0x%lx, size: %ld, tag: %d\n",
+                 (long)dest, (long)buffer, (long)size, (int)tag);
 
     *id = 0;
 
@@ -592,8 +592,8 @@ int BMI_post_sendunexpected(bmi_op_id_t 
     int ret = -1;
 
     gossip_debug(GOSSIP_BMI_DEBUG_OFFSETS,
-	"BMI_post_sendunexpected: addr: %ld, offset: 0x%lx, size: %ld\n", 
-	(long)dest, (long)buffer, (long)size);
+	"BMI_post_sendunexpected: addr: %ld, offset: 0x%lx, size: %ld, tag: %d\n", 
+	(long)dest, (long)buffer, (long)size, (int)tag);
 
     *id = 0;
 
@@ -1179,15 +1179,22 @@ int BMI_set_info(PVFS_BMI_addr_t addr,
 	return(0);
     }
 
-    /*
-     * Pass the TCP address structure as the parameter for this operation,
-     * holding the lock.
-     */
-    if (option == BMI_TCP_CLOSE_SOCKET) {
-        inout_parameter = tmp_ref->method_addr;
-        ret = tmp_ref->interface->BMI_meth_set_info(option, inout_parameter);
+    /* if the caller requests a TCP specific close socket action */
+    if (option == BMI_TCP_CLOSE_SOCKET) 
+    {
+        /* check to see if the address is in fact a tcp address */
+        if(strcmp(tmp_ref->interface->method_name, "bmi_tcp") == 0)
+        {
+            /* take the same action as in the BMI_DEC_ADDR_REF case to clean
+             * out the entire address structure and anything linked to it so 
+             * that the next addr_lookup starts from scratch
+             */
+	    gossip_debug(GOSSIP_BMI_DEBUG_CONTROL, "Closing bmi_tcp connection at caller's request.\n"); 
+            ref_list_rem(cur_ref_list, addr);
+            dealloc_ref_st(tmp_ref);
+        }
         gen_mutex_unlock(&ref_mutex);
-        return ret;
+        return 0;
     }
 
     gen_mutex_unlock(&ref_mutex);
@@ -1439,8 +1446,8 @@ int BMI_post_send_list(bmi_op_id_t * id,
     int i;
 
     gossip_debug(GOSSIP_BMI_DEBUG_OFFSETS,
-	"BMI_post_send_list: addr: %ld, count: %d, total_size: %ld\n", 
-	(long)dest, list_count, (long)total_size);
+	"BMI_post_send_list: addr: %ld, count: %d, total_size: %ld, tag: %d\n", 
+	(long)dest, list_count, (long)total_size, (int)tag);
 
     for(i=0; i<list_count; i++)
     {
@@ -1506,8 +1513,8 @@ int BMI_post_recv_list(bmi_op_id_t * id,
     int i;
 
     gossip_debug(GOSSIP_BMI_DEBUG_OFFSETS,
-	"BMI_post_recv_list: addr: %ld, count: %d, total_size: %ld\n", 
-	(long)src, list_count, (long)total_expected_size);
+	"BMI_post_recv_list: addr: %ld, count: %d, total_size: %ld, tag: %d\n", 
+	(long)src, list_count, (long)total_expected_size, (int)tag);
 
     for(i=0; i<list_count; i++)
     {
@@ -1572,8 +1579,8 @@ int BMI_post_sendunexpected_list(bmi_op_
 
     gossip_debug(GOSSIP_BMI_DEBUG_OFFSETS,
 	"BMI_post_sendunexpected_list: addr: %ld, count: %d, "
-                 "total_size: %ld\n",  (long)dest, list_count,
-                 (long)total_size);
+                 "total_size: %ld, tag: %d\n",  (long)dest, list_count,
+                 (long)total_size, (int)tag);
 
     for(i=0; i<list_count; i++)
     {
@@ -1627,6 +1634,14 @@ int BMI_cancel(bmi_op_id_t id, 
                  "%s: cancel id %llu\n", __func__, llu(id));
 
     target_op = id_gen_safe_lookup(id);
+    if(target_op == NULL)
+    {
+        /* if we can't find the operation, then assume it has already
+         * completed naturally.
+         */
+        return(0);
+    }
+
     assert(target_op->op_id == id);
 
     if(active_method_table[target_op->addr->method_type]->BMI_meth_cancel)