[Pvfs2-users] 2.7.0 ping reports multiple roots

[Andrew Pochinsky]

Hi all,
	I'm having a problem with running 2.7.0 on multiple x64-64 servers.  
PVFS is build with CC='gcc -m64' on opensuse 10.3 running linux  
2.6.24.4, config is generated by pvfs2-genconfig. When pvfs2-ping is  
ran against this file system, it reports that the root handle is  
owned by multiple hosts. Other clients behave strange too: pvfs2- 
mkdir works, but pvfs2-cp does not. pvfs2-ls shows what one would  
expect after mkdir, pvfs2-fs-dump is not quite happy. Did anyone see  
such behavior before?
Thanks,
--andrew
P.S. Below are a config file, ping and fs-dump outputs. All clients  
are ran from the metadata server.

# cat pvfs2-fs-test2.conf
------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------
<Defaults>
         UnexpectedRequests 50
         EventLogging none
         LogStamp datetime
         BMIModules bmi_tcp
         FlowModules flowproto_multiqueue
         PerfUpdateInterval 1000
         ServerJobBMITimeoutSecs 30
         ServerJobFlowTimeoutSecs 30
         ClientJobBMITimeoutSecs 300
         ClientJobFlowTimeoutSecs 300
         ClientRetryLimit 5
         ClientRetryDelayMilliSecs 2000

         StorageSpace /storage/0/2
         LogFile /var/log/pvfs2-server2.log
</Defaults>

<Aliases>
         Alias 192.168.2.20 tcp://192.168.2.20:3402
         Alias 192.168.2.21 tcp://192.168.2.21:3402
         Alias 192.168.2.22 tcp://192.168.2.22:3402
</Aliases>

<Filesystem>
         Name pvfs2-fs-test2
         ID 1092537493
         RootHandle 1048576
         <MetaHandleRanges>
                 Range 192.168.2.20 3-2305843009213693953
         </MetaHandleRanges>
         <DataHandleRanges>
                 Range 192.168.2.20  
2305843009213693954-4611686018427387904
                 Range 192.168.2.21  
4611686018427387905-6917529027641081855
                 Range 192.168.2.22  
6917529027641081856-9223372036854775806
         </DataHandleRanges>
         <StorageHints>
                 TroveSyncMeta yes
                 TroveSyncData no
         </StorageHints>
</Filesystem>
------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------
pvfs2-ping output:
# PVFS2TAB_FILE=/dist/pvfs2/pvfs2tab2 pvfs2-ping -m /data/test2  
1>ping-out 2>&1
# cat ping-out
------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------

(1) Parsing tab file...

(2) Initializing system interface...

(3) Initializing each file system found in tab file: /dist/pvfs2/ 
pvfs2tab2...

    PVFS2 servers: tcp://192.168.2.20:3402
    Storage name: pvfs2-fs-test2
    Local mount point: /data/test2
    /data/test2: Ok
Failure: check root handle failed
PVFS_mgmt_setparam_all: Detailed per-server errors are available  
(error class: 0)

         Exactly one server must own the root handle.
         In this setup, multiple servers report ownership of root  
handle.

Per-server errors:
         Server: tcp://192.168.2.20:3402: Reports ownership of root  
handle
         Server: tcp://192.168.2.21:3402: Reports ownership of root  
handle
         Server: tcp://192.168.2.22:3402: Reports ownership of root  
handle

(4) Searching for /data/test2 in pvfstab...

    PVFS2 servers: tcp://192.168.2.20:3402
    Storage name: pvfs2-fs-test2
    Local mount point: /data/test2

    meta servers:
    tcp://192.168.2.20:3402

    data servers:
    tcp://192.168.2.20:3402
    tcp://192.168.2.21:3402
    tcp://192.168.2.22:3402

(5) Verifying that all servers are responding...

    meta servers:
    tcp://192.168.2.20:3402 Ok

    data servers:
    tcp://192.168.2.20:3402 Ok
    tcp://192.168.2.21:3402 Ok
    tcp://192.168.2.22:3402 Ok

(6) Verifying that fsid 1092537493 is acceptable to all servers...

    Ok; all servers understand fs_id 1092537493

(7) Verifying that root handle is owned by one server...

    Root handle: 1048576
=============================================================

The PVFS2 filesystem at /data/test2 appears to have problems.

------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------
Results of pvfs2-fs-dump:
------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------
# PVFS2TAB_FILE=/dist/pvfs2/pvfs2tab2 pvfs2-fs-dump -m /data/test2
Ugh! handle does not seem to be owned by the server!
File: <Root>
   handle = 1048576, type = Directory, server = 0
File: foo
   handle = 1048578, type = Directory, server = 0
File: bazz
   handle = 1048571, type = Directory, server = 0
File: two
   handle = 1048570, type = Directory, server = 0
File: foo-bar
   handle = 1048579, type = Directory, server = 0
File: lost+found
   handle = 1048577, type = Directory, server = 0
remaining handles:
pvfs-fs-dump: All handles acounted for!
------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------
Tab file:
------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------
  # cat pvfs2tab2
tcp://192.168.2.20:3402/pvfs2-fs-test2 /data/test2 pvfs2 rw 0 0
------------------------------------------------------------------------ 
-------------------------------------X8---X8---X8-----------------