[Pvfs2-users] re: no route to host problem with PVFS

Rob Ross rross at mcs.anl.gov
Wed Mar 5 14:06:34 EST 2008 

Hi Jian,

The "S" flag is the "syn" flag, and is only the first of three packets  
that must be exchanged for a TCP connection to be established. In both  
of the tcpdump logs that you sent, only these first packets are being  
sent. There should be a SYN+ACK packet coming back from the other  
host, then a packet from the first host with just the ACK flag (no S).

The kernel is responsible for this handshaking, not PVFS (this is all  
done as part of the socket connect() and accept() calls). So this kind  
of problem is not a bug in our code.

The logs make me think that you have a firewall rule that is  
interfering, perhaps something that is blocking outgoing SYN+ACK  
messages from port 3334 on host2.

Regards,

Rob

On Mar 5, 2008, at 12:25 PM, jian hu wrote:

> Hi, Rob,
>
> I got some packets via tcpdump on both host1 and host2. I conclude  
> that host1 tries to connect host2 and host2 can also see
> these request. But host2 never reply.
>
> I got this result according to the flag of the packet. The flag bit  
> is "S", which means host1 tries to connect to host2.
>
> The tcmpdump result is attached to this mail.
> I type in command "tcpdump tcp port 3334".
>
> Thanks.
>
> On Wed, Mar 5, 2008 at 10:00 AM, Rob Ross <rross at mcs.anl.gov> wrote:
> firewall blocking port 3334? if you're familiar with tcpdump or
> ethereal, you could use one of those tools to look at the network
> traffic.
>
> rob
>
> On Mar 5, 2008, at 9:26 AM, jian hu wrote:
>
> > Attached is the config file and fstab file in host1.
> >
> > I can ping each other from either host1 or host2. I can also ping
> > the hostname too cause I set this up in system-administration-
> > network in FC.
> >
> > Thank you.
> >
> > On Wed, Mar 5, 2008 at 7:53 AM, Rob Ross <rross at mcs.anl.gov> wrote:
> > Hi Jian,
> >
> > It appears that you have a misconfigured network. "No route to host"
> > is an IP networking error. I would try using UNIX tools like ping  
> and
> > ssh to ensure that you can connect from host2 to host1 and vice  
> versa.
> > You could try putting the IP address of the hosts in place of the
> > "host1" and "host2" in your config file, as another option.
> >
> > Also, it would be better if this discussion occurred on the pvfs2-
> > users mailing list. If are a member of the pvfs2-users list,  
> please CC
> > that mailing list in your reply. And please attach your /etc/fstab  
> and
> > PVFS configuration files, and note what version of PVFS you're
> > attempting to install.
> >
> > Thanks,
> >
> > Rob
> >
> > On Mar 4, 2008, at 11:45 PM, jian hu wrote:
> >
> > > Hi,
> > >
> > > I try to install pvfs2 in two nodes, host1 is metadata server and
> > > both of the two hosts are data servers.
> > >
> > > But when I use pvfs2-ping to test it, both host1 and host2 gave  
> the
> > > same error.
> > > The output is below. Could anyone tell me where I'm wrong.
> > >
> > > Thank you.
> > >
> > > [root at host2 ~]# pvfs2-ping -m /mnt/pvfs2
> > >
> > > (1) Parsing tab file...
> > >
> > > (2) Initializing system interface...
> > >
> > > (3) Initializing each file system found in tab file: /etc/fstab...
> > >
> > >    PVFS2 servers: tcp://host2:3334
> > >    Storage name: pvfs2-fs
> > >    Local mount point: /mnt/pvfs2
> > > [E 00:21:19.659722] msgpair failed, will retry: No route to host
> > > [E 00:21:21.666323] msgpair failed, will retry: No route to host
> > > [E 00:21:23.670558] msgpair failed, will retry: No route to host
> > > [E 00:21:25.676399] msgpair failed, will retry: No route to host
> > > [E 00:21:27.681516] msgpair failed, will retry: No route to host
> > > [E 00:21:29.686527] msgpair failed, will retry: No route to host
> > > [E 00:21:29.686667] *** msgpairarray_completion_fn: msgpair to
> > > server [UNKNOWN] failed: No route to host
> > > [E 00:21:29.686690] *** Out of retries.
> > >    /mnt/pvfs2: FAILURE!
> > >
> > > Failure: could not initialze at least one of the target file
> > systems.
> > >
> > > (4) Searching for /mnt/pvfs2 in pvfstab...
> > >
> > >    PVFS2 servers: tcp://host2:3334
> > >    Storage name: pvfs2-fs
> > >    Local mount point: /mnt/pvfs2
> > >
> > >    meta servers:
> > >    tcp://host1:3334
> > >
> > >    data servers:
> > >    tcp://host1:3334
> > >    tcp://host2:3334
> > >
> > > (5) Verifying that all servers are responding...
> > >
> > >    meta servers:
> > > [E 00:21:29.687620] msgpair failed, will retry: No route to host
> > > [E 00:21:29.687688] *** msgpairarray_completion_fn: msgpair to
> > > server tcp://host1:3334 failed: No route to host
> > > [E 00:21:29.687707] *** No retries requested.
> > >    tcp://host1:3334 FAILURE: PVFS_mgmt_noop failed for server:
> > tcp://
> > > host1:3334
> > > Failure: could not communicate with one of the servers.
> > >
> > > (6) Verifying that fsid 1584819384 is acceptable to all servers...
> > > [E 00:21:32.688942] msgpair failed, will retry: No route to host
> > > [E 00:21:34.690239] msgpair failed, will retry: No route to host
> > > [E 00:21:36.691466] msgpair failed, will retry: No route to host
> > > [E 00:21:38.692930] msgpair failed, will retry: No route to host
> > > [E 00:21:40.703077] msgpair failed, will retry: No route to host
> > > [E 00:21:42.708181] msgpair failed, will retry: No route to host
> > > [E 00:21:42.708247] *** msgpairarray_completion_fn: msgpair to
> > > server tcp://host1:3334 failed: No route to host
> > > [E 00:21:42.708268] *** Out of retries.
> > > PVFS_mgmt_setparam_all: Detailed per-server errors are available
> > > (error class: 0)
> > > Failure: not all servers accepted fsid 1584819384
> > >         Per-server errors:
> > >         Server: tcp://host1:3334: No route to host
> > >         Server: tcp://host2:3334: No route to host
> > >
> > > (7) Verifying that root handle is owned by one server...
> > >
> > >    Root handle: 1048576
> > > [E 00:21:42.709659] msgpair failed, will retry: No route to host
> > > [E 00:21:44.713420] msgpair failed, will retry: No route to host
> > > [E 00:21:49.717688] msgpair failed, will retry: No route to host
> > > [E 00:21:51.723536] msgpair failed, will retry: No route to host
> > > [E 00:21:56.729725] msgpair failed, will retry: No route to host
> > > [E 00:21:58.740788] msgpair failed, will retry: No route to host
> > > [E 00:21:58.740845] *** msgpairarray_completion_fn: msgpair to
> > > server tcp://host1:3334 failed: No route to host
> > > [E 00:21:58.740866] *** Out of retries.
> > > Failure: check root handle failed
> > > PVFS_mgmt_setparam_all: Detailed per-server errors are available
> > > (error class: 0)
> > >
> > >         Exactly one server must own the root handle.
> > >         In this setup, no servers own the root handle.
> > >
> > > Per-server errors:
> > > =============================================================
> > >
> > > The PVFS2 filesystem at /mnt/pvfs2 appears to have problems.
> > >
> > >
> > > --
> > > Jian Hu
> > > CSE
> > > University of Nebraska Lincoln
> > > _______________________________________________
> > > Pvfs-developers mailing list
> > > Pvfs-developers at beowulf-underground.org
> > > http://www.beowulf-underground.org/mailman/listinfo/pvfs- 
> developers
> >
> >
> >
> >
> > --
> > Jian Hu
> > CSE
> > University of Nebraska Lincoln <pvfs2-fs.conf><fstab>
>
>
>
>
> -- 
> Jian Hu
> CSE
> University of Nebraska Lincoln <host1><host2>

More information about the Pvfs2-users mailing list