[PVFS-users] pvfs mounting problems with minimalistic kernel

[Kaveh Moallemi]

Hi Rob,

by running:   pvfs-ping -h 10.0.0.1 -p 3000 -f /pvfs-meta
I get get the following:

  Kernel does not support tcp: No such file or directory
  pvfs-ping: unable to connect to 10.0.0.1:3000.
  mgr (10.0.0.1:3000) is down.
  pvfs file system /pvfs-meta has issues.

mgr-ping also gives similar results. And if I run tcpdump on the server, I 
just get:

  tcpdump: listening on eth0

Nothing changes as the slave attepts to pvfs-ping or mount ..  but, if I 
ping (standard ping) the server from the slave node, tcp dump gives the 
following:

  tcpdump: listening on eth0
  11:14:50.013282 arp who-has node1 tell node2
  11:14:50.013305 arp reply node1 is-at 0:50:ba:d2:44:ec
  11:14:50.013372 node2 > node1: icmp: echo request (DF)
  11:14:50.013417 node1 > node2: icmp: echo reply
  11:14:51.013462 node2 > node1: icmp: echo request (DF)
  11:14:51.013492 node1 > node2: icmp: echo reply
  11:14:55.012741 arp who-has node2 tell node1
  11:14:55.012808 arp reply node2 is-at 0:2:2a:b6:48:72

Now with telnet from node2:

  telnet 10.0.0.1 3000 

gives the following on node1:

  11:40:12.482060 node2.32771 > node1.3000: S 2713888958:2713888958(0) win 5840 <mss 1460,sackOK,timestamp 381265 0,nop,wscale 0> (DF)
  11:40:12.482121 arp who-has node2 tell node1
  11:40:12.482185 arp reply node2 is-at 0:2:2a:b6:48:72
  11:40:12.482197 node1.3000 > node2.32771: S 3639396032:3639396032(0) ack 2713888959 win 5792 <mss 1460,sackOK,timestamp 42224644 381265,nop,wscale 0> (DF)
  11:40:12.482278 node2.32771 > node1.3000: . ack 1 win 5840 <nop,nop,timestamp 381265 42224644> (DF)
  11:40:13.471598 node2.32771 > node1.3000: P 1:2(1) ack 1 win 5840 <nop,nop,timestamp 381364 42224644> (DF)
  11:40:13.471634 node1.3000 > node2.32771: . ack 2 win 5792 <nop,nop,timestamp 42224743 381364> (DF)
  11:40:13.878878 node2.32771 > node1.3000: P 2:3(1) ack 1 win 5840 <nop,nop,timestamp 381405 42224743> (DF)
  11:40:13.878909 node1.3000 > node2.32771: . ack 3 win 5792 <nop,nop,timestamp 42224784 381405> (DF)
  11:40:14.243800 node2.32771 > node1.3000: P 3:4(1) ack 1 win 5840 <nop,nop,timestamp 381442 42224784> (DF)
  11:40:14.243834 node1.3000 > node2.32771: . ack 4 win 5792 <nop,nop,timestamp 42224821 381442> (DF)
  11:41:14.262860 node1.3000 > node2.32771: F 1:1(0) ack 4 win 5792 <nop,nop,timestamp 42230823 381442> (DF)
  11:41:14.263114 node2.32771 > node1.3000: F 4:4(0) ack 2 win 5840 <nop,nop,timestamp 387443 42230823> (DF)
  11:41:14.263143 node1.3000 > node2.32771: . ack 5 win 5792 <nop,nop,timestamp 42230823 387443> (DF)

And, telnet 10.0.0.1 7000
gives the following on node1:

  tcpdump: listening on eth0
  11:33:16.819182 arp who-has node1 tell node2
  11:33:16.819223 arp reply node1 is-at 0:50:ba:d2:44:ec
  11:33:16.819290 node2.32769 > node1.afs3-fileserver: S 2284947310:2284947310(0) win 5840 <mss 1460,sackOK,timestamp 339704 0,nop,wscale 0> (DF)
  11:33:16.819340 node1.afs3-fileserver > node2.32769: S 3204858363:3204858363(0) ack 2284947311 win 5792 <mss 1460,sackOK,timestamp 42183078 339704,nop,wscale 0> (DF)
  11:33:16.819444 node2.32769 > node1.afs3-fileserver: . ack 1 win 5840 <nop,nop,timestamp 339704 42183078> (DF)
  11:33:25.730181 node2.32769 > node1.afs3-fileserver: P 1:2(1) ack 1 win 5840 <nop,nop,timestamp 340595 42183078> (DF)
  11:33:25.730226 node1.afs3-fileserver > node2.32769: . ack 2 win 5792 <nop,nop,timestamp 42183969 340595> (DF)
  11:35:10.797082 node1.afs3-fileserver > node2.32769: F 1:1(0) ack 2 win 5792 <nop,nop,timestamp 42194476 340595> (DF)
  11:35:10.797349 node2.32769 > node1.afs3-fileserver: F 2:2(0) ack 2 win 5840 <nop,nop,timestamp 351101 42194476> (DF)
  11:35:10.797377 node1.afs3-fileserver > node2.32769: . ack 3 win 5792 <nop,nop,timestamp 42194476 351101> (DF)

Interesting, why is port 7000 designated as "afs3-fileserver" (I don't 
have afs installed)?


Thank you Rob,

Kaveh

>From: Rob Ross <rross at mcs.anl.gov>
>To: Kaveh Moallemi <kmoallem at hotmail.com>
>CC: pvfs-users at beowulf-underground.org
>Subject: Re: [PVFS-users] pvfs mounting problems with minimalistic kernel
>Date: Sat, 15 May 2004 12:29:49 -0500 (CDT)
>
>Hi Kaveh,
>
>[ Pinging has been verified to work. ]
>
>I would suggest that you try putting the pvfs-ping utility out on one of
>the nodes that isn't working.  You can then run it with:
>
>   pvfs-ping -h 10.0.0.1 -p 3000 -f /pvfs-meta
>
>This will attempt to connect to the mgr (and if that succeeds, to the
>iods).  This program has somewhat more helpful error messages, so it might
>help us figure out what is going on.  It may also just print out "server
>not responding" though; we'll see.
>
>Do you have tcpdump on the server, and are you familiar with its usage?
>That might be a next step.
>
>Thanks,
>
>Rob
>