[Pvfs2-developers] pvfs2-cli can't identify protocol
Phil Carns
carns at mcs.anl.gov
Mon Jan 12 09:40:01 EST 2009
Ah, Ok. I didn't realize that you were using infiniband. Can any IB
gurus on the list confirm if it is responsible for the extra "sock"
entries lsof?
You can always increase the number of available file descriptors in your
init script before starting pvfs2-client if you need to ("ulimit -n
100000", for example). That might be all you need to do as long as the
number of file descriptors isn't growing without bound.
-Phil
Kumar, Amit H. wrote:
> Hi Phil,
> I do see other open files, I just did not included it. Of what you listed I don't see anything related to IPV4. May be because I am mounting it over Infiniband. Here it is ...
> pvfs2-cli 14792 root cwd DIR 8,1 4096 196609 /root
> pvfs2-cli 14792 root rtd DIR 8,1 4096 2 /
> pvfs2-cli 14792 root txt REG 8,6 46624 5701636 /opt/pvfs2/sbin/pvfs2-client
> pvfs2-cli 14792 root mem REG 8,1 130304 229708 /lib64/ld-2.5.so
> pvfs2-cli 14792 root mem REG 8,1 1687464 229709 /lib64/libc-2.5.so
> pvfs2-cli 14792 root mem REG 8,1 23360 229710 /lib64/libdl-2.5.so
> pvfs2-cli 14792 root mem REG 8,1 141344 229714 /lib64/libpthread-2.5.so
> pvfs2-cli 14792 root mem REG 8,1 241006 2392128 /usr/lib64/libibverbs.so.1.0.0
> pvfs2-cli 14792 root 0r CHR 1,3 1520 /dev/null
> pvfs2-cli 14792 root 1w CHR 1,3 1520 /dev/null
> pvfs2-cli 14792 root 2w CHR 1,3 1520 /dev/null
> pvfs2-cli 14792 root 3w REG 8,5 43162 98311 /tmp/pvfs2-client.log (deleted)
> pvfs2-cli 14793 root cwd DIR 8,1 4096 196609 /root
> pvfs2-cli 14793 root rtd DIR 8,1 4096 2 /
> pvfs2-cli 14793 root txt REG 8,6 2722000 5701637 /opt/pvfs2/sbin/pvfs2-client-core
> pvfs2-cli 14793 root mem REG 8,1 130304 229708 /lib64/ld-2.5.so
> pvfs2-cli 14793 root mem REG 8,1 1687464 229709 /lib64/libc-2.5.so
> pvfs2-cli 14793 root mem REG 8,1 23360 229710 /lib64/libdl-2.5.so
> pvfs2-cli 14793 root mem REG 8,1 141344 229714 /lib64/libpthread-2.5.so
> pvfs2-cli 14793 root mem REG 8,1 241006 2392128 /usr/lib64/libibverbs.so.1.0.0
> pvfs2-cli 14793 root mem CHR 231,192 5658 /dev/infiniband/uverbs0
> pvfs2-cli 14793 root mem REG 8,1 156563 1222167 /usr/lib64/libmlx4-rdmav2.so
> pvfs2-cli 14793 root mem REG 8,1 173084 1222165 /usr/lib64/libmthca-rdmav2.so
> pvfs2-cli 14793 root mem REG 8,1 118406 1222169 /usr/lib64/libcxgb3-rdmav2.so
> pvfs2-cli 14793 root mem REG 8,1 69644 1222174 /usr/lib64/libipathverbs-rdmav2.so
> pvfs2-cli 14793 root mem REG 8,1 68419 1222172 /usr/lib64/libnes-rdmav2.so
> pvfs2-cli 14793 root mem REG 8,1 53880 229404 /lib64/libnss_files-2.5.so
> pvfs2-cli 14793 root 0r CHR 1,3 1520 /dev/null
> pvfs2-cli 14793 root 1w CHR 1,3 1520 /dev/null
> pvfs2-cli 14793 root 2w CHR 1,3 1520 /dev/null
> pvfs2-cli 14793 root 3w REG 8,5 43162 98311 /tmp/pvfs2-client.log (deleted)
> pvfs2-cli 14793 root 4w REG 8,5 43162 98311 /tmp/pvfs2-client.log (deleted)
> pvfs2-cli 14793 root 5u CHR 253,0 12918 /dev/pvfs2-req
> pvfs2-cli 14793 root 6u CHR 231,192 5658 /dev/infiniband/uverbs0
> pvfs2-cli 14793 root 7r DIR 0,20 0 5654 infinibandevent
> pvfs2-cli 14793 root 8r DIR 0,20 0 5654 infinibandevent
>
> Thank you,
> Amit
>
>> -----Original Message-----
>> From: Phil Carns [mailto:pcarns at gmail.com] On Behalf Of Phil Carns
>> Sent: Thursday, January 08, 2009 2:25 PM
>> To: Kumar, Amit H.
>> Cc: 'Rob Ross'; pvfs2-developers at beowulf-underground.org
>> Subject: Re: [Pvfs2-developers] pvfs2-cli can't identify protocol
>>
>> Hi Amit,
>>
>> In your lsof output, do you see any other types of open files from
>> pvfs2-client besides "sock"? The output that you are showing is
>> unusual. Normally everything that pvfs2-client has open will show up
>> as
>> IPV4, REG, CHR, or DIR.
>>
>> Are you using tcp for PVFS communication?
>>
>> -Phil
>>
>> Kumar, Amit H. wrote:
>>> Hi Rob,
>>> I am using the latest version available for download (pvfs2-v2.7.1)
>>> # netstat -tan
>>> Active Internet connections (servers and established)
>>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State
>>> tcp 0 0 0.0.0.0:2049 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 0.0.0.0:677 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 0.0.0.0:57447 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 127.0.0.1:199 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 0.0.0.0:8649 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 0.0.0.0:938 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 0.0.0.0:111 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 127.0.0.1:25 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 0.0.0.0:953 0.0.0.0:*
>> LISTEN
>>> tcp 0 0 127.0.0.1:51598 127.0.0.1:199
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.237:862
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.223:878
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.221:850
>> ESTABLISHED
>>> tcp 0 0 127.0.0.1:199 127.0.0.1:51598
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.207:675
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.235:949
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.233:677
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.227:708
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.205:1003
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.243:991
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.213:718
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.249:1023
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.204:814
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.232:776
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.248:896
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.240:916
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:718 172.25.24.100:2049
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.226:950
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.224:698
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.216:751
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.250:963
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.220:1009
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.206:995
>> ESTABLISHED
>>> tcp 0 0 172.25.24.251:2049 172.25.24.222:976
>> ESTABLISHED
>>> tcp 0 0 :::80 :::*
>> LISTEN
>>> tcp 0 0 :::22 :::*
>> LISTEN
>>> tcp 0 0 :::443 :::*
>> LISTEN
>>> tcp 0 0 :::1311 :::*
>> LISTEN
>>> tcp 0 0 ::ffff:172.25.24.251:22
>> ::ffff:172.25.24.210:43811 ESTABLISHED
>>> Thank you,
>>> Amit
>>>> -----Original Message-----
>>>> From: Rob Ross [mailto:rross at mcs.anl.gov]
>>>> Sent: Wednesday, January 07, 2009 5:03 PM
>>>> To: Kumar, Amit H.
>>>> Cc: pvfs2-developers at beowulf-underground.org
>>>> Subject: Re: [Pvfs2-developers] pvfs2-cli can't identify protocol
>>>>
>>>> Hi Amit,
>>>>
>>>> What version of PVFS is this?
>>>>
>>>> What does the output of netstat -tan look like?
>>>>
>>>> Thanks,
>>>>
>>>> Rob
>>>>
>>>> On Jan 7, 2009, at 2:03 PM, Kumar, Amit H. wrote:
>>>>
>>>>> Hello All,
>>>>> I am trying to understand the following output from "lsof".
>>>>> All/most of our compute nodes (pvfs2 clients) have the following
>>>>> output as reported by "lsof"
>>>>> Few of the pvfs2-client nodes have greater than 1024 open sockets
>>>>> for just the pvfs2-client process. Current ulimit for maximum
>> number
>>>>> of open files per process is set to 1024 on all of our computer
>>>>> nodes. I see this as a potential performance problem. I was
>>>>> wondering if any of you can help me interpret the output and fix
>> any
>>>>> issues that this could be causing.
>>>>> <lsof output>
>>>>> ...............
>>>>> pvfs2-cli 27278 root 121u sock
>>>>> 0,5 13574284 can't identify protocol
>>>>> pvfs2-cli 27278 root 122u sock
>>>>> 0,5 13574285 can't identify protocol
>>>>> pvfs2-cli 27278 root 123u sock
>>>>> 0,5 13574286 can't identify protocol
>>>>> pvfs2-cli 27278 root 124u sock
>>>>> 0,5 13574287 can't identify protocol
>>>>> pvfs2-cli 27278 root 125u sock
>>>>> 0,5 13574288 can't identify protocol
>>>>> pvfs2-cli 27278 root 126u sock
>>>>> 0,5 13574289 can't identify protocol
>>>>> pvfs2-cli 27278 root 127u sock
>>>>> 0,5 13574290 can't identify protocol
>>>>> pvfs2-cli 27278 root 128u sock
>>>>> 0,5 13574291 can't identify protocol
>>>>> pvfs2-cli 27278 root 129u sock
>>>>> 0,5 13574292 can't identify protocol
>>>>> pvfs2-cli 27278 root 130u sock
>>>>> 0,5 13574303 can't identify protocol
>>>>> pvfs2-cli 27278 root 131u sock
>>>>> 0,5 13574304 can't identify protocol
>>>>> pvfs2-cli 27278 root 132u sock
>>>>> 0,5 13574326 can't identify protocol
>>>>> pvfs2-cli 27278 root 133u sock
>>>>> 0,5 13574327 can't identify protocol
>>>>> pvfs2-cli 27278 root 134u sock
>>>>> 0,5 13574328 can't identify protocol
>>>>> pvfs2-cli 27278 root 135u sock
>>>>> 0,5 13574329 can't identify protocol
>>>>> pvfs2-cli 27278 root 136u sock
>>>>> 0,5 13574330 can't identify protocol
>>>>> pvfs2-cli 27278 root 137u sock
>>>>> 0,5 13574331 can't identify protocol
>>>>> pvfs2-cli 27278 root 138u sock
>>>>> 0,5 13574332 can't identify protocol
>>>>> pvfs2-cli 27278 root 139u sock
>>>>> 0,5 13574333 can't identify protocol
>>>>> pvfs2-cli 27278 root 140u sock
>>>>> 0,5 13574334 can't identify protocol
>>>>> pvfs2-cli 27278 root 141u sock
>>>>> 0,5 13574336 can't identify protocol
>>>>> pvfs2-cli 27278 root 142u sock
>>>>> 0,5 13574337 can't identify protocol
>>>>> pvfs2-cli 27278 root 143u sock
>>>>> 0,5 13574338 can't identify protocol
>>>>> pvfs2-cli 27278 root 144u sock
>>>>> 0,5 13574344 can't identify protocol
>>>>> pvfs2-cli 27278 root 145u sock
>>>>> 0,5 13574345 can't identify protocol
>>>>> pvfs2-cli 27278 root 146u sock
>>>>> 0,5 13574346 can't identify protocol
>>>>> pvfs2-cli 27278 root 147u sock
>>>>> 0,5 13574357 can't identify protocol
>>>>> ............
>>>>> </lsof output>
>>>>> Thank you,
>>>>> Amit
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pvfs2-developers mailing list
>>>>> Pvfs2-developers at beowulf-underground.org
>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-
>> developers
>>>
>>> _______________________________________________
>>> Pvfs2-developers mailing list
>>> Pvfs2-developers at beowulf-underground.org
>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>
More information about the Pvfs2-developers
mailing list