[Pvfs2-users] Help! PVFS2 cause kernel panic

Sam Lang slang at mcs.anl.gov
Tue Jan 9 01:32:11 EST 2007


Hi Jiang,

To further Rob's suggestion, does pvfs2-ping complete successfully on  
your filesystem?

-sam

On Jan 8, 2007, at 11:51 PM, Rob Ross wrote:

> Hi,
>
> Were the servers still up and running when this happened?
>
> What's your network? Have you seen dropped packets etc?
>
> We're not used to seeing applications that are so persistent in
> continuing to perform I/O in the presence of failures, so there may  
> be a
> bug here that is being uncovered by this continuous reading in an  
> error
> case...
>
> Thanks for the report!
>
> Rob
>
> jiang yi wrote:
>> HI Murali,
>> I use the RTSP over PVFS2 filesystem(pvfs2-1.5.1). Build a Media  
>> server
>> system with 4 computer nodes and 2 io nodes.
>> But these days the PVFS2 cause 3 computer nodes kernel panic. help  
>> me!
>>
>> This is any of the log in /var/log/messages.
>> Help me analyse this and tell me how to fix the problem. Or I will  
>> die ^_^
>> Thank you so much!!
>>
>>
>> Aug 29 14:32:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Aug 29 14:32:46 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1046588, -- returning -110
>> Aug 29 17:17:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Aug 29 17:17:53 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1047656, -- returning -110
>> Aug 29 18:05:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Aug 29 18:05:36 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1046827, -- returning -110
>> Aug 29 18:14:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Aug 29 18:14:03 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1046590, -- returning -110
>> Aug 29 19:24:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Aug 29 19:24:03 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1048431, -- returning -110
>> Aug 29 20:04:40 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Aug 29 20:04:40 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1046679, -- returning -110
>> Sep 14 16:14:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 16:14:55 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 16:14:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 16:14:56 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 16:15:37 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 16:15:37 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 16:21:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 16:21:53 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 16:23:04 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 16:23:04 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1042058, -- returning -110
>> Sep 14 16:27:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 16:27:53 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:15:39 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:15:39 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:15:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:15:52 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:16:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:16:09 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:16:10 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:16:10 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:21:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:21:52 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:27:59 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:27:59 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:28:12 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:28:12 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 17:39:26 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:39:26 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039033, -- returning -110
>> Sep 14 17:43:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 17:43:06 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1043039, -- returning -110
>> Sep 14 18:08:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 18:08:54 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1040923, -- returning -110
>> Sep 14 18:18:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 18:18:53 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1040923, -- returning -110
>> Sep 14 18:27:07 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 18:27:07 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1042057, -- returning -110
>> Sep 14 18:33:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 18:33:52 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039041, -- returning -110
>> Sep 14 18:33:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 18:33:53 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 18:34:01 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 18:34:01 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038496, -- returning -110
>> Sep 14 18:50:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 18:50:06 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 19:03:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 19:03:36 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038489, -- returning -110
>> Sep 14 20:34:49 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 20:34:49 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038485, -- returning -110
>> Sep 14 20:34:51 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 20:34:51 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038485, -- returning -110
>> Sep 14 20:34:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 20:34:56 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038485, -- returning -110
>> Sep 14 21:16:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 21:16:55 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038485, -- returning -110
>> Sep 14 21:25:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 21:25:52 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039037, -- returning -110
>> Sep 14 21:33:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 21:33:09 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038485, -- returning -110
>> Sep 14 21:52:30 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 21:52:30 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1039031, -- returning -110
>> Sep 14 22:17:05 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 22:17:05 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038485, -- returning -110
>> Sep 14 22:17:45 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Sep 14 22:17:45 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038485, -- returning -110
>> Sep 30 14:29:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out
>> and retries exhausted. aborting attempt.
>> Sep 30 14:30:18 redhat2 last message repeated 2 times
>> Sep 30 14:35:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out
>> and retries exhausted. aborting attempt.
>> Sep 30 14:36:18 redhat2 last message repeated 2 times
>> Nov  2 22:16:08 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Nov  2 22:16:08 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1038504, -- returning -110
>> Nov 12 14:54:57 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1036658, -- returning -2
>> Nov 12 14:55:00 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1036651, -- returning -2
>> Nov 12 14:57:00 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1036651, -- returning -2
>> Nov 12 14:57:04 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1036658, -- returning -2
>> Nov 12 14:57:42 redhat2 last message repeated 2 times
>> Nov 12 15:00:08 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1036651, -- returning -2
>> Nov 12 15:00:08 redhat2 last message repeated 2 times
>> Dec  5 12:51:01 redhat2 kernel: printk: 2 messages suppressed.
>> Dec  5 12:51:05 redhat2 kernel: printk: 1 messages suppressed.
>> Dec  5 12:51:20 redhat2 kernel: printk: 3 messages suppressed.
>> Dec  5 12:51:35 redhat2 kernel: printk: 2 messages suppressed.
>> Dec  5 12:52:26 redhat2 kernel: printk: 1 messages suppressed.
>> Dec  5 12:53:29 redhat2 kernel: printk: 2 messages suppressed.
>> Dec  5 12:53:36 redhat2 kernel: printk: 3 messages suppressed.
>> Dec  5 12:53:43 redhat2 kernel: printk: 2 messages suppressed.
>> Dec  5 12:53:52 redhat2 kernel: printk: 4 messages suppressed.
>> Dec  5 12:54:05 redhat2 kernel: printk: 3 messages suppressed.
>> Dec  5 12:54:08 redhat2 kernel: printk: 2 messages suppressed.
>> Dec  5 12:54:13 redhat2 kernel: printk: 2 messages suppressed.
>> Dec  5 12:54:19 redhat2 kernel: printk: 5 messages suppressed.
>> Dec  5 12:54:36 redhat2 kernel: printk: 4 messages suppressed.
>> Dec  5 12:54:44 redhat2 kernel: printk: 5 messages suppressed.
>> Dec  5 12:54:46 redhat2 kernel: printk: 1 messages suppressed.
>> Dec  5 12:55:00 redhat2 kernel: printk: 1 messages suppressed.
>> Dec  5 12:55:02 redhat2 kernel: printk: 3 messages suppressed.
>> Dec  5 12:55:07 redhat2 kernel: printk: 1 messages suppressed.
>> Dec  5 12:55:38 redhat2 kernel: printk: 3 messages suppressed.
>> Dec 24 16:54:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Dec 24 16:54:54 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1042026, -- returning -110
>> Dec 24 17:04:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait  
>> timed out
>> and retries exhausted. aborting attempt.
>> Dec 24 17:04:46 redhat2 kernel: pvfs2_file_read: error writing to  
>> handle
>> 1036052, -- returning -110
>> Dec 26 10:49:33 redhat2 sshd(pam_unix)[15164]: authentication  
>> failure;
>> logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133  user=root
>> Dec 26 10:50:51 redhat2 sshd(pam_unix)[15164]: 1 more authentication
>> failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133
>> user=root
>> Dec 28 14:13:48 redhat2 kernel: printk: 1 messages suppressed.
>> Dec 28 14:14:26 redhat2 last message repeated 4 times
>> Dec 28 14:14:37 redhat2 kernel: printk: 2 messages suppressed.
>> Dec 28 14:14:46 redhat2 kernel: printk: 1 messages suppressed.
>> Dec 28 14:14:57 redhat2 kernel: printk: 2 messages suppressed.
>> Dec 28 14:42:22 redhat2 kernel: printk: 2 messages suppressed.
>> Dec 31 20:36:57 redhat2 kernel: Unable to handle kernel paging  
>> request
>> at virtual address 67c38e12
>> Dec 31 20:36:57 redhat2 kernel:  printing eip:
>> Dec 31 20:36:57 redhat2 kernel: f89e363e
>> Dec 31 20:36:57 redhat2 kernel: *pde = 00000000
>> Dec 31 20:36:57 redhat2 kernel: Oops: 0000 [#1]
>> Dec 31 20:36:57 redhat2 kernel: SMP
>> Dec 31 20:36:57 redhat2 kernel: Modules linked in: ip_vs pvfs2(U) md5
>> ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core sunrpc dm_mirror
>> dm_multipath dm_mod button battery ac uhci_hcd e1000 floppy ext3 jbd
>> qla2300 qla2xxx scsi_transport_fc mptscsih mptbase sd_mod scsi_mod
>> Dec 31 20:36:57 redhat2 kernel: CPU:    2
>> Dec 31 20:36:57 redhat2 kernel: EIP:    0060:[<f89e363e>]    Not  
>> tainted VLI
>> Dec 31 20:36:57 redhat2 kernel: EFLAGS: 00010246   (2.6.9-22.ELsmp)
>> Dec 31 20:36:57 redhat2 kernel: EIP is at
>> pvfs2_devreq_writev+0x184/0x59b [pvfs2]
>> Dec 31 20:36:57 redhat2 kernel: eax: f5d66710   ebx: 67c38e12   ecx:
>> 000001fd   edx: f6a00000
>> Dec 31 20:36:57 redhat2 kernel: esi: f5d66700   edi: fffffe50   ebp:
>> f56cced4   esp: f56cceb8
>> Dec 31 20:36:57 redhat2 kernel: ds: 007b   es: 007b   ss: 0068
>> Dec 31 20:36:57 redhat2 kernel: Process pvfs2-client-co (pid: 4151,
>> threadinfo=f56cc000 task=f6ff00b0)
>> Dec 31 20:36:57 redhat2 kernel: Stack: f56cceb4 000022a8 00000004
>> dd4fc42c dd4fc41c 00000004 f56ccf40 80000000
>> Dec 31 20:36:57 redhat2 kernel:        00000000 f765fe80 00000000
>> c02759c6 00000040 00000000 00000000 f56ccf40
>> Dec 31 20:36:57 redhat2 kernel:        00000001 00000000 00000000
>> 00000040 f63ccf48 00000a98 c0275a16 00000246
>> Dec 31 20:36:57 redhat2 kernel: Call Trace:
>> Dec 31 20:36:57 redhat2 kernel:  [<c02759c6>] sock_readv_writev 
>> +0x5e/0x81
>> Dec 31 20:36:57 redhat2 kernel:  [<c0275a16>] sock_readv+0x2d/0x34
>> Dec 31 20:36:57 redhat2 kernel:  [<f89e34ba>]
>> pvfs2_devreq_writev+0x0/0x59b [pvfs2]
>> Dec 31 20:36:57 redhat2 kernel:  [<c015a177>] do_readv_writev 
>> +0x19c/0x21d
>> Dec 31 20:36:57 redhat2 kernel:  [<c015a276>] vfs_writev+0x3e/0x43
>> Dec 31 20:36:57 redhat2 kernel:  [<c015a319>] sys_writev+0x3c/0x62
>> Dec 31 20:36:57 redhat2 kernel:  [<c02d10cf>] syscall_call+0x7/0xb
>> Dec 31 20:36:57 redhat2 kernel: Code: b9 ff ff ff 5e e9 2e 04 00  
>> 00 8b
>> 35 c4 5a 9f f8 8d 6c 24 1c 89 e8 8b 56 04 ff 56 0c 89 c7 8d 46 10  
>> e8 0c
>> c6 8e c7 8b 16 8b 1c fa <8b> 03 0f 18 00 90 8d 04 fa 39 c3 74 1c  
>> 89 da
>> 89 e8 ff 56 08 8b
>> Dec 31 20:36:57 redhat2 kernel:  <0>Fatal exception: panic in 5  
>> seconds
>>
>>
>>
>> --------------------------------------------------------------------- 
>> ---
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>



More information about the Pvfs2-users mailing list