[Pvfs2-users] Segmentation fault on insmod pvfs2.ko (pvfs2.6.3-patched, kernel 2.6.22-rc4)

Wed Jun 27 09:39:06 EDT 2007

On Jun 27, 2007, at 8:23 AM, Sam Lang wrote:

>
> On Jun 27, 2007, at 2:50 AM, Murali Vilayannur wrote:
>
>> Mark,
>> Something really bad is going on with these new kernels. This is the
>> 2nd (or perhaps 3rd) time that someone reported this exact same  
>> error.
>> Weird. I cannot think of any reason based on a cursory code glance
>> that could explain this error.
>> Am I missing something, Pete, Sam? Can you guys reproduce this error?
>
> Hi Murali,
>
> Maybe its suse related?  I've seen it happen on a sles9 box with a  
> 2.6.5 kernel.

Actually looking more closely, the backtrace is a little different:

0xc00000005ab57a40  0xc00000000009a0a8  .kmem_cache_create +0x524
0xc00000005ab57b30  0xd0000000003f75a0  .op_cache_initialize +0x3c
0xc00000005ab57bc0  0xd00000000040085c  .pvfs2_init +0x58
0xc00000005ab57c70  0xc000000000084cd4  .sys_init_module +0x23c
0xc00000005ab57d10  0xc00000000001045c  .ret_from_syscall_1 +0x0

This is also when loading the module, but it appears to only happen  
after the module has been loaded/unloaded, so I'm not sure Mark's  
observed behavior is the same.

-sam

> -sam
>
>>
>> kmem_cache_create gives this error message on
>> - a NULL name - which is not the case
>> - if called from an interrupt context - I doubt if this is  
>> possible at all.
>> - size < 4 bytes: not possible. pvfs2_inode_t is 392 bytes
>> - size > 128K: again not possible
>> - no constructor && a destructor specified - not possible from a  
>> code review.
>>
>> Either our build process is broken or your build compiler is broken.
>> Are you using the same compiler to build the kernel module that was
>> used for the core kernel? If for instance, the kernel args to the
>> function are expected in registers instead of on the stack or
>> something fundamental like that, it may break in ways like this.
>> I cannot reproduce this on my setup unfortunately and I am going  
>> to suggest that
>> you try this on some older kernels first if possible and possibly on
>> slightly older gcc versions.
>> Perhaps your setup is too bleeding edge :)
>> Murali
>>
>> On 6/26/07, Mark Van De Vyver <mvyver at gmail.com> wrote:
>>> Hi,
>>> I'd apprciate any assistance in resolving the following segmentation
>>> fault that occurs when I load the pvfs2 kernel module pvfs2.ko.
>>>
>>> I've included the error messages I see as well as the steps I  
>>> used to
>>> compile PVFS2.  I use the PVFS-2.6.3 patched with pvfs2-lkv21.patch
>>> and a patch sent by for the 2.6.22 kernel.
>>>
>>> I'm running opensuse 10.2 with and updated kernel pulled from the  
>>> suse
>>> factory repository:
>>> # uname -a
>>> Linux frontend 2.6.22-rc4-49-default #1 SMP 2007/06/08 14:42:31 UTC
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> # insmod /lib/modules/`uname -r`/kernel/fs/pvfs2/pvfs2.ko
>>> Segmentation fault
>>>
>>> in /var/log/messages, I see:
>>>
>>> Jun 27 16:18:07 frontend kernel: kmem_cache_create: Early error in
>>> slab pvfs2_inode_cache
>>> Jun 27 16:18:07 frontend kernel: ------------[ cut  
>>> here ]------------
>>> Jun 27 16:18:07 frontend kernel: kernel BUG at mm/slab.c:2139!
>>> Jun 27 16:18:07 frontend kernel: invalid opcode: 0000 [1] SMP
>>> Jun 27 16:18:07 frontend kernel: last sysfs file:
>>> /devices/system/cpu/cpu0/cpufreq/scaling_governor
>>> Jun 27 16:18:07 frontend kernel: CPU 0
>>> Jun 27 16:18:07 frontend kernel: Modules linked in: pvfs2(F)  
>>> ip6t_LOG
>>> nf_conntrack_ipv6 ipt_MASQUERADE xt_pkttype xt_TCPMSS ipt_LOG  
>>> xt_limit
>>> nfsd exportfs lockd nfs_acl sunrpc af_packet cpufreq_conservative
>>> cpufreq_ondemand cpufreq_userspace cpufreq_powersave powernow_k8
>>> freq_table button battery ac ip6t_REJECT xt_tcpudp ipt_REJECT  
>>> xt_state
>>> iptable_mangle iptable_nat nf_nat iptable_filter ip6table_mangle
>>> nf_conntrack_ipv4 nf_conntrack nfnetlink ip_tables ip6table_filter
>>> ip6_tables x_tables ipv6 loop usbhid hid ff_memless e1000
>>> cfi_cmdset_0002 cfi_util mtdpart jedec_probe cfi_probe gen_probe
>>> ck804xrom shpchp ide_cd pci_hotplug rtc_cmos rtc_core mtdcore  
>>> ehci_hcd
>>> rtc_lib cdrom ohci_hcd chipreg k8temp i2c_nforce2 hwmon usbcore
>>> map_funcs i2c_core parport_pc lp parport xfs dm_snapshot edd dm_mod
>>> fan sg sata_nv libata amd74xx thermal processor sd_mod scsi_mod
>>> ide_disk ide_core
>>> Jun 27 16:18:07 frontend kernel: Pid: 10147, comm: insmod  
>>> Tainted: GF
>>>    N 2.6.22-rc4-49-default #1
>>> Jun 27 16:18:07 frontend kernel: RIP: 0010:[<ffffffff802856e8>]
>>> [<ffffffff802856e8>] kmem_cache_create+0x6a/0x42e
>>> Jun 27 16:18:07 frontend kernel: RSP: 0000:ffff8101c4263dc8   
>>> EFLAGS: 00010296
>>> Jun 27 16:18:07 frontend kernel: RAX: 000000000000003c RBX:
>>> ffffffff8851a900 RCX: ffffffff804aa728
>>> Jun 27 16:18:07 frontend kernel: RDX: ffffffff804aa728 RSI:
>>> 0000000000000096 RDI: ffffffff804aa720
>>> Jun 27 16:18:07 frontend kernel: RBP: 0000000000000288 R08:
>>> ffffffff804aa728 R09: ffffffff885077fb
>>> Jun 27 16:18:07 frontend kernel: R10: 0000000000000046 R11:
>>> ffff81020be62000 R12: 0000000000000000
>>> Jun 27 16:18:07 frontend kernel: R13: 0000000000000000 R14:
>>> ffffc20002dfaee0 R15: 000000000000001c
>>> Jun 27 16:18:07 frontend kernel: FS:  00002b709d2666f0(0000)
>>> GS:ffffffff804ec000(0000) knlGS:00000000f7a276d0
>>> Jun 27 16:18:07 frontend kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
>>> 000000008005003b
>>> Jun 27 16:18:07 frontend kernel: CR2: 00002b709cd7f00f CR3:
>>> 00000001e8c42000 CR4: 00000000000006e0
>>> Jun 27 16:18:07 frontend kernel: Process insmod (pid: 10147,
>>> threadinfo ffff8101c4262000, task ffff810214cb4890)
>>> Jun 27 16:18:07 frontend kernel: Stack:  ffffffff8850781c
>>> ffffffff88511322 fffffffffffffff8 0000000100000001
>>> Jun 27 16:18:07 frontend kernel:  0000000000000000 0000000000000fc8
>>> 0000000100000000 ffffffff8851a900
>>> Jun 27 16:18:07 frontend kernel:  0000000000000000 ffffffff8851a900
>>> ffff81020be624e8 ffffc20002dfaee0
>>> Jun 27 16:18:07 frontend kernel: Call Trace:
>>> Jun 27 16:18:07 frontend kernel:  [<ffffffff8850781c>]
>>> :pvfs2:pvfs2_inode_cache_ctor+0x0/0x50
>>> Jun 27 16:18:07 frontend kernel:  [<ffffffff8850764b>]
>>> :pvfs2:pvfs2_inode_cache_initialize+0x27/0x4b
>>> Jun 27 16:18:07 frontend kernel:  [<ffffffff880c105b>]
>>> :pvfs2:pvfs2_init+0x5b/0x289
>>> Jun 27 16:18:07 frontend kernel:  [<ffffffff8024e43f>]
>>> sys_init_module+0x16bf/0x1822
>>> Jun 27 16:18:07 frontend kernel:  [<ffffffff80209b6e>] system_call 
>>> +0x7e/0x83
>>> Jun 27 16:18:07 frontend kernel:
>>> Jun 27 16:18:07 frontend kernel:
>>> Jun 27 16:18:07 frontend kernel: Code: 0f 0b eb fe 48 c7 c7 80 fb 4a
>>> 80 e8 bd 95 16 00 48 8b 1d 61
>>> Jun 27 16:18:07 frontend kernel: RIP  [<ffffffff802856e8>]
>>> kmem_cache_create+0x6a/0x42e
>>> Jun 27 16:18:07 frontend kernel:  RSP <ffff8101c4263dc8>
>>>
>>>
>>> I've attached the log files that are created below.
>>>
>>> PVFS_VER=2.6.3
>>> (
>>> pushd /usr/src/linux
>>> make oldconfig
>>> make scripts
>>> make prepare
>>> make modules
>>> popd
>>> ) &>/tmp/pvfs-${PVFS_VER}-kernel-config.log
>>>
>>> pushd /usr/src/pvfs2
>>> make distclean
>>> ./configure --with-kernel=/usr/src/linux/ --enable-kernel-sendfile
>>> --prefix=/opt/pvfs2/ &>/tmp/pvfs-${PVFS_VER}-configure.log
>>> cat /tmp/pvfs-${PVFS_VER}-configure.log | more
>>> (
>>> make all
>>> make install
>>> make kmod_install
>>> ) &>/tmp/pvfs-${PVFS_VER}-make-kmod-install.log
>>> popd
>>>
>>>
>>> I would appreciate any help someone can offer.
>>> Thanks in advance.
>>>
>>> Mark
>>>
>>> _______________________________________________
>>> Pvfs2-users mailing list
>>> Pvfs2-users at beowulf-underground.org
>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>
>>>
>>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>