Re: Re: kswapd OOPS under 2.4.19-pre8 (ext3, Reiserfs + (soft)raid0)

William Lee Irwin III (wli@holomorphy.com)
Mon, 20 May 2002 15:36:13 -0700


On Mon, May 20, 2002 at 03:26:56PM -0500, Todd R. Eigenschink wrote:
> Someone else had suggested testing the memory and power supply.
> memtest86 is easy to run, so I'll try that. It'll have to be tonight,
> now.

Bitflips are usually things where a pointer turns up invalid (or
non-NULL) and the difference between it and a valid pointer (or NULL)
is one bit. I don't see that here and don't like blaming hardware.

On Mon, May 20, 2002 at 03:26:56PM -0500, Todd R. Eigenschink wrote:
> Well, after my posting from earlier today, I recompiled the kernel
> after stripping some more stuff. I just induced an oops in that one,
> so I can list the call stack for it.

Nice, I presume you've got -g there? Any chance of doing something like
objdump --disassemble --source vmlinux and sending me the annotated
disassembly of __wake_up()? I want to doublecheck something...

On Mon, May 20, 2002 at 03:26:56PM -0500, Todd R. Eigenschink wrote:
> No IDE stuff this time; this looks a lot like most of the other ones
> I've seen. This morning was the first time I've ever seen IDE stuff
> in the post-oops call stack.

This is pretty strange, yes.

On Mon, May 20, 2002 at 03:26:56PM -0500, Todd R. Eigenschink wrote:
> It seems I can pretty much induce them at will, now. I started up
> four simultaneous Webtrends sessions, which grow fairly quickly to
> 400-600 MB each, give or take. (The machine has 2 GB of RAM, so it
> only swaps a little, sometimes.) Within half an hour, it fell over.
> Here's the oops itself, then the gdb output.

Great stuff! Thanks.

On Mon, May 20, 2002 at 03:26:56PM -0500, Todd R. Eigenschink wrote:
> Oops: 0000
> CPU: 1
> EIP: 0010:[<c0116363>] Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010087
> eax: c2802db4 ebx: c2002db4 ecx: 00000000 edx: 00000003
> esi: c2802db0 edi: c2802db0 ebp: f7bf3ee8 esp: f7bf3ecc
> ds: 0018 es: 0018 ss: 0018

Okay, %ecx is 0 -- no bitflip, just plain old NULL...

On Mon, May 20, 2002 at 03:26:56PM -0500, Todd R. Eigenschink wrote:
> Code; c0116363 <__wake_up+3b/c0>
> 00000000 <_EIP>:
> Code; c0116363 <__wake_up+3b/c0> <=====
> 0: 8b 01 mov (%ecx),%eax <=====
> Code; c0116365 <__wake_up+3d/c0>
> 2: 85 45 fc test %eax,0xfffffffc(%ebp)
> Code; c0116368 <__wake_up+40/c0>
> 5: 74 66 je 6d <_EIP+0x6d> c01163d0 <__wake_up+a8/c

Okay, the offending instruction is mov (%ecx), %eax -- dereferencing the
NULL %ecx...

On Mon, May 20, 2002 at 03:26:56PM -0500, Todd R. Eigenschink wrote:
> (gdb) list *__wake_up+0x3b
> 0x973 is in __wake_up (sched.c:731).
> 726 unsigned int state;
> 727 wait_queue_t *curr = list_entry(tmp, wait_queue_t, task_list);
> 728
> 729 CHECK_MAGIC(curr->__magic);
> 730 p = curr->task;
> 731 state = p->state;
> 732 if (state & mode) {
> 733 WQ_NOTE_WAKER(curr);
> 734 if (try_to_wake_up(p, sync) && (curr->flags&WQ_FLAG_EXCLUSIVE) && !--nr_exclusive)
> 735 break;

This makes it pretty clear the offending instruction corresponds to the
first dereference of curr->task. Someone's leaving a NULL pointer in
there when they shouldn't. So this entire call chain has nothing to do
with the offender -- it only trips over the bad pointer the offending
code left behind. This looks like a PITA. The objdump --disassemble
--source stuff is just to have the assembly and source next to each
other for a "more convincing" demonstration, not that this isn't already
pretty good as it stands. Of course, finding the offender will be painful.

Cheers,
Bill
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/