Re: Kernel deadlock using nbd over acenic driver

Steven Whitehouse (steve@gw.chygwyn.com)
Wed, 15 May 2002 20:43:49 +0100 (BST)


Hi,

thanks for the info. I'm starting to form some ideas of what the problem
with nbd might be. Here is my initial idea of what might be going on:

1. Something happens which starts lots of I/O (e.g. the ext3/xfs journal
flush that Xiangping says usually triggers the problem)
2. One of the threads doing the writes blocks running the device I/O
queue and causing nbd_send_req(), nbd_xmit() to block in the
sendmsg() call (trying to allocate memory GFP_NOIO). I think we
only need to have each memory zones free pages just below pages_min
at the right time to trigger this.
3. Since bdflush will most likely be running it waits for the dirty
blocks its submitted to finish being written back to the
nbd device to finish.

So something to try is this, in nbd_send_req() add the lines:

if (current->flags & PF_MEMALLOC == 0) {
current->flags |= PF_MEMALLOC;
we_set_memalloc = 1;
}

before the first nbd_xmit() call and

if (we_set_memalloc)
current->flags &= ~PF_MEMALLOC;

at the end just before the return; rememebring to declare the variable:

int we_set_memalloc = 0;

at the top of the function. We know that since the box stays responsive to
pings that there must be some free memory, so I suspect some kind of
"priority inversion" is at work here.

Another interesting idea... if we changed the icmp receive function so that
it leaked all echo request packets rather than recycling them we could
measure the free memory after the box has hung by seeing how many times
we can ping it before it stops replying. An interesting way to
measure free memory, but probably quite effective :-)

It looks like adding the line:

if (icmph->type == ICMP_ECHO) return;

just after (icmp_pointers[icmph->type].handler)(skb); in icmp.c:icmp_rcv()
should do the trick in this case.

Those are my current thoughts. Let me know if you are sure that some of
what I've said is right/wrong, otherwise I'll have a go tomorrow at
trying to prove some of it on my test system here,

Steve.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/