random reboots of diskless nodes - 2.4.7 (fwd)

[Ryan Sweet]

I've posted about this problem before, but in the meantime I've managed to
test under several different configurations to help rule out some possible
causes.

Short version: 2.4.7 on nfsroot diskless nodes randomly re-boots and I
don't think it is a hardware problem or a problem with the server (which
is stable).  Rather than "try this, try that..." I (and more importantly
my boss) would really like to find (and then hopefully fix) the root cause
of the problem.

See below for the long version and more details.

Questions:
- what the heck can I do to isolate the problem?
- why would the system re-boot instead of hanging on whatever caused it to
crash (ie, why don't I see an oops message?)
- how can I tell the system not to re-boot when it crashes (or is it
crashing at all???)
- is it worth trying all the newer kernel versions (this does not sound
very appealing, especially given the troubles reported with 2.4.10 and
also the split over which vm to use, etc..., also the changelogs don't
really point to anything that appears to precisely describe my problem)?
- if I patch with kgdb and use a null modem connection from the gateway to
run gdb can I expect to gain any useful info from a backtrace?

thanks for your help,
-Ryan Sweet,

 - Long Version -

The problem:

At seemingly random intervals, one or more nodes of a diskless nfsroot
cluster will re-boot without being told to....  No information is available
on the console or in the logs.  Sometimes the systems are up for a weeks,
sometimes it is only a few hours.  Out of 23 nodes, we can pretty
reliably get at least one failure every couple of days or less.  As the
number of nodes decreases, our average time to failure increases.

Notably, the problem happens far less frequently (but still occurs
eventually) when running kernels without SMP support.  Also, 2.4.7 seems
to offer increased stability over previous versions, but still fails.

There seems to be no degradation of performance or measurable increase in
resource consumption or other activity before the re-boot occurs.  Memory
pressure ddoes not seem to be a problem (of 768MB, we are never using more
than 256). lmsensors using the on-board hardware sensors does not record
any anomalies.  In fact, performance on the system when it is not
re-booting is very good.  A tcpdump shows that the last network activity
before the re-boot was an nfs file open, but since the systems are
nfsroot, that pretty much is the majority of the network activity.