Re: [PATCH][2.4] generic cluster APIC support for systems with m ore than 8 CPUs

William Lee Irwin III (wli@holomorphy.com)
Fri, 20 Dec 2002 15:33:39 -0800


On Fri, Dec 20, 2002 at 04:57:28PM -0600, Protasevich, Natalie wrote:
> Briefly, our ES7000 boxes are non-NUMA, but use clustered APICs (logical
> with Cascades, and physical with Gallatins/Fosters). Our code is pretty much
> within the clustered APIC code (when both physical and logical are
> implemented). Even with NUMA that is forced in clustered APIC case, we are
> usually OK as a single-node case.

Okay, so nothing wild like a non-APIC interrupt controller is going on
here. (c.f. Voyager for an example).

On Fri, Dec 20, 2002 at 04:57:28PM -0600, Protasevich, Natalie wrote:
> There are only a few problems with porting the Linux kernel to the ES7000:
> we use 8-bit APIC IDs - this makes us use APIC_LDR instead of
> APIC_ID throughout the code;
> we have special RTE destination values on IO-APIC - the "if" in the
> programming IO-APIC line code;
> we introduce severe IRQ override case - we remap ISA interrupts to a
> different interrupt range (all the "i < 16" clauses).
> Also, I usually have to add things like XTPR mechanism for Fosters/Gallatins
> and disable conventional IRQ balancing, since our IO-APIC doesn't work this
> way... (All of the above is in the SuSE code base).

Venkatesh, do you think you can handle these generically? Aside from
machine-specific configurations this all looks like perfectly generic.

If it's publicly discussable, what's the difference wrt. the IO-APIC?
IIRC NUMA-Q had a similar issue, where flat logical destinations were
being programmed into the IO-APIC by the IRQ balancing code, but the
NUMA-Q IO-APIC was programmed to accept physical destinations in the
RTE's via the DESTMOD bit, using physical broadcast by default, and
achieving node-locality as physical destinations may not refer to
off-node cpus. There probably isn't an issue of node locality, but even
if the IO-APIC's are programmed for logical DESTMOD it won't work with
the flat logical gunk the original IRQ balance patch programmed up.

#ifdef CONFIG_CLUSTERED_APIC
#define INT_DELIVERY_MODE 0 /* physical delivery on LOCAL quad */
#else
#define INT_DELIVERY_MODE 1 /* logical delivery broadcast to all procs */
#endif

#ifdef CONFIG_SMP
#define TARGET_CPUS (clustered_apic_mode ? 0xf : cpu_online_map)
#else
#define TARGET_CPUS 0x01
#endif

And while setting up the RTE's in io_apic.c:

entry.delivery_mode = dest_LowestPrio;
entry.dest_mode = INT_DELIVERY_MODE;
entry.mask = 0; /* enable IRQ */
entry.dest.logical.logical_dest = TARGET_CPUS;

... which is rather blatant abuse of entry.dest.logical.logical_dest
for the NUMA-Q case, but never mind that.

On Fri, Dec 20, 2002 at 04:57:28PM -0600, Protasevich, Natalie wrote:
> I worked with the SuSE tree which has clustered code (at the first glance)
> close to the patch being discussed here.
> The 2.5 tree gives us a benefit of the subarch that will accomodate
> (hopefully) our special cases.
> But I may need to add more hooks.

It'd be great to have the APIC interface general enough to handle all
these machines.

Bill
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/