Re: [PATCH][2.5] smp_call_function needs mb() - oopsable

[Zwane Mwaikambo]

On Thu, 3 Apr 2003, Linus Torvalds wrote:

> On Thu, 3 Apr 2003, Zwane Mwaikambo wrote:
> >
> > I have a 3 Processor Pentium 133 system w/ 512k external cache which is 
> > oopsing reliably in the exact same location.
> 
> Whee. What a piece of "interesting hardware".

Yep, it appears to have PCI bus funnies too..

> I really think that your patch is a bit questionable. The wmb() on the 
> sender side looks correct as-is, and to me it looks like it is the 
> _receiver_ side that might need a read-barrier before it reads 
> call_data(). 

I'm compiling with rmb before the APIC EOI, which is after the local 
variable assignments (i'll post the results in a bit, slow build box).

> I really thought that the interrupt should be a serializing event, but I 
> can't find that in the intel databooks (they make "iret" a serializing 
> instruction, but not _taking_ an interrupt, unless I missed something).
> 
> Can you check if you get the right behaviour if you have a read barrier in 
> the receive path? I actually think we need a full mb() on _both_ paths, 
> since the current wmb() only guarantees that writes will be seen "in 
> order wrt other writes", and while the IPI generation really _is_ a write 
> in itself, I wonder if the Intel CPU's might not consider it something 
> special..

I haven't actually seen anything mentioning writing to ICR 
having a serializing side effect. However the forced read around write 
with family < P6 (CONFIG_X86_GOOD_APIC) in Linux should ensure that no 
reads or writes pass the APIC/ICR write due to the xchg, however wether 
'implied' lock ensures that it's treated the same as implicit lock i don't 
know.

> I'm not opposed to your patch per se, but I really do believe that it is 
> potentially wrong. If we have no serialization on the read side, your 
> patch might not actually fully plug the real bug, only hide it. I'd like 
> to know if a read barrier on the read side (without the full barrier on 
> the write side) is sufficient. It _should_ be (but see my worry about the 
> APIC write maybe being considered "outside the scope" of the normal cache 
> coherency protocols).

Wouldn't APIC writes be then treated differently on a P4 (which uses the 
system bus) and the P5/P6 which has it's own serial bus? Imo we should not 
rely on APIC loads/stores which is why i added an rmb after the APIC EOI 
for the rmb in smp_call_function_interrupt test.

	Zwane
-- 
function.linuxpower.ca