Gigabit/SMP performance problem

[Avery Fay]

Hello,

I'm working with a dual xeon platform with 4 dual e1000 cards on different 
pci-x buses. I'm having trouble getting better performance with the second 
cpu enabled (ht disabled). With a UP kernel (redhat's 2.4.18), I can route 
about 2.9 gigabits/s at around 90% cpu utilization. With a SMP kernel 
(redhat's 2.4.18), I can route about 2.8 gigabits/s with both cpus at 
around 90% utilization. This suggests to me that the network code is 
serialized. I would expect one of two things from my understanding of the 
2.4.x networking improvements (softirqs allowing execution on more than 
one cpu):

1.) with smp I would get ~2.9 gb/s but the combined cpu utilization would 
be that of one cpu at 90%.
2.) or with smp I would get more than ~2.9 gb/s.

Has anyone been able to utilize more than one cpu with pure forwarding?

Note: I realize that I am not using a stock kernel. I was in the past, but 
I ran into the same problem (smp not improving performance), just at lower 
speeds (redhat's kernel was faster). Therefore, this problem is neither 
introduced nor solved by redhat's kernel. If anyone has suggestions for 
improvements, I can move back to a stock kernel.

Note #2: I've tried tweaking a lot of different things including binding 
irq's to specific cpus, playing around with e1000 modules settings, etc.

Thanks in advance and please CC me with any suggestions as I'm not 
subscribed to the list.

Avery Fay

P.S. Only got one response on the linux-net list so I'm posting here. One 
thing I did learn from that response is that redhat's kernel is faster 
because they use a napi version of the e1000 driver.