Re: Linux and Network Flight Recorder

[Jamie Lokier]

Matti Aarnio wrote:
> >   Zero-copy      115Mbyte/s      6% CPU
> >   Single-copy     82Mbyte/s     97% CPU
> 
> 	And pray tell, what does this "6% CPU" really mean ?
> 	BSD and Linux do account interrupt-side work differently.

It means there's 94% left to play with (duh).
There are no interrupts.  [*]

> 	Unless the "Zero-copy" system really begins by supplying
> 	set of buffer descriptors that 1) are in userspace, and
> 	2) already are locked into memory, and 3) can be mapped
> 	into physical addresses for the network card DMA, I don't
> 	believe into this claimed zero-copy thing...

That's more or less what we do, yes.
Except we skip the kernel.

> 	Now how to tell the user-space in most efficient manner, that new
> 	packet(s) have arrived ?  Busy poll is -- not so nice a thing...

It's the fastest and has the lowest CPU overhead, and the lowest latency
if done properly, but I agree it's not nice.  Like interrupts it doesn't
generate any bus traffic when nothing's happening.

> >   - Alteon ACEnic Tigon-2 PCI
> >   - 400MHz Pentium II (single processor)
> >   - BX chipset
> >   - 64Mb main memory
> >   - 512k L2 cache
> > 
> > I don't very much is actually _done_ with that data though.

... except a minimal amount of protocol and thread dispatch.

FWIW, our driver could probably handle filtering & recording at Gigabit
rate, or indeed multiple 100Mbit/s cards.  But I'm not going to write it.

enjoy,
-- Jamie

[*]

  This implementation works by polling DMA descriptors, which normally
  get to live in CPU cache so polling is fast.  Poll statements are
  inserted in the user-space code.

  The polls are fast but intrusive.  We could automate their insertion.
  However it's a pain and doesn't generalise e.g. for Linux running
  normal apps.

  In RL per-packet interrupts are a killer at this rate.
  No-one has got this performance with interrupts AFAIK.

  I'm contemplating RT-Linux style fast timer interrupts to poll the DMA
  descriptors as a generalisation which might just work.  We'd lose in
  latency and add a (fixed) extra CPU overhead, but it would be
  predictable, and any user-space code would be ok.