Re: Two hdds on one channel - why so slow?

Rob Landley (landley@trommello.org)
Tue, 8 Jan 2002 07:41:42 -0500


On Friday 04 January 2002 12:40 pm, Jesse Pollard wrote:
> --------- Received message begins Here ---------
>
> > In article <200201041402.IAA80257@tomcat.admin.navo.hpc.mil> you wrote:
> > > In my experience, SCSI is not cost effective for systems with a single
> > > disk. As soon as you go to 4 or more disks, the throughput of SCSI
> > > takes over unless you are expanding a pre-existing workstation
> > > configuration.
> >
> > IDE Scales fine to 8 Channels (aka 8 Drives). Anything more than 8 Drives
> > on an HBA is insane anyway.
> >
> > I love the FC-to-IDE(8) Solution. You get Hardware Raid with 8 Channels,
> > each drive a didicated channel, thats much more reliable than usual 2 or
> > 3 channel SCSI Configurations.
> >
> > Do you realy run more than say 10 hard disk devices on a single SCSI Bus,
> > ever?
>
> Only place I've seen that (not sure it was true SCSI though) was on a
> Cray file server - each target (4) was itself a raid 5, with 5 transports.
> That would be total of 20 disks. The four targets were striped together in
> software to form a single filesystem. I think that was our first over 300GB
> filesystem (several years ago). Now we are using 1 TB to 5TB filesystems
> with nearly 5 million files each.

A couple years back I played with a 20-way SCSI software RAID, which was
actually dual qlogic fiber channel, 10 drives per controller. That was done
for throughput reasons (attempting to capture uncompressed HDTV signals to
disk, needed something like 160 megabytes per second, which of course
required a 66 mhz 64 bit PCI bus...) It broke the kernel in more than one
place, by the way. Long since fixed, I believe. (Try it and see. :)

More recently I've played around a little with many-way IDE software RAID,
which is much more fun. The hard part is really getting enough power for all
the drives (Ever seen an enormous tower case with three power supplies
mounted in it, AND a lot of splitters?) The goal was to see how cheaply we
could get to 10 terabytes of storage, so we didn't really care about
throughput there (the output of the cluster was the gigabit ethernet uplink
on the switch, and the output of each node was a pair of bonded 100baseT
ethernet adapters). So we were willing to hanging two drives off each IDE
controller (master and slave), which halved the throughput but that wasn't
the bottleneck anyway. (Dual 100baseT is 22 megabytes per second, a SINGLE
modern IDE drive can swamp that. And the raid could max out the PCI bus
pretty easily with one drive per controller...)

You can cheaply get generic IDE expansion cards with at least 2 controllers
on each card (4 drives per card), and you can get a board with onboard video,
at least one onboard NIC (we had two onboard, that's why we did the bonding),
two onboard IDE controllers, and 4 free PCI slots. That's 4 drives onboard,
16 through PCI, total of 20 drives. (You need to tweak the linux kernel to
allow that many IDE controllers. PCI plug and pray is your friend here...)

The goal was to see how cheaply we could get to 10 terabytes of storage. We
didn't do the whole cluster, but I think we determined we only needed 4 or 5
nodes to do it. Then the dot-com crash hit and that company's business model
changed, project got shelved...

But if you think about serving MPEG 4 video streams through simple TCP/IP
(DVD quality's only about 150 kilobytes per second with mpeg 4, add in 5 megs
of buffer at the client side and who cares about latency...) Really big IDE
software RAID systems are quite nice. :)

Rob
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/