Re: Two hdds on one channel - why so slow?

[Rob Landley]

On Tuesday 08 January 2002 04:18 pm, William Park wrote:
> On Tue, Jan 08, 2002 at 07:41:42AM -0500, Rob Landley wrote:
> > The goal was to see how cheaply we could get to 10 terabytes of storage. 
> > We didn't do the whole cluster, but I think we determined we only needed
> > 4 or 5 nodes to do it.  Then the dot-com crash hit and that company's
> > business model changed, project got shelved...
>
> Hi Rob, how did you manage to get 10TB storage?  It's my understanding
> that kernel block device still counts 1kB blocks using 32bit (signed)
> integer.  So, that's 2TB in total.  Are you talking about 5 x 2TB?

Made a cluster.

We were extracting stuff out of it via URL, with a database to lookup where 
each URL lived, so we could have different files live on different servers.  
(If we'd wanted everything to look like it lived on exactly the same machine, 
we could have had one machine mount the other machines' space via samba or 
nfs, but that would have created extra network traffic inside the cluster.)

The proposed design was to have the whole cluster look like it was at 1 
public IP address via IP masquerading and port forwarding (port 80 is the 
apache on node 0, 81 is the apache on node 1, 82 is the apache on node 2...)  
This was just to save world-routable IPs.  We didn't get that far...

Bascially, we just wanted lots of storage, cheap and reliable (we were doing 
RAID 5 across the disks in each cluster), and didn't care what it looked 
like.  We were also experimenting with DVD jukeboxes to feed data into the 
cluster (the cluster was cache for larger offline storage, the project was to 
license syndicated television content (old episodes of mash, battlestar 
galactica, you name it) and provide video on demand for a flat monthly fee.  
Each local cable company would have a cluster, which would pull data through 
the internet from servers in atlanta or california, wherever an ultimate 
content licensor lived.  It could be shipped around on DVD stacks too...)

Fun project.  Too bad it didn't work out...

Rob