Two RAID1 mirrors are faster than three

[Chuck Ebbert]

  All these tests were run on 2.4.21-rc2-ac1...

  We start out with a three disk raid1 array in failure mode:
        md21 : active raid1 hdi9[1] hdg9[2]
              3145856 blocks [3/2] [_UU]
        
  There are three 500 MiB contiguous files in the volume root:
        -rw-r--r--    1 root     root     524288000 May 11 13:53 file1
        -rw-r--r--    1 root     root     524288000 May 11 13:54 file2
        -rw-r--r--    1 root     root     524288000 May 11 13:55 file3

  First test is a single sequential stream:
        # umount /mnt/md21; mount /mnt/md21
        # time dd if=/mnt/md21/file1 of=/dev/null bs=64k &
        real    0m18.230s   <==== 27.5 MiB/s 
        user    0m0.020s
        sys     0m3.760s

  Now two streams:
        # umount /mnt/md21; mount /mnt/md21
        # time dd if=/mnt/md21/file1 of=/dev/null bs=64k &
        # time dd if=/mnt/md21/file2 of=/dev/null bs=64k &
        real    0m18.065s   <==== 27.6 MiB/s
        user    0m0.040s
        sys     0m4.280s
        real    0m23.197s   <==== 21.6 MiB/s
        user    0m0.020s
        sys     0m4.250s

  Add the third disk back into the array:
        # raidhotadd /dev/md21 /dev/hde9
        [rebuilt 3000MiB in 313s == (3000+3000)/313 == 19MiB/s throughput]
        
  Now rerun the two streams test:
        # umount /mnt/md21; mount /mnt/md21
        # time dd if=/mnt/md21/file1 of=/dev/null bs=64k &
        # time dd if=/mnt/md21/file2 of=/dev/null bs=64k &
        real    0m50.336s   <==== 9.94 MiB/s (!)
        user    0m0.030s
        sys     0m4.350s
        real    0m50.431s   <==== 9.91 MiB/s (!)
        user    0m0.030s
        sys     0m4.200s

  So 50% more hardware (disks, channels) gives a 60% performance drop
  when using raid1 with sequential reads... and raid1.c:read_balance()
  is the culprit.

Uniform Multi-Platform E-IDE driver Revision: 7.00beta3-.2.4
  
HPT370: IDE controller at PCI slot 00:0d.0  
HPT370: chipset revision 3

PDC20262: IDE controller at PCI slot 00:10.0
PDC20262: chipset revision 1

hde: MAXTOR 4K060H3, ATA DISK drive
hdg: MAXTOR 4K060H3, ATA DISK drive
hdi: MAXTOR 4K060H3, ATA DISK drive

 hde: hde1 hde2 hde3 hde4 < hde5 hde6 hde7 hde8 hde9 hde10 >
 hdg: hdg1 hdg2 hdg3 hdg4 < hdg5 hdg6 hdg7 hdg8 hdg9 hdg10 >
 hdi: hdi1 hdi2 hdi3 hdi4 < hdi5 hdi6 hdi7 hdi8 hdi9 hdi10 >