Degrading disk read performance under 2.2.16

[Corin Hartland-Swann]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to [e-mail] for more info.

--1876187939-1261560904-965921409=:20743
Content-Type: TEXT/PLAIN; charset=US-ASCII

Hi there,

Firstly, IANAKH (I am not a kernel-hacker), so please CC: everything to
linux-raid. And be gentle with me :)

I have been experimenting with RAID for some time, and found that the big
read-access slowdown between kernels 2.2.15 and 2.2.16 on RAID-1 mirrors
was because of the raw disk accesses (on IDE) rather than a RAID problem.
I benchmarked single IDE disk performance on the following setup:

Intel 810E Chipset Motherboard (CA810EAL), Pentium III-667, 32M RAM,
Maxtor DiamondMax Plus 40 40.9GB UDMA66 Disk, Model 54098U8

I have attached the (edited) kernel config I used for all 2.2 kernels FYI.

I used tiotest to benchmark, using a file size of 256MB, block size of 4K,
and with 1, 2, 4, 16, 32 threads. The performance starts to get hit as
soon as you have multiple threads reading simultaneously, and degrades
much more quickly on 2.2.16. Strangely, writes do not degrade.

Size is MB, BlkSz is Bytes, Read and Write are MB/sec, Seeks are Seeks/sec

==> 2.2.15 <==
 Dir   Size   BlkSz  Thr#  Read (CPU%)   Write (CPU%)   Seeks (CPU%)
----- ------ ------- ---- ------------- -------------- --------------
/mnt/  256    4096    1   27.1371 10.3% 26.7979 23.0%  146.187 0.95%
/mnt/  256    4096    2   27.1219 10.7% 26.6606 23.2%  142.233 0.60%
/mnt/  256    4096    4   26.9915 10.6% 26.4289 22.9%  142.789 0.50%
/mnt/  256    4096    16  26.4320 10.5% 26.1310 23.0%  147.424 0.52%
/mnt/  256    4096    32  25.3407 10.1% 25.6822 22.7%  150.750 0.57%

==> 2.2.16 <==
 Dir   Size   BlkSz  Thr#  Read (CPU%)   Write (CPU%)   Seeks (CPU%)
----- ------ ------- ---- ------------- -------------- --------------
/mnt/  256    4096    1   27.0948 10.5% 25.9272 22.5%  147.264 0.83%
/mnt/  256    4096    2   20.5753 8.42% 25.9316 22.7%  143.113 0.56%
/mnt/  256    4096    4   13.0397 6.09% 25.7723 22.7%  143.619 0.51%
/mnt/  256    4096    16  9.38573 5.40% 25.6285 23.1%  147.815 0.51%
/mnt/  256    4096    32  6.96358 5.90% 25.2889 22.9%  151.182 0.57%

The performance of 2.2.17-pre9 is basically exactly the same as 2.2.16,
except that it died with do_vm_free_pages() when trying to run it with 64
threads.

I had a look in drivers/block/ide.c, and the only big change to seems to
be adding a new element 'expiry' to ide_hwgroup_t. ide_set_handler() has
been changed to take it as an extra argument, however, the only place that
I can find calls to ide_set_handler() with a non-NULL expiry is in
ide-cd.c, (I haven't tested CD-ROMs to see if there's a slow-down there as
well). All calls to ide_set_handler() in ide.c and ide-disk.c specify
expiry as NULL.

The only place where hwgroup->expiry is set is in ide_set_handler (always
passed NULL by the HD routines). The only place that it is examined is in
ide_timer_expiry() which should not execute any more code if expiry is
NULL.

I also tested 2.4.0-test5, and performance there sucks with one thread,
but doesn't degrade any more quickly than 2.2.15

==> 2.4.0-test5 <==

 Dir   Size   BlkSz  Thr#  Read (CPU%)   Write (CPU%)   Seeks (CPU%)
----- ------ ------- ---- ------------- -------------- --------------
/mnt/  256    4096    1   8.15398 49.6% 7.64104 71.2%  125.675 5.90%
/mnt/  256    4096    2   8.01528 56.1% 7.63085 69.8%  123.076 5.59%
/mnt/  256    4096    4   7.79720 61.6% 7.62060 68.1%  125.437 5.75%
/mnt/  256    4096    16  7.64617 65.2% 7.60466 62.5%  129.431 5.83%
/mnt/  256    4096    32  7.60580 67.6% 7.58702 51.7%  132.791 5.95%

I haven't tried any of this with SCSI disks, so the problem may well lie
in VFS or the VM subsystem (I know there are problems there as well). I
don't know anything about those, so I guess I should just leave it to the
professionals now :)

Thanks,

Corin

/------------------------+-------------------------------------\
| Corin Hartland-Swann   | Direct: [phone]        |
| Commerce Internet Ltd  | Mobile: [phone]        |
| 22 Cavendish Buildings |    Tel: [phone]        |
| Gilbert Street         |    Fax: [phone]        |
| Mayfair                |    Web: http://www.commerce.uk.net/ |
| London W1K 5HJ         | E-Mail: [e-mail]        |
\------------------------+-------------------------------------/

--1876187939-1261560904-965921409=:20743
Content-Type: TEXT/PLAIN; charset=US-ASCII; name=kernel-config
Content-Transfer-Encoding: BASE64
Content-ID: <[e-mail]>
Content-Description: 
Content-Disposition: attachment; filename=kernel-config

Q09ORklHX0VYUEVSSU1FTlRBTD15DQpDT05GSUdfTTY4Nj15DQpDT05GSUdf
WDg2X1dQX1dPUktTX09LPXkNCkNPTkZJR19YODZfSU5WTFBHPXkNCkNPTkZJ
R19YODZfQlNXQVA9eQ0KQ09ORklHX1g4Nl9QT1BBRF9PSz15DQpDT05GSUdf
WDg2X1RTQz15DQpDT05GSUdfWDg2X0dPT0RfQVBJQz15DQpDT05GSUdfMUdC
PXkNCkNPTkZJR19NVFJSPXkNCkNPTkZJR19NT0RVTEVTPXkNCkNPTkZJR19N
T0RWRVJTSU9OUz15DQpDT05GSUdfS01PRD15DQpDT05GSUdfTkVUPXkNCkNP
TkZJR19QQ0k9eQ0KQ09ORklHX1BDSV9HT0FOWT15DQpDT05GSUdfUENJX0JJ
T1M9eQ0KQ09ORklHX1BDSV9ESVJFQ1Q9eQ0KQ09ORklHX1BDSV9RVUlSS1M9
eQ0KQ09ORklHX1BDSV9PTERfUFJPQz15DQpDT05GSUdfU1lTVklQQz15DQpD
T05GSUdfU1lTQ1RMPXkNCkNPTkZJR19CSU5GTVRfQU9VVD15DQpDT05GSUdf
QklORk1UX0VMRj15DQpDT05GSUdfQklORk1UX01JU0M9eQ0KQ09ORklHX0JM
S19ERVZfRkQ9eQ0KQ09ORklHX0JMS19ERVZfSURFPXkNCkNPTkZJR19CTEtf
REVWX0lERURJU0s9eQ0KQ09ORklHX0JMS19ERVZfSURFQ0Q9eQ0KQ09ORklH
X0JMS19ERVZfQ01ENjQwPXkNCkNPTkZJR19CTEtfREVWX1JaMTAwMD15DQpD
T05GSUdfQkxLX0RFVl9JREVQQ0k9eQ0KQ09ORklHX0JMS19ERVZfSURFRE1B
PXkNCkNPTkZJR19JREVETUFfQVVUTz15DQpDT05GSUdfQkxLX0RFVl9MT09Q
PXkNCkNPTkZJR19CTEtfREVWX01EPXkNCkNPTkZJR19NRF9MSU5FQVI9eQ0K
Q09ORklHX01EX1NUUklQRUQ9eQ0KQ09ORklHX01EX01JUlJPUklORz15DQpD
T05GSUdfTURfUkFJRDU9eQ0KQ09ORklHX01EX0JPT1Q9eQ0KQ09ORklHX1BB
UklERV9QQVJQT1JUPXkNCkNPTkZJR19QQUNLRVQ9eQ0KQ09ORklHX1VOSVg9
eQ0KQ09ORklHX0lORVQ9eQ0KQ09ORklHX0lQX0FMSUFTPXkNCkNPTkZJR19T
S0JfTEFSR0U9eQ0KQ09ORklHX1NDU0k9eQ0KQ09ORklHX0JMS19ERVZfU0Q9
eQ0KQ09ORklHX0NIUl9ERVZfU1Q9eQ0KQ09ORklHX0JMS19ERVZfU1I9eQ0K
Q09ORklHX1NDU0lfQ09OU1RBTlRTPXkNCkNPTkZJR19TQ1NJX0FJQzdYWFg9
eQ0KQ09ORklHX0FJQzdYWFhfQ01EU19QRVJfREVWSUNFPTgNCkNPTkZJR19B
SUM3WFhYX1JFU0VUX0RFTEFZPTUNCkNPTkZJR19TQ1NJX0FEVkFOU1lTPXkN
CkNPTkZJR19TQ1NJX0lOSUExMDA9eQ0KQ09ORklHX05FVERFVklDRVM9eQ0K
Q09ORklHX0RVTU1ZPXkNCkNPTkZJR19ORVRfRVRIRVJORVQ9eQ0KQ09ORklH
X05FVF9WRU5ET1JfM0NPTT15DQpDT05GSUdfVk9SVEVYPXkNCkNPTkZJR19O
RVRfRUlTQT15DQpDT05GSUdfRUVYUFJFU1NfUFJPMTAwPXkNCkNPTkZJR19W
SUFfUkhJTkU9eQ0KQ09ORklHX1ZUPXkNCkNPTkZJR19WVF9DT05TT0xFPXkN
CkNPTkZJR19TRVJJQUw9eQ0KQ09ORklHX1NFUklBTF9DT05TT0xFPXkNCkNP
TkZJR19VTklYOThfUFRZUz15DQpDT05GSUdfVU5JWDk4X1BUWV9DT1VOVD0y
MDQ4DQpDT05GSUdfTU9VU0U9eQ0KQ09ORklHX1BTTU9VU0U9eQ0KQ09ORklH
XzgyQzcxMF9NT1VTRT15DQpDT05GSUdfUlRDPXkNCkNPTkZJR19BVVRPRlNf
RlM9eQ0KQ09ORklHX0lTTzk2NjBfRlM9eQ0KQ09ORklHX1BST0NfRlM9eQ0K
Q09ORklHX0RFVlBUU19GUz15DQpDT05GSUdfRVhUMl9GUz15DQpDT05GSUdf
TkZTX0ZTPXkNCkNPTkZJR19TVU5SUEM9eQ0KQ09ORklHX0xPQ0tEPXkNCkNP
TkZJR19WR0FfQ09OU09MRT15DQpDT05GSUdfVklERU9fU0VMRUNUPXkNCg==
--1876187939-1261560904-965921409=:20743--