Re: Asynch I/O broken in 2.2.15

[Jeff V. Merkey]

This is a multi-part message in MIME format.
--------------E4EC02E2C43465EAE2F76295
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

"Stephen C. Tweedie" wrote:
> 
> Hi,
> 
> On Wed, Apr 12, 2000 at 05:50:47PM -0600, Jeff V. Merkey wrote:
> 
> > 1.  Linux 2.2.15 and 2.3.99 do not support true Asynch IO.
> 
> Yes they do.
> 
> > Asynch IO
> > allows a caller to post a very large number of requests, is not **NOT**
> > polled, and should be interrupt driven.
> 
> There is a limit on the total number of IO requests outstanding in
> the Linux kernel at once, but other than that all of the above works
> as required.
> 
> > The first request should "kick"
> > the queue head into the driver, then the driver should feed the next
> > buffer on the IO completion interrupt until the queue head is empty.
> 
> That is exactly what happens, with the exception that kick-starting
> an empty queue requires that the caller first submits the IOs, and
> then runs the disk task queue (as otherwise there is no way for the
> kernel to know if you are immediately going to submit another IO
> which could be merged with the first one you submitted).
> 
> > The way it's implemented in Linux requires that the bottom half ISR be
> > "polled" within wait_on_buffer() or some other function to initiate the
> > I/O and block until the IO returns -- how is this asynchronous I/O.
> 
> No it does not: the bh->b_end_io() function is called asynchronously
> once IO completes.  You do not have to wait_on_buffer if you'd prefer
> to get a callback.
> 
> > 2.  While the semantics appear to be Asynch, the interface is actually
> > synchronous in behavior -- you have to poll the drives after posting IO
> > to start the disks.
> 
> No you don't.
> 
> > The dependency on the lock_kernel() call on 2.2.15
> > guarantees that most callers will block when attempting multiple calls
> > to ll_rw_blk(), so there went the parallelism.  I have noticed that if I
> > hold the kernel lock over the I/O (which is what is happening in
> > filemap.c and buffer.c) then the swapper does not crash the system.
> 
> The global kernel lock is automatically dropped by the scheduler if
> the process holding it goes to sleep.  If you hold the lock and do
> blocking IO, you do not lose parallelism.
> 
> > 3.  Feeding more than a handful of requests at one time to THE LINUX
> > BUFFER CACHE (forget NWFS code - I tried this with just Linux) will
> > cause the swapper process to Ooops and die.  I implemented some very
> > simple code that just calls the buffer cache and tries to do asynch I/O
> > and "**KAPOW!!!*** , the system crashes.  The code is attached.
> 
> No wonder, because your code is buggy.  It does a getblk() to find
> an existing block in the buffer cache, and scribbles on the b_end_io
> field without doing any locking to see if the buffer is already in
> use by somebody else.
> 
> The swapper does true async IO without oopsing.   The page cache uses
> true async IO for readahead without oopsing.

I have another segment of code that uses memory I allocated (and that no
one is using or sharing) and it still Oppes.  If there is a limit of
outstanding requests, then what is it, and why does this code section
Oops.

Also, your statement about the pages being locked is incorrect, these
are block addresses on NetWare partitions, no one else should be using
them or even touching them (and are not).  How can what you said be
true?

Jeff

> 
> --Stephen
--------------E4EC02E2C43465EAE2F76295
Content-Type: application/x-unknown-content-type-cfile;
 name="aio.c"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="aio.c"

DQoNCnZvaWQgZW5kX2FzeW5jaF9pbyhzdHJ1Y3QgYnVmZmVyX2hlYWQgKmJoLCBpbnQgdXB0
b2RhdGUpDQp7DQogICAgcmVnaXN0ZXIgaW50IGk7DQogICAgQVNZTkNIX0lPICppbyA9IChB
U1lOQ0hfSU8gKiliaC0+Yl9kZXZfaWQ7DQoNCiAgICBtYXJrX2J1ZmZlcl91cHRvZGF0ZShi
aCwgdXB0b2RhdGUpOw0KICAgIHVubG9ja19idWZmZXIoYmgpOw0KDQogICAgaWYgKCFpbykN
CiAgICB7DQogICAgICAgTldGU1ByaW50KCJud2ZzOiAgYWlvIGNhbGxiYWNrIGhhcyBOVUxM
IGhhbmRsZVxuIik7DQogICAgICAgcmV0dXJuOw0KICAgIH0NCg0KICAgIGlmICghdGVzdF9i
aXQoQkhfVXB0b2RhdGUsICZiaC0+Yl9zdGF0ZSkpDQogICAgICAgaW8tPmNjb2RlID0gQVNJ
T19JT19FUlJPUjsNCg0KICAgIGlvLT5jb21wbGV0ZSsrOw0KICAgIGlmIChpby0+Y29tcGxl
dGUgPj0gaW8tPmNvdW50KQ0KICAgIHsNCiAgICAgICBmb3IgKGk9MDsgaSA8IGlvLT5jb3Vu
dDsgaSsrKQ0KICAgICAgICAgIHB1dF9iaChpby0+YmhbaV0pOw0KDQogICAgICAgaWYgKGlv
LT5jYWxsX2JhY2tfcm91dGluZSkNCiAgICAgICAgICAoaW8tPmNhbGxfYmFja19yb3V0aW5l
KShpbyk7DQogICAgICAgaW8tPnNpZ25hdHVyZSA9IDA7DQoNCiNpZiAoUFJPRklMRV9BSU8p
DQogICAgICAgcHJvZmlsZV9jb21wbGV0ZSgpOw0KI2VuZGlmDQogICAgfQ0KICAgIHJldHVy
bjsNCn0NCg0KVUxPTkcgYVJlYWREaXNrU2VjdG9ycyhVTE9ORyBkaXNrLCBVTE9ORyBTdGFy
dGluZ0xCQSwgQllURSAqU2VjdG9yLA0KCQkgICAgICBVTE9ORyBzZWN0b3JzLCBVTE9ORyBy
ZWFkQWhlYWQsIEFTWU5DSF9JTyAqaW8pDQp7DQogICAgcmVnaXN0ZXIgVUxPTkcgaSwgaiwg
Ynl0ZXNSZWFkID0gMDsNCiAgICByZWdpc3RlciBVTE9ORyBicHM7DQogICAgcmVnaXN0ZXIg
TldESVNLICpOV0Rpc2s7DQoNCiAgICBpZiAoc2VjdG9ycyA+IDgpDQogICAgICAgcmV0dXJu
IDA7DQoNCiAgICBOV0Rpc2sgPSBTeXN0ZW1EaXNrW2Rpc2tdOw0KICAgIGJwcyA9IE5XRGlz
ay0+Qnl0ZXNQZXJTZWN0b3I7DQoNCiAgICBmb3IgKGk9MDsgaSA8IHNlY3RvcnM7IGkrKykN
CiAgICB7DQogICAgICAgaW8tPmJoW2ldID0gZ2V0X2JoKCk7DQogICAgICAgaWYgKCFpby0+
YmhbaV0pDQogICAgICAgew0KCSAgZm9yIChqPTA7IGogPCBpOyBqKyspDQoJICAgICBpZiAo
aW8tPmJoW2pdKQ0KCQlwdXRfYmgoaW8tPmJoW2pdKTsNCgkgIHJldHVybiAwOw0KICAgICAg
IH0NCiAgICB9DQoNCiAgICBpby0+Y2NvZGUgPSAwOw0KICAgIGlvLT5jb3VudCA9IHNlY3Rv
cnM7DQogICAgaW8tPmNvbXBsZXRlID0gMDsNCiAgICBmb3IgKGk9MDsgaSA8IHNlY3RvcnM7
IGkrKykNCiAgICB7DQogICAgICAgaW8tPmJoW2ldLT5iX3RoaXNfcGFnZSA9IGlvLT5iaFso
aSArIDEpICUgc2VjdG9yc107IC8vIGNyZWF0ZSBjaXJjdWxhciBsaXN0DQogICAgICAgaW8t
PmJoW2ldLT5iX3N0YXRlID0gMDsNCiAgICAgICBpby0+YmhbaV0tPmJfbmV4dF9mcmVlID0g
KHN0cnVjdCBidWZmZXJfaGVhZCAqKU5VTEw7DQogICAgICAgaW8tPmJoW2ldLT5iX3NpemUg
PSA1MTI7DQogICAgICAgaW8tPmJoW2ldLT5iX2RhdGEgPSAoY2hhciAqKSZTZWN0b3JbaSAq
IGJwc107DQogICAgICAgaW8tPmJoW2ldLT5iX2xpc3QgPSBCVUZfQ0xFQU47DQogICAgICAg
aW8tPmJoW2ldLT5iX2RldiA9IChpbnQpTldEaXNrLT5QaHlzaWNhbERpc2tIYW5kbGU7DQog
ICAgICAgaW8tPmJoW2ldLT5iX2Jsb2NrbnIgPSAoU3RhcnRpbmdMQkEgKyBpKTsNCiAgICAg
ICBpby0+YmhbaV0tPmJfY291bnQgPSAxOw0KICAgICAgIGlvLT5iaFtpXS0+Yl9mbHVzaHRp
bWUgPSAwOw0KICAgICAgIGlvLT5iaFtpXS0+Yl9lbmRfaW8gPSBlbmRfYXN5bmNoX2lvOw0K
ICAgICAgIGlvLT5iaFtpXS0+Yl9kZXZfaWQgPSBpbzsNCiAgICAgICBjbGVhcl9iaXQoQkhf
VXB0b2RhdGUsICZpby0+YmhbaV0tPmJfc3RhdGUpOw0KICAgIH0NCg0KICAgIGxvY2tfa2Vy
bmVsKCk7DQogICAgbGxfcndfYmxvY2soUkVBRCwgc2VjdG9ycywgJmlvLT5iaFswXSk7DQog
ICAgdW5sb2NrX2tlcm5lbCgpOw0KDQogICAgcnVuX3Rhc2tfcXVldWUoJnRxX2Rpc2spOw0K
DQojaWYgKERFQlVHX0FJTykNCiAgICBOV0ZTUHJpbnQoImlvIGFzeW5jaCByZWFkIGRpc2st
JWQgbGJhLSVkXG4iLCAoaW50KWRpc2ssIChpbnQpU3RhcnRpbmdMQkEpOw0KI2VuZGlmDQoN
CiAgICBieXRlc1JlYWQgPSAoc2VjdG9ycyAqIGJwcyk7DQogICAgcmV0dXJuIChieXRlc1Jl
YWQpOw0KDQp9DQoNClVMT05HIGFXcml0ZURpc2tTZWN0b3JzKFVMT05HIGRpc2ssIFVMT05H
IFN0YXJ0aW5nTEJBLCBCWVRFICpTZWN0b3IsDQoJCSAgICAgICBVTE9ORyBzZWN0b3JzLCBV
TE9ORyByZWFkQWhlYWQsIEFTWU5DSF9JTyAqaW8pDQp7DQogICAgcmVnaXN0ZXIgVUxPTkcg
aSwgaiwgYnl0ZXNXcml0dGVuID0gMDsNCiAgICByZWdpc3RlciBVTE9ORyBicHM7DQogICAg
cmVnaXN0ZXIgTldESVNLICpOV0Rpc2s7DQoNCiAgICBpZiAoc2VjdG9ycyA+IDgpDQogICAg
ICAgcmV0dXJuIDA7DQoNCiAgICBOV0Rpc2sgPSBTeXN0ZW1EaXNrW2Rpc2tdOw0KICAgIGJw
cyA9IE5XRGlzay0+Qnl0ZXNQZXJTZWN0b3I7DQoNCiAgICBmb3IgKGk9MDsgaSA8IHNlY3Rv
cnM7IGkrKykNCiAgICB7DQogICAgICAgaW8tPmJoW2ldID0gZ2V0X2JoKCk7DQogICAgICAg
aWYgKCFpby0+YmhbaV0pDQogICAgICAgew0KCSAgZm9yIChqPTA7IGogPCBpOyBqKyspDQoJ
ICAgICBpZiAoaW8tPmJoW2pdKQ0KCQlwdXRfYmgoaW8tPmJoW2pdKTsNCgkgIHJldHVybiAw
Ow0KICAgICAgIH0NCiAgICB9DQoNCiAgICBpby0+Y2NvZGUgPSAwOw0KICAgIGlvLT5jb3Vu
dCA9IHNlY3RvcnM7DQogICAgaW8tPmNvbXBsZXRlID0gMDsNCiAgICBmb3IgKGk9MDsgaSA8
IHNlY3RvcnM7IGkrKykNCiAgICB7DQogICAgICAgaW8tPmJoW2ldLT5iX3RoaXNfcGFnZSA9
IGlvLT5iaFsoaSArIDEpICUgc2VjdG9yc107IC8vIGNyZWF0ZSBjaXJjdWxhciBsaXN0DQog
ICAgICAgaW8tPmJoW2ldLT5iX3N0YXRlID0gMDsNCiAgICAgICBpby0+YmhbaV0tPmJfbmV4
dF9mcmVlID0gKHN0cnVjdCBidWZmZXJfaGVhZCAqKSBOVUxMOw0KICAgICAgIGlvLT5iaFtp
XS0+Yl9zaXplID0gNTEyOw0KICAgICAgIGlvLT5iaFtpXS0+Yl9kYXRhID0gKGNoYXIgKikm
U2VjdG9yW2kgKiBicHNdOw0KICAgICAgIGlvLT5iaFtpXS0+Yl9saXN0ID0gQlVGX0NMRUFO
Ow0KICAgICAgIGlvLT5iaFtpXS0+Yl9kZXYgPSAoaW50KU5XRGlzay0+UGh5c2ljYWxEaXNr
SGFuZGxlOw0KICAgICAgIGlvLT5iaFtpXS0+Yl9ibG9ja25yID0gKFN0YXJ0aW5nTEJBICsg
aSk7DQogICAgICAgaW8tPmJoW2ldLT5iX2NvdW50ID0gMTsNCiAgICAgICBpby0+YmhbaV0t
PmJfZmx1c2h0aW1lID0gMDsNCiAgICAgICBpby0+YmhbaV0tPmJfZW5kX2lvID0gZW5kX2Fz
eW5jaF9pbzsNCiAgICAgICBpby0+YmhbaV0tPmJfZGV2X2lkID0gaW87DQogICAgICAgc2V0
X2JpdChCSF9VcHRvZGF0ZSwgJmlvLT5iaFtpXS0+Yl9zdGF0ZSk7DQogICAgICAgc2V0X2Jp
dChCSF9EaXJ0eSwgJmlvLT5iaFtpXS0+Yl9zdGF0ZSk7DQogICAgfQ0KDQogICAgbG9ja19r
ZXJuZWwoKTsNCiAgICBsbF9yd19ibG9jayhXUklURSwgc2VjdG9ycywgJmlvLT5iaFswXSk7
DQogICAgdW5sb2NrX2tlcm5lbCgpOw0KDQogICAgcnVuX3Rhc2tfcXVldWUoJnRxX2Rpc2sp
Ow0KDQojaWYgKERFQlVHX0FJTykNCiAgICBOV0ZTUHJpbnQoImlvIGFzeW5jaCB3cml0ZSBk
aXNrLSVkIGxiYS0lZFxuIiwgKGludClkaXNrLCAoaW50KVN0YXJ0aW5nTEJBKTsNCiNlbmRp
Zg0KDQogICAgYnl0ZXNXcml0dGVuID0gKHNlY3RvcnMgKiBicHMpOw0KICAgIHJldHVybiAo
Ynl0ZXNXcml0dGVuKTsNCg0KfQ0KDQpVTE9ORyBhWmVyb0ZpbGxEaXNrU2VjdG9ycyhVTE9O
RyBkaXNrLCBVTE9ORyBTdGFydGluZ0xCQSwgVUxPTkcgc2VjdG9ycywNCgkJCSAgVUxPTkcg
cmVhZEFoZWFkLCBBU1lOQ0hfSU8gKmlvKQ0Kew0KICAgIHJlZ2lzdGVyIFVMT05HIGksIGos
IGJ5dGVzV3JpdHRlbiA9IDA7DQogICAgcmVnaXN0ZXIgVUxPTkcgYnBzOw0KICAgIHJlZ2lz
dGVyIE5XRElTSyAqTldEaXNrOw0KDQogICAgaWYgKHNlY3RvcnMgPiA4KQ0KICAgICAgIHJl
dHVybiAwOw0KDQogICAgTldEaXNrID0gU3lzdGVtRGlza1tkaXNrXTsNCiAgICBicHMgPSBO
V0Rpc2stPkJ5dGVzUGVyU2VjdG9yOw0KDQogICAgZm9yIChpPTA7IGkgPCBzZWN0b3JzOyBp
KyspDQogICAgew0KICAgICAgIGlvLT5iaFtpXSA9IGdldF9iaCgpOw0KICAgICAgIGlmICgh
aW8tPmJoW2ldKQ0KICAgICAgIHsNCgkgIGZvciAoaj0wOyBqIDwgaTsgaisrKQ0KCSAgICAg
aWYgKGlvLT5iaFtqXSkNCgkJcHV0X2JoKGlvLT5iaFtqXSk7DQoJICByZXR1cm4gMDsNCiAg
ICAgICB9DQogICAgfQ0KDQogICAgaW8tPmNjb2RlID0gMDsNCiAgICBpby0+Y291bnQgPSBz
ZWN0b3JzOw0KICAgIGlvLT5jb21wbGV0ZSA9IDA7DQogICAgZm9yIChpPTA7IGkgPCBzZWN0
b3JzOyBpKyspDQogICAgew0KICAgICAgIGlvLT5iaFtpXS0+Yl90aGlzX3BhZ2UgPSBpby0+
YmhbKGkgKyAxKSAlIHNlY3RvcnNdOyAvLyBjcmVhdGUgY2lyY3VsYXIgbGlzdA0KICAgICAg
IGlvLT5iaFtpXS0+Yl9zdGF0ZSA9IDA7DQogICAgICAgaW8tPmJoW2ldLT5iX25leHRfZnJl
ZSA9IChzdHJ1Y3QgYnVmZmVyX2hlYWQgKikgTlVMTDsNCiAgICAgICBpby0+YmhbaV0tPmJf
c2l6ZSA9IDUxMjsNCiAgICAgICBpby0+YmhbaV0tPmJfZGF0YSA9IChjaGFyICopIFplcm9C
dWZmZXI7DQogICAgICAgaW8tPmJoW2ldLT5iX2xpc3QgPSBCVUZfQ0xFQU47DQogICAgICAg
aW8tPmJoW2ldLT5iX2RldiA9IChpbnQpTldEaXNrLT5QaHlzaWNhbERpc2tIYW5kbGU7DQog
ICAgICAgaW8tPmJoW2ldLT5iX2Jsb2NrbnIgPSAoU3RhcnRpbmdMQkEgKyBpKTsNCiAgICAg
ICBpby0+YmhbaV0tPmJfY291bnQgPSAxOw0KICAgICAgIGlvLT5iaFtpXS0+Yl9mbHVzaHRp
bWUgPSAwOw0KICAgICAgIGlvLT5iaFtpXS0+Yl9lbmRfaW8gPSBlbmRfYXN5bmNoX2lvOw0K
ICAgICAgIGlvLT5iaFtpXS0+Yl9kZXZfaWQgPSBpbzsNCiAgICAgICBzZXRfYml0KEJIX1Vw
dG9kYXRlLCAmaW8tPmJoW2ldLT5iX3N0YXRlKTsNCiAgICAgICBzZXRfYml0KEJIX0RpcnR5
LCAmaW8tPmJoW2ldLT5iX3N0YXRlKTsNCiAgICB9DQoNCiAgICBsb2NrX2tlcm5lbCgpOw0K
ICAgIGxsX3J3X2Jsb2NrKFdSSVRFLCBzZWN0b3JzLCAmaW8tPmJoWzBdKTsNCiAgICB1bmxv
Y2tfa2VybmVsKCk7DQoNCiAgICBydW5fdGFza19xdWV1ZSgmdHFfZGlzayk7DQoNCiNpZiAo
REVCVUdfQUlPKQ0KICAgIE5XRlNQcmludCgiaW8gYXN5bmNoIGZpbGwgZGlzay0lZCBsYmEt
JWRcbiIsIChpbnQpZGlzaywgKGludClTdGFydGluZ0xCQSk7DQojZW5kaWYNCg0KICAgIGJ5
dGVzV3JpdHRlbiA9IChzZWN0b3JzICogYnBzKTsNCiAgICByZXR1cm4gKGJ5dGVzV3JpdHRl
bik7DQp9DQoNCg==
--------------E4EC02E2C43465EAE2F76295--