mlock + OOM = infinite loop

[Jim Crilly]

I have a dual Athlon 1.2Ghz box with 1.2G of memory, I decided to run 2 
copies of memtest to test the memory with each testing 629M (the 
existance of memtest86 slipped my mind). With the 2.4.17 and 2.4.18-rc2 
kernels (Andre's IDE amd kdb patches applied) the kernel loops trying to 
free memory so the second mlock can succeed which never happens, I let 
it run for about 2 days while I was away and it never became responsive, 
minus kdb.

But with Rik's rmap12a patch it successfully starts the OOM and kills 
one of the memtest processes, the only strange thing is it kills the 
first instance letting the second succeed, I would think it would be 
nicer to just fail in the second mlock and let the first continue.

Also all the kernels were compiled with SMP support, if that makes a 
difference.

Jim