Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ken Brownfield]

Andrew Morton kindly pointed out that my crack pipe is dangerously empty
and I didn't specify what patch I was talking about.  In my defense, I
was up all last night tracking down the ext3 bug that Andrew fixed right
under me. ;)

I replied to the wrong message, which I've pasted below.  This is wrt
Martin's VM patch per the previous discussion.

Apologies,
-- 
Ken.
[e-mail]

On Fri, Jan 11, 2002 at 02:41:17PM -0600, Ken Brownfield wrote:
| After more testing, my original observations seem to be holding up,
| except that under heavy VM load (e.g., "make -j bzImage") the machine's
| overall performance seems far lower.  For instance, without the patch
| the -j build finishes in ~10 minutes (2x933P3/256MB) but with the patch
| I haven't had the patience to let it finish after more than an hour.
| 
| This is perhaps because the vmscan patch is too aggressively shrinking
| the caches, or causing thrashing in another area?  I'm also noticing
| that the amount of swap used is nearly an order of magnitude higher,
| which doesn't make sense at first glance...  Also, there are extended
| periods where idle CPU is 50-80%.
| 
| Maybe the patch or at least its intent can be merged with Andrea's work
| if applicable?
| 
| Thanks,
| -- 
| Ken.
| [e-mail]

What I SHOULD have replied to:

| Date:   Tue, 8 Jan 2002 09:19:57 -0600
| From: Ken Brownfield <[e-mail]>
| To: Stephan von Krawczynski <[e-mail]>,
|         "M.H.VanLeeuwen" <[e-mail]>, [e-mail]
| Cc: [e-mail]
| Subject: Update Re: [2.4.17/18pre] VM and swap - it's really unusable
| User-Agent: Mutt/1.2.5.1i
| In-Reply-To: <[e-mail]>; from [e-mail] o
| n Sat, Jan 05, 2002 at 04:08:33PM +0100
| Precedence: bulk
| X-Mailing-List:         [e-mail]
| 
| I stayed at work all night banging out tests on a few of our machines
| here.  I took 2.4.18-pre2 and 2.4.18-pre2 with the vmscan patch from
| "M.H.VanLeeuwen" <[e-mail]>.
| 
| My sustained test consisted of this type of load:
| 
|         ls -lR / > /dev/null &
|         /usr/bin/slocate -u -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/
| usr/tmp,/afs,/net" &
|         dd if=/dev/sda3 of=/sda3 bs=1024k &
|         # Hit TUX on this machine repeatedly; html page with 1000 images
|         # Wait for memory to be mostly used by buff/page cache
|         ./a.out &
|         # repeat finished commands -- keep all commands running
|         # after a.out finishes, alow buff/page to refill before repeating
| 
| The a.out in this case is a little program (attached, c.c) to allocate
| and write to an amount of memory equal to physical RAM.  The example I
| chose below is from a 2xP3/600 with 1GB of RAM and 2GB swap.
| 
| This was not a formal benchmark -- I think benchmarks have been
| presented before by other folks, and looking at benchmarks does not
| necessarily indicate the real-world problems that exist.  My intent was
| to reproduce the issues I've been seeing, and then apply the MH (and
| only the MH) patch and observe.
| 
| 2.4.18-pre2
| 
| Once slocate starts and gets close to filling RAM with buffer/page
| cache, kupdated and kswapd have periodic spikes of 50-100% CPU.
| 
| When a.out starts, kswapd and kupdated begin to eat significant portions
| of CPU (20-100%) and I/O becomes more and more sluggish as a.out
| allocates.
| 
| When a.out uses all free RAM and should begin eating cache, significant
| swapping begins and cache is not decreased significantly until the
| machine goes 100-200MB into swap.
| 
| Here are two readprofile outputs, sorted by ticks and load.
| 
| 229689 default_idle                             4417.0962
|   4794 file_read_actor                           18.4385
|    405 __rdtsc_delay                             14.4643
|   3763 do_anonymous_page                         14.0410
|   3796 statm_pgd_range                            9.7835
|   1535 prune_icache                               6.9773
|    153 __free_pages                               4.7812
|   1420 create_bounce                              4.1765
|    583 sym53c8xx_intr                             3.9392
|    221 atomic_dec_and_lock                        2.7625
|   5214 generic_file_write                         2.5659
| 
| 273464 total                                      0.1903
| 234168 default_idle                             4503.2308
|   5298 generic_file_write                         2.6073
|   4868 file_read_actor                           18.7231
|   3799 statm_pgd_range                            9.7912
|   3763 do_anonymous_page                         14.0410
|   1535 prune_icache                               6.9773
|   1526 shrink_cache                               1.6234
|   1469 create_bounce                              4.3206
|    643 rmqueue                                    1.1320
|    591 sym53c8xx_intr                             3.9932
|    505 __make_request                             0.2902
| 
| 
| 2.4.18-pre2 with MH
| 
| With the MH patch applied, the issues I witnessed above did not seem to
| reproduce.  Memory allocation under pressure seemed faster and smoother.
| kswapd never went above 5-15% CPU.  When a.out allocated memory, it did
| not begin swapping until buffer/page cache had been nearly completely
| cannibalized.
| 
| And when a.out caused swapping, it was controlled and behaved like you
| would expect the VM to bahave -- slowly swapping out unused pages
| instead of large swap write-outs without the patch.
| 
| Martin, have you done throughput benchmarks with MH/rmap/aa, BTW?
| 
| But both kernels still seem to be sluggish when it comes to doing small
| I/O operations (vi, ls, etc) during heavy swapping activity.
| 
| Here are the readprofile results:
| 
| 206243 default_idle                             3966.2115
|   6486 file_read_actor                           24.9462
|    409 __rdtsc_delay                             14.6071
|   2798 do_anonymous_page                         10.4403
|    185 __free_pages                               5.7812
|   1846 statm_pgd_range                            4.7577
|    469 sym53c8xx_intr                             3.1689
|    176 atomic_dec_and_lock                        2.2000
|    349 end_buffer_io_async                        1.9830
|    492 refill_inactive                            1.8358
|     94 system_call                                1.8077
| 
| 245776 total                                      0.1710
| 216238 default_idle                             4158.4231
|   6486 file_read_actor                           24.9462
|   2799 do_anonymous_page                         10.4440
|   1855 statm_pgd_range                            4.7809
|   1611 generic_file_write                         0.7928
|    839 __make_request                             0.4822
|    820 shrink_cache                               0.7374
|    540 rmqueue                                    0.9507
|    534 create_bounce                              1.5706
|    492 refill_inactive                            1.8358
|    487 sym53c8xx_intr                             3.2905
| 
| 
| There may be significant differences in the profile outputs for those
| with VM fu.  
| 
| Summary: MH swaps _after_ cache has been properly cannibalized, and
| swapping activity starts when expected and is properly throttled.
| kswapd and kupdated don't seem to go into berserk 100% CPU mode.
| 
| At any rate, I now have the MH patch (and Andrew Morton's mini-ll and
| read-latency2 patches) in production, and I like what I see so far.  I'd
| vote for them to go into 2.4.18, IMHO.  Maybe the full low-latency patch
| if it's not truly 2.5 material.
| 
| My next cook-off will be with -aa and rmap, although if the rather small
| MH patch fixes my last issues it may be worth putting all VM effort into
| a 2.5 VM cook-off. :)  Hopefully the useful stuff in -aa can get pulled
| in at some point soon, though.
| 
| Thanks much to Martin H. VanLeeuwen for his patch and Stephan von
| Krawczynski for his recommendations.  I'll let MH cook for a while and
| I'll follow up later.
| -- 
| Ken.
| [e-mail]
| 
| c.c:
| 
| #include <stdio.h>
| 
| #define MB_OF_RAM 1024
| 
| int
| main()
| {
|         long stuffsize = MB_OF_RAM * 1048576 ;
|         char *stuff ;
| 
|         if ( stuff = (char *)malloc( stuffsize ) ) {
|                 long chunksize = 1048576 ;
|                 long c ;
| 
|                 for ( c=0 ; c<chunksize ; c++ )
|                         *(stuff+c) = '\0' ;
|                 /* hack; last chunk discarded if stuffsize%chunksize != 0 */
|                 for ( ; (c+chunksize)<stuffsize ; c+=chunksize )
|                         memcpy( stuff+c, stuff, chunksize );
|         
|                 sleep( 120 );
|         }
|         else
|                 printf("OOPS\n");
| 
|         exit( 0 );
| }
| 
| 
| On Sat, Jan 05, 2002 at 04:08:33PM +0100, Stephan von Krawczynski wrote:
| [...]
| | I am pretty impressed by Martins test case where merely all VM patches fail
| | with the exception of his own :-) The thing is, this test is not of nature
| | "very special" but more like "system driven to limit by normal processes". And
| | this is the real interesting part about it.
| [...]
|