Re: [PATCH 2.5.58] new NUMA scheduler: fix

[Michael Hohnbaum]

On Wed, 2003-01-15 at 07:10, Erich Focht wrote:
> Thanks for the patience with the problems in the yesterday patches. I
> resend the patches in the same form. Made following changes:
> 
> - moved NODE_BALANCE_{RATE,MIN,MAX} to topology.h
> - removed the divide in the find_busiest_node() loop (thanks, Martin)
> - removed the modulo (%) in in the cross-node balancing trigger
> - re-added node_nr_running_init() stub, nr_running_init() and comments
>   from Christoph
> - removed the constant factor 4 in find_busiest_node. The
>   find_busiest_queue routine will take care of the case where the
>   busiest_node is running only few processes (at most one per CPU) and
>   return busiest=NULL .

These applied clean and looked good.  I ran numbers for kernels
with patches 1-4 and patches 1-5.  Results below.
> 
> I hope we can start tuning the parameters now. In the basic NUMA
> scheduler part these are:
>   NODE_THRESHOLD     : minimum percentage of node overload to
>                        trigger cross-node balancing
>   NODE_BALANCE_RATE  : arch specific, cross-node balancing is called
>                        after this many intra-node load balance calls

I need to spend a some time staring at this code and putting together
a series of tests to try.  I think the basic mechanism looks good, so
hopefully we are down to finding the best numbers for the architecture.
> 
> In the extended NUMA scheduler the fixed value of NODE_BALANCE_RATE is
> replaced by a variable rate set to :
>   NODE_BALANCE_MIN : if own node load is less then avg_load/2
>   NODE_BALANCE_MAX : if load is larger than avg_load/2
> Together with the reduced number of steals across nodes this might
> help us in achieving equal load among nodes. I'm not aware of any
> simple benchmark which can demonstrate this...
> 
> Regards,
> Erich
> ----
>
results:

stock58h: linux 2.5.58 with cputime stats patch
sched14r2-58: stock58h with latest NUMA sched patches 1 through 4
sched15r2-58: stock58h with latest NUMA sched patches 1 through 5
sched1-4-58: stock58h with previous NUMA sched patches 1 through 4

Kernbench:
                        Elapsed       User     System        CPU
        sched14r2-58    29.488s    284.02s    82.132s    1241.8%
        sched15r2-58    29.778s   282.792s    83.478s    1229.8%
            stock58h    31.656s    305.85s    89.232s    1248.2%
         sched1-4-58    29.886s   287.506s     82.94s      1239%

Schedbench 4:
                        AvgUser    Elapsed  TotalUser   TotalSys
        sched14r2-58      22.50      37.20      90.03       0.94
        sched15r2-58      16.63      24.23      66.56       0.69
            stock58h      27.73      42.80     110.96       0.85
         sched1-4-58      32.86      46.41     131.47       0.85

Schedbench 8:
                        AvgUser    Elapsed  TotalUser   TotalSys
        sched14r2-58      30.27      43.22     242.23       1.75
        sched15r2-58      30.90      42.46     247.28       1.48
            stock58h      45.97      61.87     367.81       2.11
         sched1-4-58      31.39      49.18     251.22       2.15

Schedbench 16:
                        AvgUser    Elapsed  TotalUser   TotalSys
        sched14r2-58      52.78      57.28     844.61       3.70
        sched15r2-58      48.44      65.31     775.25       3.30
            stock58h      60.91      83.63     974.71       6.18
         sched1-4-58      54.31      62.11     869.11       3.84

Schedbench 32:
                        AvgUser    Elapsed  TotalUser   TotalSys
        sched14r2-58      56.60     116.99    1811.56       5.94
        sched15r2-58      56.42     116.75    1805.82       6.45
            stock58h      84.26     195.16    2696.65      16.53
         sched1-4-58      61.49     140.51    1968.06       9.57

Schedbench 64:
                        AvgUser    Elapsed  TotalUser   TotalSys
        sched14r2-58      56.48     232.63    3615.55      16.02
        sched15r2-58      56.38     236.25    3609.03      15.41
            stock58h     123.27     511.77    7889.77      27.78
         sched1-4-58      63.39     266.40    4057.92      20.55

 
-- 

Michael Hohnbaum                      503-578-5486
[e-mail]                   T/L 775-5486