Re: 2.2 to 2.4... serious TCP send slowdowns

[Hayden Myers]

On 20 Jul 2002, Alan Cox wrote:

> Your buffers are way too small buf_cnt wants to be probably 60K or
> higher. Making it large ensures one write syscall will fill all
> available space in the queue immediately drastically reducing syscall
> and wakeup rates. Also avoiding breaks in streaming.

I've played around with changing the buf_cnt size and tests in house have
surprisingly shown a slight slowdown when increasing it to 64k.  This is
most likely inconclusive but it didn't seem to make a large difference.   
I also tried to do away with the read and writen syscalls and replace them
with a sendfile call but this seems to have made things even slower.  

> 
> Without tcpdump data its hard to guess
> 
Tcpdump output is where I'm seeing the difference in the clients receive
window.  Below is tcpdump from the server 

[root@install spinbox]# /usr/sbin/tcpdump src port 80
tcpdump: listening on eth0
11:37:21.003009 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: S 273731802:273731802(0) ack
2533363500 win 5792 <mss 1460,sackOK,timestamp 25697 104440615,nop,wscale
0> (DF)
11:37:21.006489 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: . ack 302 win 6432
<nop,nop,timestamp 25698 104440615> (DF)
11:37:21.009357 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: P 1:16(15) ack 302 win 6432
<nop,nop,timestamp 25698 104440615> (DF)
11:37:21.009529 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: P 16:123(107) ack 302 win 6432
<nop,nop,timestamp 25698 104440616> (DF)
11:37:21.009696 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: P 123:263(140) ack 302 win 6432
<nop,nop,timestamp 25698 104440616> (DF)
11:37:21.010081 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: . 263:1711(1448) ack 302 win
6432 <nop,nop,timestamp 25698 104440616> (DF)
11:37:21.010116 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: . 1711:3159(1448) ack 302 win
6432 <nop,nop,timestamp 25698 104440616> (DF)11:37:21.010687
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53687: P
3159:4607(1448) ack 302 win 6432 <nop,nop,timestamp 25698 104440616>
(DF)11:37:21.010698 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: . 4607:6055(1448) ack 302 win
6432 <nop,nop,timestamp 25698 104440616> (DF)11:37:21.010726
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53687: .
6055:7503(1448) ack 302 win 6432 <nop,nop,timestamp 25698 104440616>
(DF)11:37:21.010736 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: P 7503:8951(1448) ack 302 win
6432 <nop,nop,timestamp 25698 104440616> (DF)11:37:21.011557
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53687: .
8951:10399(1448) ack 302 win 6432 <nop,nop,timestamp 25698 104440616> (DF)
11:37:21.011571 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: . 10399:11847(1448) ack 302 win
6432 <nop,nop,timestamp 25698 104440616> (DF)
11:37:21.011583 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: FP 11847:12744(897) ack 302 win
6432 <nop,nop,timestamp 25698 104440616> (DF)
11:37:21.058316 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: S 265781655:265781655(0) ack
2534779761 win 5792 <mss 1460,sackOK,timestamp 25703 104440621,nop,wscale
0> (DF)
11:37:21.059682 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: . ack 334 win 6432
<nop,nop,timestamp 25703 104440621> (DF)
11:37:21.061403 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: P 1:16(15) ack 334 win 6432
<nop,nop,timestamp 25703 104440621> (DF)
11:37:21.061574 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: P 16:123(107) ack 334 win 6432
<nop,nop,timestamp 25703 104440621> (DF)
11:37:21.061732 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: P 123:263(140) ack 334 win 6432
<nop,nop,timestamp 25703 104440621> (DF)
11:37:21.061973 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: . 263:1711(1448) ack 334 win
6432 <nop,nop,timestamp 25703 104440621> (DF)
11:37:21.062000 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: . 1711:3159(1448) ack 334 win
6432 <nop,nop,timestamp 25703 104440621> (DF)11:37:21.062572
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53688: P
3159:4607(1448) ack 334 win 6432 <nop,nop,timestamp 25703 104440621>
(DF)11:37:21.062583 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: . 4607:6055(1448) ack 334 win
6432 <nop,nop,timestamp 25703 104440621> (DF)11:37:21.062611
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53688: .
6055:7503(1448) ack 334 win 6432 <nop,nop,timestamp 25703 104440621>
(DF)11:37:21.062619 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: P 7503:8951(1448) ack 334 win
6432 <nop,nop,timestamp 25703 104440621> (DF)11:37:21.063147
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53688: .
8951:10399(1448) ack 334 win 6432 <nop,nop,timestamp 25703 104440621> (DF)
11:37:21.063156 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: . 10399:11847(1448) ack 334 win
6432 <nop,nop,timestamp 25703 104440621> (DF)
11:37:21.063167 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: FP 11847:12744(897) ack 334 win
6432 <nop,nop,timestamp 25703 104440621> (DF)
11:37:21.093947 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53687: . ack 303 win 6432
<nop,nop,timestamp 25707 104440624> (DF)
11:37:21.112002 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53688: . ack 335 win 6432
<nop,nop,timestamp 25708 104440626> (DF)

According to the tcpdump manpage win 6432 is the number of bytes of
receive buffer space available the other direction of the connection. 

Below is a tcpdump session for the same request on the same client but
with the server on a 2.2.20 kernel.

[root@install spinbox]# /usr/sbin/tcpdump src port 80
tcpdump: listening on eth0
11:45:00.379901 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: S 754852391:754852391(0) ack
2999938034 win 30660 <mss 1460,sackOK,timestamp 11434 104486504,nop,wscale
0> (DF)
11:45:00.383374 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: . ack 302 win 30660
<nop,nop,timestamp 11435 104486504> (DF)
11:45:00.386345 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: P 1:16(15) ack 302 win 31856
<nop,nop,timestamp 11435 104486504> (DF)
11:45:00.386571 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: P 16:47(31) ack 302 win 31856
<nop,nop,timestamp 11435 104486504> (DF)
11:45:00.386855 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: P 47:73(26) ack 302 win 31856
<nop,nop,timestamp 11435 104486504> (DF)
11:45:00.387116 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: P 73:154(81) ack 302 win 31856
<nop,nop,timestamp 11435 104486504> (DF)
11:45:00.387314 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: P 154:263(109) ack 302 win 31856
<nop,nop,timestamp 11435 104486504> (DF)
11:45:00.427821 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: S 754496145:754496145(0) ack
3011203697 win 30660 <mss 1460,sackOK,timestamp 11439 104486508,nop,wscale
0> (DF)
11:45:00.429069 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: . ack 334 win 30660
<nop,nop,timestamp 11439 104486508> (DF)
11:45:00.435172 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: P 1:16(15) ack 334 win 31856
<nop,nop,timestamp 11440 104486508> (DF)
11:45:00.435392 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: P 16:47(31) ack 334 win 31856
<nop,nop,timestamp 11440 104486509> (DF)
11:45:00.435636 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: P 47:73(26) ack 334 win 31856
<nop,nop,timestamp 11440 104486509> (DF)
11:45:00.435825 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: P 73:210(137) ack 334 win 31856
<nop,nop,timestamp 11440 104486509> (DF)
11:45:00.436047 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: P 210:263(53) ack 334 win 31856
<nop,nop,timestamp 11440 104486509> (DF)
11:45:00.468318 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: P 263:1711(1448) ack 302 win
31856 <nop,nop,timestamp 11443 104486504> (DF)11:45:00.468380
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53702: P
1711:3159(1448) ack 302 win 31856 <nop,nop,timestamp 11443 104486504> (DF)
11:45:00.468422 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: P 3159:4607(1448) ack 302 win
31856 <nop,nop,timestamp 11443 104486504> (DF)
11:45:00.468467 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: . 4607:6055(1448) ack 302 win
31856 <nop,nop,timestamp 11443 104486504> (DF)
11:45:00.468945 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: . 6055:7503(1448) ack 302 win
31856 <nop,nop,timestamp 11443 104486512> (DF)
11:45:00.468959 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: . 7503:8951(1448) ack 302 win
31856 <nop,nop,timestamp 11443 104486512> (DF)
11:45:00.468980 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: . 8951:10399(1448) ack 302 win
31856 <nop,nop,timestamp 11443 104486512> (DF)
11:45:00.469235 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: . 10399:11847(1448) ack 302 win
31856 <nop,nop,timestamp 11443 104486512> (DF)
11:45:00.469250 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: FP 11847:12744(897) ack 302 win
31856 <nop,nop,timestamp 11443 104486512> (DF)
11:45:00.470040 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: P 263:1711(1448) ack 334 win
31856 <nop,nop,timestamp 11443 104486509> (DF)11:45:00.470077
install.skyline.net.http > leg-66-247-99-8-RLY.sprinthome.com.53703: P
1711:3159(1448) ack 334 win 31856 <nop,nop,timestamp 11443 104486509> (DF)
11:45:00.470120 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: P 3159:4607(1448) ack 334 win
31856 <nop,nop,timestamp 11443 104486509> (DF)
11:45:00.470257 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: . 4607:6055(1448) ack 334 win
31856 <nop,nop,timestamp 11443 104486509> (DF)
11:45:00.470638 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: . 6055:7503(1448) ack 334 win
31856 <nop,nop,timestamp 11443 104486513> (DF)
11:45:00.470654 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: . 7503:8951(1448) ack 334 win
31856 <nop,nop,timestamp 11443 104486513> (DF)
11:45:00.470675 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: . 8951:10399(1448) ack 334 win
31856 <nop,nop,timestamp 11443 104486513> (DF)
11:45:00.471015 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: . 10399:11847(1448) ack 334 win
31856 <nop,nop,timestamp 11443 104486513> (DF)
11:45:00.471048 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: FP 11847:12744(897) ack 334 win
31856 <nop,nop,timestamp 11443 104486513> (DF)
11:45:00.487840 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53702: . ack 303 win 31856
<nop,nop,timestamp 11445 104486514> (DF)
11:45:00.532997 install.skyline.net.http >
leg-66-247-99-8-RLY.sprinthome.com.53703: . ack 335 win 31856
<nop,nop,timestamp 11450 104486519> (DF)

Looks to be about the same amount of data going in each packet but the
clients receive window is much larger for some reason.  I tried tweaking
all of the window parameters I can find both in the kernel and in proc but
none seem to increase this number.  Is there a reason this number is much
lower in a vanilla 2.4 kernel?  I'm not very familiar with the 2.4 and
it's ways yet but I've been reading abou the autotuning algorithms that
have been put in place and lay suspect to them for the differences.  Any
ideas as to why the windows are very different?

Hayden Myers	
Support Manager
Skyline Network Technologies	
[e-mail]
[phone] option 2