Re: TIME_WAIT/MSL is 2 seconds--bug/feature?

[Joe Cooper]

Andi Kleen wrote:
> 
> [e-mail] (Joe Cooper) writes:
>
> > According to a small utility called msl_test distributed with the
> > Polygraph web cache benchmarking suite the Linux 2.2.13 TCP stack has a
> > TIME_WAIT of 2 seconds.  As far as I can tell, the TCP specs recommend
> > 60 seconds.  Additionally, this is the required setting for IRCache
> > bakeoffs (and the only way to accurately compare to IRCache results).  
> >
> > Now, my confusiong lies in the fact that a grep through the kernel
> > source gives me this line in 'include/net/tcp.h':
> >
> > #define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to successfully
> >                                 * close the socket, about 60 seconds  */
> >
> 
> You could send the full tcpdump of such a msl test, so that we can
> determine if it is a bug in the stack or in the utility.  I would guess
> for the later, e.g. when you just open a connection to your local host
> and then watch the TIME_WAIT socket after close with netstat -t you'll
> see that it lasts much longer than 2s.
> 
> -Andi

Thanks, Andi for the tips.  I've done a little more digging and testing,
and I've included a tcpdump below.  netstat -t is reporting TIME_WAIT
for 60 seconds after any kind of connection (telnet, ftp, http) just as
you expected, except for the msl_test program.  I've been in contact
with the author of msl_test also, but haven't heard back yet.  However,
I'd be surprised if msl_test is the culprit as it has been tested on
several stacks other than Linux with predictable results (i.e. correctly
reporting the TIME_WAIT).

msl_test uses SYN packets (from a previously established connection) to
find if the server responds (it shouldn't until TIME_WAIT has passed). 
Better docs are here: http://polygraph.ircache.net/doc/msl_test.html

The tcpdump is below, 10.1.0.1 is the msl_test machine running FreeBSD
and 10.1.0.20 is the Linux box running squid on port 3128 (same results
came from httpd on port 80 only with more extraneous stuff in the
output):

16:07:34.061846 arp who-has 10.1.0.20 tell 10.1.0.1
16:07:34.061910 arp reply 10.1.0.20 is-at 0:80:ad:97:f5:60
16:07:34.062042 10.1.0.1.1044 > 10.1.0.20.3128: S 212151377:212151377(0)
win 16384 <mss 1460> (DF) (ttl 64, id 2266)
16:07:34.062277 10.1.0.20.3128 > 10.1.0.1.1044: S 697157277:697157277(0)
ack 212151378 win 32120 <mss 1460> (DF) (ttl 64, id 9625)
16:07:34.062431 10.1.0.1.1044 > 10.1.0.20.3128: . ack 1 win 17520 (DF)
(ttl 64, id 2267)
16:07:34.062509 10.1.0.1.1044 > 10.1.0.20.3128: P 1:17(16) ack 1 win
17520 (DF) (ttl 64, id 2268)
16:07:34.062543 10.1.0.20.3128 > 10.1.0.1.1044: . ack 17 win 32120 (DF)
(ttl 64, id 9626)
16:07:34.064109 10.1.0.20.3128 > 10.1.0.1.1044: P 1:1068(1067) ack 17
win 32120 (DF) (ttl 64, id 9627)
16:07:34.064414 10.1.0.20.3128 > 10.1.0.1.1044: F 1068:1068(0) ack 17
win 32120 (DF) (ttl 64, id 9628)
16:07:34.064453 10.1.0.1.1044 > 10.1.0.20.3128: . ack 1068 win 16453
(DF) (ttl 64, id 2269)
16:07:34.064558 10.1.0.1.1044 > 10.1.0.20.3128: . ack 1069 win 17520
(DF) (ttl 64, id 2270)
16:07:35.064774 10.1.0.1.1044 > 10.1.0.20.3128: F 17:17(0) ack 1069 win
17520 (DF) (ttl 64, id 2287)
16:07:35.064842 10.1.0.20.3128 > 10.1.0.1.1044: . ack 18 win 32120 (DF)
(ttl 64, id 9629)
16:07:36.064957 10.1.0.1.1044 > 10.1.0.20.3128: S 212151377:212151377(0)
win 16384 <mss 1460> (DF) (ttl 64, id 2266)
16:07:36.065021 10.1.0.20.3128 > 10.1.0.1.1044: R
3597810019:3597810019(0) ack 1 win 0 (ttl 255, id 9630)
16:07:37.065602 10.1.0.1.1044 > 10.1.0.20.3128: S 212151377:212151377(0)
win 16384 <mss 1460> (DF) (ttl 64, id 2266)
16:07:37.065671 10.1.0.20.3128 > 10.1.0.1.1044: S 700160844:700160844(0)
ack 212151378 win 32120 <mss 1460> (DF) (ttl 64, id 9631)
16:07:40.221333 10.1.0.20.3128 > 10.1.0.1.1044: S 700160844:700160844(0)
ack 212151378 win 32120 <mss 1460> (DF) (ttl 64, id 9632)
16:07:40.221500 arp who-has 10.1.0.20 tell 10.1.0.1
16:07:40.221524 arp reply 10.1.0.20 is-at 0:80:ad:97:f5:60
16:07:40.221631 10.1.0.1.1044 > 10.1.0.20.3128: R 212151378:212151378(0)
win 0 (ttl 64, id 2289)

I'm afraid I can't quite figure out why this series of events would
equal a TIME_WAIT of 2 seconds, but it does according to msl_test and
netstat -t.  

Any further enlightenment would be most helpful.  Thanks to everyone who
has commented so far.
-- 
Joe Cooper <[e-mail]>
http://www.swelltech.com