132

I run a particular program on linux which sometimes crashes. If you open it quickly after that, it listens on socket 49201 instead of 49200 as it did the first time. netstat reveals that 49200 is in a TIME_WAIT state.

Is there a program you can run to immediately force that socket move out of the TIME_WAIT state?

Rehan
  • 1,423
  • 2
  • 10
  • 5
  • 1
    If you are here due to ["too many `TIME_WAIT` on server"](http://stackoverflow.com/q/1803566/632951), just [skip through](http://serverfault.com/a/329851/87017) the first three answers which avoid the question instead of answering it. – Pacerier Jan 23 '16 at 04:34
  • I accepted what was the 4th answer so it should be at the top now – Rehan Jun 16 '20 at 17:25
  • Crosslink: related question on Stack Overflow: [c - Error: Address already in use while binding socket with address but the port number is shown free by `netstat` - Stack Overflow](https://stackoverflow.com/questions/5106674/error-address-already-in-use-while-binding-socket-with-address-but-the-port-num) – user202729 Dec 22 '22 at 08:44

8 Answers8

165
/etc/init.d/networking restart

Let me elaborate. Transmission Control Protocol (TCP) is designed to be a bidirectional, ordered, and reliable data transmission protocol between two end points (programs). In this context, the term reliable means that it will retransmit the packets if it gets lost in the middle. TCP guarantees the reliability by sending back Acknowledgment (ACK) packets back for a single or a range of packets received from the peer.

This goes same for the control signals such as termination request/response. RFC 793 defines the TIME-WAIT state to be as follows:

TIME-WAIT - represents waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request.

See the following TCP state diagram: alt text

TCP is a bidirectional communication protocol, so when the connection is established, there is not a difference between the client and the server. Also, either one can call quits, and both peers needs to agree on closing to fully close an established TCP connection.

Let's call the first one to call the quits as the active closer, and the other peer the passive closer. When the active closer sends FIN, the state goes to FIN-WAIT-1. Then it receives an ACK for the sent FIN and the state goes to FIN-WAIT-2. Once it receives FIN also from the passive closer, the active closer sends the ACK to the FIN and the state goes to TIME-WAIT. In case the passive closer did not received the ACK to the second FIN, it will retransmit the FIN packet.

RFC 793 sets the TIME-OUT to be twice the Maximum Segment Lifetime, or 2MSL. Since MSL, the maximum time a packet can wander around Internet, is set to 2 minutes, 2MSL is 4 minutes. Since there is no ACK to an ACK, the active closer can't do anything but to wait 4 minutes if it adheres to the TCP/IP protocol correctly, just in case the passive sender has not received the ACK to its FIN (theoretically).

In reality, missing packets are probably rare, and very rare if it's all happening within the LAN or within a single machine.

To answer the question verbatim, How to forcibly close a socket in TIME_WAIT?, I will still stick to my original answer:

/etc/init.d/networking restart

Practically speaking, I would program it so it ignores TIME-WAIT state using SO_REUSEADDR option as WMR mentioned. What exactly does SO_REUSEADDR do?

This socket option tells the kernel that even if this port is busy (in
the TIME_WAIT state), go ahead and reuse it anyway. If it is busy, but with another state, you will still get an address already in use error. It is useful if your server has been shut down, and then restarted right away while sockets are still active on its port. You should be aware that if any unexpected data comes in, it may confuse your server, but while this is possible, it is not likely.

Community
  • 1
Eugene Yokota
  • 1,919
  • 3
  • 17
  • 12
  • 12
    Great answer, but not the correct answer to his question. Restarting the networking would work, but then so would rebooting, so this can't be right. – Chris Huang-Leaver Aug 06 '09 at 17:03
  • 3
    @Chris Huang-Leaver, the question is "Is there a program you can run to immediately force that socket move out of the TIME_WAIT state?" if rebooting could be considered running a program, then it too would be a right answer. Why do you think this can't be right? – Eugene Yokota Aug 06 '09 at 17:59
  • 9
    WMR has the most useful answer (which is what I do when I run into this kind of issue). Restarting the network is too drastic to be solution, and could take longer than simply waiting for the timeout.The correct answer to his question is 'No', but SO won't let you type two letter answers :-) – Chris Huang-Leaver Sep 02 '09 at 11:23
  • 7
    oh okay, next time some process hangs on SIGTERM I'll just smash my computer instead of fixing it. – Longpoke May 13 '10 at 16:13
  • 1
    The generalization of this is "restart network services". The specific location `/etc/init.d/networking` is platform-specific (Debian?) so the precise command line will be different (sometimes rather radically so) for other systems. I agree with other commenters that this seems like severe overkill and obviously disruptive for any unrelated network services. – tripleee Jun 26 '19 at 08:22
54

I don't know if you have the source code of that particular program you're running, but if so you could just set SO_REUSEADDR via setsockopt(2) which allows you to bind on the same local address even if the socket is in TIME_WAIT state (unless that socket is actively listening, see socket(7)).

For more information on the TIME_WAIT state see the Unix socket FAQ.

WMR
  • 871
  • 5
  • 6
  • but i did not get the already bound error. when i execute the program again it listens in post(123456) also i can see a that the system is showing TIME_WAIT for that port but still i can connect. why? – Jayapal Chandran Sep 27 '10 at 08:26
  • 3
    Even with SO_REUSEADDR, it is still possible to get the "Address already in use" error. For details, refer to http://hea-www.harvard.edu/~fine/Tech/addrinuse.html. – Jingguo Yao Jun 27 '13 at 02:16
  • @WMR `SO_REUSEADDR` does not "close" a socket. It merely enables you to reuse those that are already opened. So the question is still "How to forcibly close a socket in `TIME_WAIT`?" – Pacerier Jan 23 '16 at 04:15
  • This is the correct answer, but the question was not totally correct. At least solved my problem nicely (not like restart the whole network breaking all other connections as well). – V-Mark Dec 29 '18 at 11:08
  • `SO_REUSEADDR` will let `bind()` proceed; but if you then want to listen to that socket, `listen()` will return `EADDRINUSE` all the same. In other words, this answer may help client software using ephemeral ports, but doesn't solve the issue for server software. – Will Jun 16 '19 at 04:06
34

As far as I know there is no way to forcibly close the socket outside of writing a better signal handler into your program, but there is a /proc file which controls how long the timeout takes. The file is

/proc/sys/net/ipv4/tcp_tw_recycle

and you can set the timeout to 1 second by doing this:

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

However, this page contains a warning about possible reliability issues when setting this variable.

There is also a related file

/proc/sys/net/ipv4/tcp_tw_reuse

which controls whether TIME_WAIT sockets can be reused (presumably without any timeout).

Incidentally, the kernel documentation warns you not to change either of these values without 'advice/requests of technical experts'. Which I am not.

The program must have been written to attempt a binding to port 49200 and then increment by 1 if the port is already in use. Therefore, if you have control of the source code, you could change this behaviour to wait a few seconds and try again on the same port, instead of incrementing.

Leigh Caldwell
  • 441
  • 3
  • 3
  • think the second two examples should be s/rw/tw/ I'd edit, but lack enough rep. –  Oct 06 '09 at 03:26
  • 1
    Taken from kernel documentation: Caution. Both tcp_tw_recycle and tcp_tw_reuse can cause problems. You should not enable either without understanding network topology in between node(s) which are using or used by the node where the parameter is enabled. Connections which go via nodes that are aware of TCP connection states, such as firewall, NAT or load balancer may start dropping frames because of the setting. The problem will become visible when there is large enough number of connections. –  Feb 22 '10 at 18:54
  • Setting it to `1` works for future connections, but what about those current ones that are already opened? – Pacerier Jan 23 '16 at 04:16
  • anybody knows what was beyond the (now broken) http://bram.creative4vision.nl/debian/timewait.html link ? – sylvainulg Aug 23 '21 at 10:01
21

Actually there is a way to kill a connection - killcx. They claim it works in any state of the connection (which I have not verified). You need to know the interface where communication happens though, it seems to assume eth0 by default.

UPDATE: another solution is cutter which comes in some linux distros' repositories.

akostadinov
  • 1,178
  • 1
  • 9
  • 18
3

Yes, there is. You can use ss to immediately destroy all sockets that are in TIME-WAIT state like this:

ss state time-wait sport = 49200 -K

After that, restarting your listener should succeed instead of its bind call failing due to Address already in use.

NB: This command requires super-user privileges.

Alternatively, if you are able to modify your program you can simply set SO_REUSEADDR before binding the socket such that the bind still succeeds despite having stale connections being in TIME-WAIT state.

maxschlepzig
  • 895
  • 7
  • 16
  • 2
    Indeed this looks to me to be the only correct answer to OP's question: can be done after the fact rather than changing beforehand various settings (of course SO_REUSEADDR should always be used for a server, but if it wasn't done when initially run...), and really removes the kernel's known entry. Any answer telling packets must be sent don't change anything on the local network's stack state when it's in TIME_WAIT state. Of course the ability to remove a socket entry with `ss` was probably not available when the question was asked. – A.B Feb 19 '23 at 15:51
3

Another option is to use the SO_LINGER option with a timeout of 0. This way, when you close the socket is forcibly closed, sending a RST instead of going into the FIN/ACK closing behavior. This will avoid the TIME_WAIT state, and may be more appropriate for some uses.

  • 3
    It also loses any outbound data that is still in transit, and may cause an error at the other end. Not recommended. – user207421 Mar 31 '11 at 09:46
  • @EJP Failing early is almost always the right call. Networking isn't reliable, and fighting that will slow things down. A crashed app can't assume that any data made it out safely. – Tobu Dec 09 '14 at 02:06
  • 2
    Actually, I'd recommend this any day when the other endpoint is a buggy, embedded industrial bus gateway that implements its own application-layer reliable transport over TCP, where said transport prevents the connection from ever closing unless it receives RST and thus fills up the connection limit on that gateway. There. I gave you a very specific and very real example that, sadly, requires resorting to hacks like this. – andyn May 28 '15 at 18:33
  • @Tobu Networking isn't reliable, but TCP tries to be, and making that worse doesn't constitute making anything better, and letting TCP do its job doesn't constitute 'fighting' anything. – user207421 Mar 22 '18 at 02:45
3

An alternative solution would be to have some reliable proxy or port forwarding software that listens on port 49200, then forward the connection on to one of several instances of your less reliable program using different ports... HAPROXY springs to mind.

Incidentally the port your connecting on is quite high. You could try using an unused one just above the 0-1024 range. Your system is less likely to use a lower port number as an ephemeral port.

andrew pate
  • 271
  • 1
  • 2
  • 6
0

TIME_WAIT is most common issue in socket programming client server architecture. Wait for few seconds trying periodically is the best solution for it. For real time applications they need server must get up immediately There is SO_REUSEADDR option for them.