Off The Grid
   Simple listserv
   xml tools
Karel as an adult

Summary for the Fast and the Furious

What is tcp-fwd?

Whilst satisfying a requirement for TCP forwarding and dispatching, and whilst playing around with Python, I actually wrote a halfway decent and usable piece of code that I'd like to share. Here's tcp-fwd, a Python script that: So, tcp-fwd is basically a TCP forwarder and a balancer; but it's somewhat smart in the following aspects: you can state a number of back end servers, and tcp-fwd will happily distribute incoming traffic in a least-used manner (the back end that's been used the least, gets the new connection).

But wait, there's more! When a connection to a back end server fails, then tcp-fwd will choose a different back end for this connection, so that the client doesn't even notice that one of the back ends is down. To be even smarter, tcp-fwdtake that server 'out of rotation' for a number of seconds so that the process can recover. Only after that the server is taken back into rotation.

And there's even more than that. tcp-fwd can be instructed to cut off long-running connections where no activity is seen, so that it implements an 'idle timer'. And there are more goodies; different log levels, back end state reporting, binding to a specific IP address.

I wrote tcp-fwd to scratch my own itch. It's pretty much related to Crossroads, my pet balancer. But as knowledge and technology move on, I think that a Python tool is easier to write and to maintain than a custom C++ program, and its speed is just as good. So it's a step forward...

Usage Information

usage: tcp-fwd [-h] [--port PORT] [--logfile LOGFILE]
               [--loglevel {debug,info,error}]
               [--logserverinterval LOGSERVERINTERVAL] [--logpayload]
               [--daemonize] [--bindaddress BINDADDRESS] [--bufsize BUFSIZE]
               [--conntimeout CONNTIMEOUT] [--transmittimeout TRANSMITTIMEOUT]
               [--leetime LEETIME]
               server [server ...]

Welcome to tcp-fwd V1.00. Copyright (c) Karel Kubat <> 2015ff.
All rights reserved.

positional arguments:
  server                server(s) to forward to (at least 1 required),
                        specified as HOST:PORT

optional arguments:
  -h, --help            show this help message and exit
  --port PORT, -p PORT  tcp port to listen to, default 8080
  --logfile LOGFILE, -l LOGFILE
                        log file, default /tmp/tcp-fwd.log
  --loglevel {debug,info,error}, -L {debug,info,error}
                        logging level, default is info
                        server health logging interval, default 0 (off)
  --logpayload, -P      log network payload in loglevel debug
  --daemonize, -D       daemonize server, default is stay in foreground
                        bind address to listen to, default (all
                        available NIC's)
  --bufsize BUFSIZE, -B BUFSIZE
                        network buffer size, default 4096
                        connection timeout, default 5 secs
                        network transmit timeout, default 0 secs (off)
  --leetime LEETIME     lee time: nr of secs before retrying a dead server,
                        default 5 secs

If you specify more than one forwarding servers, then next ones are selected
using a least-connections algorithm. Not reachable servers are

Command-line Flags and Arguments

-h or --help
Usage information.
-p PORT or --port PORT
TCP port that tcp-fwd will listen to.
-l FILE or --logfile FILE
Log file where debugging, informational or error messages are sent to. The logfile is appended and may be 'rotated away' while tcp-fwd is running. I.e., your 'logrotate' scripts may move away and compress existing logs, without having to restart the running tcp-fwd process.
--loglevel LEVEL or -L LEVEL
Verbosity level. The level may be one of debug, info or error. The default is info.
--logserverinterval SEC or -s SEC
When the interval is non-zero, then each 'sec' seconds the state (health) of all servers is logged using level info (therefore, this information will be only visible with log level info or debug. This option is present so that a logfile surfer can pick up these messages to alert sysops of back end unavailability. This information appears in the following format:
2015-02-23 17:09:57,721 INFO Server health overview:
2015-02-23 17:09:57,721 INFO Server hostname1:port1: 22 times tried, DEAD
2015-02-23 17:09:57,721 INFO Server hostname2:port2: 23 times tried, alive
The server marked as DEAD is -of course- unavailable. It's been tried 22 times, which is the mininum of the available servers in this example. Therefore it will be retried again when a next client connects (unless the leetime is still active, see below).
--logpayload or -P
In log level debug the bytes that are shuttled to and fro are also logged in the logfile. Note that this heavily slows down the dispatcher and should only be used when really necessary.
--daemonize or -D
When present, 'deamonizes' the tcp-fwd process.
--bindaddress ADR or -b ADR
Binds the listener to the given address, which should be in dotted decimal form (e.g., The default is which binds tcp-fwd to all available network addresses of the system.
--bufsize SZ or -b SZ
Specifies the size of network buffers, the default is 4096. Smaller buffers lead to a smaller memory footprint of tcp-fwd, but require more reads and writes which potentically slows down the process. Also, ridiculously large network buffers are overkill; network transmits of large data arrive by definition in separate chunks and can't fill a huge buffer in one go. If you want to determine whether it's worth while increasing the buffer size, then enable --loglevel debug together with --logpayload and check the log file. When the logging indicates that the full number of characters is obtained during each read, then increasing the buffers will result in less reads (but more used memory). For example, the following abbreviated log shows an SSH session. The number of transmitted characters doesn't exceed 644; hence, a buffer size of say 1024 would be enough.
2015-02-23 17:31:19,472 DEBUG Received from socket 5: [16 bytes]: \000\000\000\f\n...
2015-02-23 17:31:19,472 DEBUG Received from socket 5: [52 bytes]: \000\000\000 \335\316\024.\272...
2015-02-23 17:31:19,473 DEBUG Received from socket 8: [52 bytes]: \000\000\000 2\005\351\032it...
2015-02-23 17:31:19,473 DEBUG Received from socket 5: [68 bytes]: \000\000\0000R\313...
2015-02-23 17:31:19,474 DEBUG Received from socket 8: [68 bytes]: \000\000\0000\330...
2015-02-23 17:31:19,475 DEBUG Received from socket 5: [372 bytes]: \000\000\001`n...
2015-02-23 17:31:19,478 DEBUG Received from socket 8: [324 bytes]: \000\000\0010\250...
2015-02-23 17:31:19,483 DEBUG Received from socket 5: [644 bytes]: \000\000\002p\253...
2015-02-23 17:31:19,494 DEBUG Received from socket 8: [36 bytes]: \000\000\000\020?...
--conntimeout SEC or -c SEC
When tcp-fwd attempts to connect to a back end server, then this flag limits the allowed time. After the indicated number of seconds, tcp-fwd decides that the server is dead.
--transmittimeout SEC or -t SEC
When a running TCP connection is idle for more than the indicated number of seconds, then tcp-fwd interrupts it. This implements an idle-time like feature. This is typically used for online connections like SSH, telnet.
--leetime SEC
When a back end server is marked dead (due to a previously failing attempt to connect), then tcp-fwd doesn't try to use the server for the indicted number of seconds. This allows the server to come to its senses and prevents too frequent connection attempts. The default is 5 seconds.

Use Cases

Server Farm

Tcp-fwd can be used in all cases where you have a server farm that provides similar services and want to (a) offer one point of contact to clients, (b) fail-over, incase one of the back ends goes down. This could be e.g., a remote desktop farm. Example: tcp-fwd would be configured to listen to the standard RDP port 3389 on a system called 'rdpgate'. This system would forward all connections to one of a set of true RDP back ends, called 'rdp1' to 'rdp5'. The invocation:
  tcp-fwd -p3389 rdp1:3389 rdp2:3389 rdp3:3389 rdp4:3389 rdp5:3389 
Tcp-fwd is network payload agnostic, which means that it doesn't know and doesn't care what it ships. 'Can it be used for HTTPS?' Yes, it can, as long as all back ends have the right (identical) certificate installed. Clients will neither know nor care by which back end they are serviced.

Port Forwarding

Yes, tcp-fwd can also function as a port forwarder. E.g., to make your webserver listen not only to port 80, but also to port 81, you could start tcp-fwd as:
tcp-fwd -p81 localhost:80  
It's a bit of a stretch as to why you'd want it, but hey, it's possible.

What it won't do

Tcp-fwd doesn't provide any means of letting clients reconnect to the same back end server as before. There are generally two approaches to this: by remembering the client's IP address, or by traffic shaping (inserting a stickiness-cookie in HTTP traffic). Tcp-fwd does neither; it's just a plain vanilla balancer/forwarder.

Tcp-fwd only works with TCP. UDP isn't supported. If you are interested in balancing and forwarding UDP, take a look at dnspb, my DNS proxy/balancer which also handles UDP.


How it works internally

If you must know: it's a simple threaded daemon. The main thread accepts connections and hands theme off to worker threads that service one TCP link. These workers shuttle network bytes to and fro using select().