Socket Programming: Clients, Servers and More

A Pipe with Two Ends

In the last chapter, we saw everything the operating system has to do to deliver a bunch of bytes from one process to another, anywhere in the world. For us, though — for programmers — all of it happens behind a single interface: the socket. This paradigm was introduced in 4.2BSD, the 1983 version of Berkeley's flavor of Unix — and pretty much every operating system and programming language since then has adopted this interface, more or less. In Python, it's wrapped in the socket module; and it's a pretty thin layer, which makes it a great way to understand the details of networking in a hands-on, practical manner.

The way it works is actually pretty simple, at least in principle: to the operating system, a socket is a file descriptor, like an open file. We write some bytes into it, and those bytes are sent out into the world; and we read some bytes from it, and those bytes are received from somewhere out there. The only difference is that with a file, we write bytes into it and read them back later, so in effect, we communicate with ourselves, whereas with a socket we are communicating with another process — which might even be on another continent.

To communicate over a socket, however, one process has to know where to find the other: for this reason, there are actually two roles to a socket: a server that waits on a particular port, and a client that connects to a particular server on a particular port. This asymmetry only exists when the connection is being established, though; once a connection is established, both sides are equal, and either side can send or receive.

Here's the plan: we'll start with the raw socket API, and see how it works for both the server and the client. Then we'll cover some uncomfortable truths about blocking and partial reads, and develop a small binary protocol, as well as the classes and layers we wrap it in — so we can rewrite our tiny little server again and again. When I teach this class in person, I demonstrate the first part with two terminals, one for each side — here, though, we'll do both in a single REPL, which works thanks to a small detail I'll explain in a moment:

>>> import socket
>>> server = socket.socket()
>>> server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>>> server.bind(('127.0.0.1', 1234))
>>> server.listen(5)
>>> client = socket.socket()
>>> client.connect(('127.0.0.1', 1234))
>>> connection, address = server.accept()
>>> address
('127.0.0.1', 56504)
>>> client.getsockname(), client.getpeername()
(('127.0.0.1', 56504), ('127.0.0.1', 1234))
>>> connection.getsockname(), connection.getpeername()
(('127.0.0.1', 1234), ('127.0.0.1', 56504))

Let's take a moment to understand what exactly is happening. In the server, we call socket.socket() — and although it's not immediately apparent, we actually get back a TCP socket over IPv4 by default: the AF_INET family and SOCK_STREAM type. That setsockopt thing is some voodoo to be explained in the next section; for now, we ignore it and move to bind. This command is actually quite crucial: it attaches our socket to a particular address and port, so that it can receive information sent to this here "phone number". We use 127.0.0.1 for the address, which means "this computer", and 1234 for the port. Then, listen(5) puts our socket in a listening mode, indicating we'd like to receive connections.

In the client, we call socket.socket() again, and then connect to the server's address. Notice that even though the server hasn't called accept yet, the client's call already returned: the operating system actually completed the TCP handshake for us, and simply queued the connection on the server side, so accept merely pops the next connection out of this queue. That's why we can do everything in a single REPL: with two terminals, the accept call would just block until a connection was made. In any case, accept returns two things: a brand new socket, dedicated to this particular connection, as well as the address of the client who made it. Notice that the listening socket itself is never used for communication: it's just a factory of connections.

And see how the addresses make sense: the client is on 127.0.0.1 with some high port, which it never asked for — an ephemeral port, allocated by the operating system when the connection was made, and its peer is the server, which listens on port 1234. On the server side, the connection's address is the mirror image. Think of it as a pipe with two ends: one in each process, which can be on two sides of the world.

>>> connection.settimeout(1)
>>> connection.recv(1024)
TimeoutError: timed out
>>> client.sendall(b'hello')
>>> connection.recv(1024)
b'hello'
>>> connection.sendall(b'hi there')
>>> client.recv(1024)
b'hi there'
>>> client.close()
>>> connection.recv(1024)
b''
>>> connection.close()
>>> server.close()
>>> server
<socket.socket [closed] fd=-1, family=2, type=1, proto=0>

Let's break it down. The recv(1024) call is a blocking call, which means that it stops the flow of our program until some data arrives. To demonstrate this, I've actually set a timeout of one second first, so the call raises a TimeoutError if nothing arrives by then, and we don't freeze the REPL forever. Once the client sends hello, however, the server's recv call returns it immediately — and the server sends a response, which the client receives. The 1024 is the maximum number of bytes we want to receive; but since recv is perfectly content with getting anything at all, it returns as soon as something arrives.

To hang up the phone, the client calls close() on its end. The next time the server tries to receive a message, it gets an empty bytes object, the same sentinel as we had in the loops chapter — indicating that the party is over. Each side closes its own end of the "pipe"; and finally, we close the listening socket, too, whose representation now indicates it's [closed]. Just note that everything we sent and received was in bytes — there's no such thing as sending a string over a socket, so we'd have to encode it into some sequence of bytes first, using the various encodings we discussed in the chapter about native types.

The Ghost of Your Previous Server

As we've seen, the server uses bind to associate itself with an address, and that address is a tuple of (host, port): the host indicates which network interface to accept connections on, and the port is the number used to communicate.

If we use 127.0.0.1 as the host, it means we only accept connections from this machine — so no one outside of it can actually reach this port. If we want to accept connections from anyone, we need to use 0.0.0.0, which basically means "all of the above" — whoever connects, come on in. If the machine has several network cards, we can choose a specific one to listen on by specifying its IP address.

Similarly, if we use 0 for the port, we're asking the operating system to assign us any free port from the available pool. By the way, on Linux, binding to a port below 1024 traditionally requires root privileges, which is one reason development servers usually run on ports like 8000 or 5000.

Now, the voodoo incantation. Whenever I show this to students, I tell them, "just add this line, it'll save you a lot of suffering". Here's what happens if you don't:

>>> server = socket.socket()
>>> server.bind(('127.0.0.1', 2345))
>>> server.listen()
>>> client = socket.create_connection(('127.0.0.1', 2345))
>>> connection, address = server.accept()
>>> connection.close()
>>> server.close()
>>> server = socket.socket()
>>> server.bind(('127.0.0.1', 2345))
OSError: [Errno 48] Address already in use
>>> server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>>> server.bind(('127.0.0.1', 2345))

What happens is, we start a server, accept a connection, close it on the server side, and close the server. Then we try to start another server on the same port, and get OSError: [Errno 48] Address already in use (on Linux, it's errno 98). But if you think about it, the port is no longer in use! We don't have a server listening on it, so what's going on?

The truth is, it's the ghost of your previous server. Whichever side of a TCP connection closes it first keeps it in a state called TIME_WAIT for a while (from half a minute to a couple of minutes, depending on the operating system), in case any stray packets from the old connection are still on their way. Setting the SO_REUSEADDR flag tells the operating system, "if this happens, it's cool; just let us bind anyway", which is exactly what we want when we're trying to restart a crashed server during development.

This is why the first exercise of my course included this hint: use setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) to avoid annoying errors about the port already being in use, otherwise you'd have to wait a whole minute for it to be freed up after every restart.

The other mysterious number is the argument to listen, the backlog. People tend to put 5 here, or 5,000, or 1, without really knowing what it means. It's the number of pending connections the operating system is willing to keep in its pocket — connections that have completed, or are in the process of completing, the handshake, but that we haven't really accepted yet.

>>> server = socket.socket()
>>> server.bind(('127.0.0.1', 3456))
>>> server.listen(1)
>>> clients = []
>>> for i in range(3):
...     client = socket.socket()
...     client.settimeout(1)
...     try:
...         client.connect(('127.0.0.1', 3456))
...         print(i, 'connected')
...     except TimeoutError:
...         print(i, 'timed out')
...     clients.append(client)
0 connected
1 timed out
2 timed out

When the first client connects, everything is fine — it waits in the queue; but when the second and third clients try to connect, they time out, because the queue is full, and the operating system is simply ignoring them. That's on my Mac; the exact limit for a given backlog is OS-specific, but that's what happens in general. This queue is also the resource that a SYN flood attack, which we saw in the previous chapter, tries to deplete.

All this boilerplate is something we'll repeat a lot, so it's nice to organize it in a function; and it's a nice opportunity to demonstrate the keyword-only parameters from our first functions chapter. Imagine how confusing it would be to read create_server(1234, '0.0.0.0', 5, True) — what's the 5 for? What's the True?

>>> def create_server(port, *, host='127.0.0.1', backlog=5, reuseaddr=True):
...     server = socket.socket()
...     if reuseaddr:
...         server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
...     server.bind((host, port))
...     server.listen(backlog)
...     return server
>>> create_server(1234, '0.0.0.0', 5, True)
TypeError: create_server() takes 1 positional argument but 4 were given
>>> listener = create_server(1234, backlog=100)
>>> listener.getsockname()
('127.0.0.1', 1234)
>>> create_server(0).getsockname()
('127.0.0.1', 56531)
>>> import inspect
>>> inspect.signature(socket.create_server)
<Signature (address, *, family=<AddressFamily.AF_INET: 2>, backlog=None, reuse_port=False, dualstack_ipv6=False)>

Here, everything after the star has to be passed by keyword, so the invocations are self-documenting; and passing in 0 as the port will actually get one allocated by the operating system, which we can get by calling getsockname. Incidentally, since Python 3.8, the standard library has its own socket.create_server, whose options are also keyword-only; and it even sets SO_REUSEADDR for you on POSIX systems.

An Echo Server

With all this setup out of the way, a server is just a loop: we call accept, which blocks until a connection is made — and when one is, we handle it, and go back to accepting the next one. The simplest and most useful handler I can think of at this point is an echo, which takes whatever you give it and sends it right back, letting you verify that a connection works as intended. Here's how it might look, running in a daemon thread, together with a client to play with:

>>> import threading
>>> def handle(connection):
...     with connection:
...         while True:
...             data = connection.recv(4096)
...             if not data:
...                 break
...             connection.sendall(data)
>>> def serve(listener):
...     with listener:
...         while True:
...             connection, address = listener.accept()
...             print('server: connection from', address)
...             handle(connection)
>>> threading.Thread(target=serve, args=(listener,), daemon=True).start()
>>> with socket.create_connection(('127.0.0.1', 1234)) as client:
...     client.sendall(b'hello')
...     print(client.recv(4096))
...     client.sendall(b'world')
...     print(client.recv(4096))
server: connection from ('127.0.0.1', 56532)
b'hello'
b'world'

Let's go over handle: it receives data up to 4096 bytes. If the result is empty, it means that the client has hung up, so we break; and otherwise we send everything back. It's our old infinite loop with a break in the middle, from our chapter about loops, isn't it? The with connection: part is just a nice way to close the socket, however we exit — it's a context manager, just like files, remember? (See our chapter about their lifecycle and context management.) As for serve, it's an infinite loop over accept. I put it in a background thread just so that our REPL remains free to play the client, but that's not how a server would normally run (and what threads really are is the subject of the next chapter, anyway) — for now, a daemon thread is just one that doesn't keep the program alive when we exit the REPL.

Anyway, the client uses socket.create_connection, which is a handy shortcut to resolve a host and port and connect to them, trying each address returned by getaddrinfo like we've seen in the previous chapter. Both sides are included in our output: the server prints out the client's address and ephemeral port, while the client gets back everything it sent.

Incidentally, we can do the same in the shell with netcat, the Swiss army knife of TCP connections. With nc -l 1234, we can listen on a port, and with nc 127.0.0.1 1234 we can connect to it. Whatever we type on one side shows up on the other — go ahead and compare the amount of code required to achieve this. It's another one of those reasons that I keep encouraging people to get comfortable with Linux:

$ printf 'hello, netcat\n' | nc -w 1 127.0.0.1 1234
hello, netcat

And indeed, the line is sent and received right back; the -w 1 just tells netcat to give up after a second of inactivity, rather than waiting indefinitely. But let's get back to our code, because there's another interesting thing we can do with it. The accept loop is also a natural fit for a generator — it's an infinite stream of connections, much like the infinite generators we've seen in the generators chapter. So we can actually rewrite it like so:

>>> def connections(listener):
...     while True:
...         connection, address = listener.accept()
...         yield connection
>>> def serve(listener):
...     with listener:
...         for connection in connections(listener):
...             handle(connection)

And suddenly, our server code reads almost like what it does: for each connection, handle it — all the accepting stuff is tucked away in a nice function that yields connections forever.

The obvious problem with this server is that while it's busy handling one client, it's not accepting any other clients, which are left waiting in the backlog queue until the current one leaves. We'll address this issue at the end of the chapter, but first…

recv(4) Doesn't Mean Four Bytes

In the last chapter, we saw that TCP is a stream; it doesn't have a notion of messages, just bytes, in order. This is exactly how the socket API is structured, and this is precisely the problem: it's the single most common source of networking bugs I see.

For example:

>>> import struct, time
>>> listener = create_server(5678)
>>> client = socket.create_connection(('127.0.0.1', 5678))
>>> connection, address = listener.accept()
>>> client.sendall(b'hello')
>>> client.sendall(b'world')
>>> time.sleep(0.1)
>>> connection.recv(1024)
b'helloworld'
>>> def trickle(sock, data):
...     for byte in data:
...         sock.sendall(bytes([byte]))
...         time.sleep(0.01)
>>> threading.Thread(target=trickle, args=(client, struct.pack('<I', 1000))).start()
>>> connection.recv(4)
b'\xe8'

That is, the client sends hello and then world, but by the time the server receives them, both are already waiting in the kernel's buffer — so the one recv(1024) actually returns helloworld, and the boundary between the two messages is lost. Similarly, a helper thread might send a 4-byte integer one byte at a time, in which case recv(4) returns just the first byte. In other words, recv(n) means "receive at most n bytes", and it can return as soon as any of them are available, even if it's just one.

This is not hypothetical: the tests of the first exercise I give in my course do precisely that, just to see if students account for this or not; and the exercise itself warns that even something as simple as recv(4) is not guaranteed to return 4 bytes. The reason it's so underappreciated is that it's surprisingly difficult to recreate: when you're playing with a small example on your computer, it usually does work — recv(4) does get 4 bytes, at least on localhost, with tiny messages. And then, when it gets to production, and you start sending over a real network to real clients, it suddenly stops working; but by then, it's too late to reconsider your design.

To give a concrete example, try sending 10 MB, and see for yourself what the receiver actually gets:

>>> sender = threading.Thread(target=client.sendall, args=(b'x' * 10_000_000,))
>>> sender.start()
>>> time.sleep(0.1)
>>> len(connection.recv(10_000_000))
2478172
>>> len(connection.recv(10_000_000))
1685628

That is, it asked for 10 million bytes, but got 2.5 million, and then another 1.7 million: whatever was available in the kernel's buffers at the moment. The exact numbers are hard to predict, and can vary a lot from run to run or machine to machine, but it's usually less than what you asked for.

This is true for the sender, too: send returns how many bytes the kernel actually accepted into its buffers, which may be less than what you gave it, in which case it's up to you to resend the rest. For example, if we were to send 10 MB over a socket with a timeout:

>>> other = socket.create_connection(('127.0.0.1', 5678))
>>> other.settimeout(1)
>>> other.send(b'y' * 10_000_000)
327212

We'd get 327,212, because that's how many bytes the sender managed to push into the socket before the receiver — or rather, the lack of one; nobody's reading on the other side — caused its buffers to fill up; and with a timeout set, the socket won't block indefinitely, so send just returns what it managed. This is why sendall is a loop around send, which keeps trying until everything is sent (or raises an error), so in practice, you should always use it.

Another subtlety worth mentioning is that when the peer closes the connection, recv is first going to return whatever data is still there, and only then an empty bytes object; so it's important to know that an empty result always means the end of the stream, and not "nothing so far" — if there's nothing so far, a blocking socket will just wait, while a non-blocking one will raise an exception.

Unfortunately, there's no recvall in the standard library: the socket can't know how much data you expect to receive, so this has to be part of the protocol. But we can easily write it ourselves, and then use it on our trickled header:

>>> def recv_all(connection, size):
...     chunks = []
...     while size > 0:
...         chunk = connection.recv(size)
...         if not chunk:
...             raise ConnectionError('incomplete data')
...         chunks.append(chunk)
...         size -= len(chunk)
...     return b''.join(chunks)
>>> threading.Thread(target=trickle, args=(client, struct.pack('<I', 1000))).start()
>>> struct.unpack('<I', recv_all(connection, 4))
(1000,)
>>> client.sendall(b'\x01')
>>> client.close()
>>> recv_all(connection, 4)
ConnectionError: incomplete data

That is, ask for however many bytes we still need; and if at any point we get an empty chunk, it means the peer has closed the connection before sending us everything — which is an error, so we raise a ConnectionError. Otherwise, we collect the chunk and subtract its length — eventually arriving at (1000,) when we reassemble the header. And if the client only sends a single byte and closes, we get an error, rather than garbage.

This might feel like an insignificant detail, but I think it's rather important. You might have already asked yourself, when looking at the code above — why the list of chunks and the single join at the end, instead of a simple data += chunk? The reason is that lesson from the performance chapter, about the allocations. Since bytes are immutable, each += would allocate a new, slightly bigger object, and copy everything into it, which adds up to quadratic behavior in the worst case scenario; whereas collecting all the chunks and concatenating them at once allocates the result exactly once. Judging by the homework assignments I get, a lot of people are not aware of this.

A Binary Protocol

Let's start with the course project. In my advanced system design course, students build a system for a brain-computer interface: some imaginary hardware (at least, for now) capable of reading people's minds and uploading snapshots of their cognitions. In the first exercise, they were tasked with uploading a "thought": some user ID, a timestamp and a string. The protocol I had in mind was pretty simple:

All in little endian, we have 8 bytes of user ID (a uint64), 8 bytes of timestamp (also a uint64, in seconds since the epoch), 4 bytes of thought size (a uint32), and then as many bytes of the thought itself, in UTF-8. To do this with struct, we'll need the struct format <QQI, but before that, a word of warning: if you've never worked with formats that don't start with a byte-order prefix before, you might want to know that they assume your platform's native sizes and alignment. For example:

>>> struct.calcsize('LLI')
20
>>> struct.calcsize('<LLI')
12
>>> struct.calcsize('<QQI')
20
>>> struct.pack('<QQI', 1, 1571915525, 10) + "I'm hungry".encode()
b"\x01\x00\x00\x00\x00\x00\x00\x00\x05\x87\xb1]\x00\x00\x00\x00\n\x00\x00\x00I'm hungry"

So LLI happens to be 20 bytes on a 64-bit Mac or Linux, where a native long is 8 bytes; but if we add an explicit endianess, then standard sizes apply, and L is 4 bytes, so <LLI is 12; and on Windows, a long is 4 bytes even in native mode. If we want to adhere to the protocol and be honest, we really meant <QQI: explicit endianess, explicit sizes (fun fact: the reference server in my own exercise's tests used LLI, and worked by pure luck). Anyway, now we're ready.

The client simply packs the header, connects, sends the header and the data with sendall, and closes the connection. It also has a small command-line interface to parse the address, the user ID and the thought, and print done or an error message:

# client.py
import socket
import struct
import sys
import time


def upload_thought(address, user_id, thought):
    data = thought.encode()
    header = struct.pack('<QQI', user_id, int(time.time()), len(data))
    with socket.create_connection(address) as connection:
        connection.sendall(header + data)


def main(argv):
    if len(argv) != 4:
        print(f'USAGE: {argv[0]} <address> <user_id> <thought>')
        return 1
    try:
        host, port = argv[1].split(':')
        upload_thought((host, int(port)), int(argv[2]), argv[3])
        print('done')
    except Exception as error:
        print(f'ERROR: {error}')
        return 1


if __name__ == '__main__':
    sys.exit(main(sys.argv))

The server simply listens, and for every connection, receives exactly the header's size, unpacks it, receives exactly size more bytes, decodes them and prints the thought. As an aside, the exercise also required it to wait a second before printing, and this will become relevant at the end of the chapter:

# server.py
import datetime as dt
import socket
import struct
import sys
import time

HEADER = struct.Struct('<QQI')


def run_server(address):
    server = socket.socket()
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind(address)
    server.listen(1000)
    with server:
        while True:
            connection, _ = server.accept()
            with connection:
                user_id, timestamp, size = HEADER.unpack(receive_all(connection, HEADER.size))
                thought = receive_all(connection, size).decode()
            time.sleep(1)
            print(f'[{dt.datetime.fromtimestamp(timestamp)}] user {user_id}: {thought}', flush=True)


def receive_all(connection, size):
    chunks = []
    while size > 0:
        chunk = connection.recv(size)
        if not chunk:
            raise RuntimeError('incomplete data')
        chunks.append(chunk)
        size -= len(chunk)
    return b''.join(chunks)


def main(argv):
    host, port = argv[1].split(':')
    try:
        run_server((host, int(port)))
    except KeyboardInterrupt:
        pass


if __name__ == '__main__':
    sys.exit(main(sys.argv))

Let's see it in action:

$ python server.py 127.0.0.1:4567 &
$ python client.py 127.0.0.1:4567 1 "I'm hungry"
done
[2026-10-14 19:39:27] user 1: I'm hungry
$ python client.py 127.0.0.1:4567
USAGE: client.py <address> <user_id> <thought>
$ kill %1
$ python client.py 127.0.0.1:4567 1 "I'm hungry"
ERROR: [Errno 61] Connection refused

The client prints done, and the server prints the timestamp, the user and the thought. If we forget to pass some arguments, the client prints its usage; and if we kill the server, it tells us Connection refused, because there's no one listening on that port.

As an aside, both scripts have the same shape: a function that does the actual work, and a thin main(argv) that parses the arguments and returns an exit code for sys.exit. This way, the function can be imported and used independently of the command-line interface, and this latter use case is just one more way to call it — we'll discuss it more in the chapter about CLIs.

What we've done, effectively, is implement our first application protocol on top of TCP. The fundamental problem of any protocol over a stream is figuring out where the message ends; and the three classic solutions to this are: using a delimiter, like a newline (as many text-based protocols do), or the empty line that ends HTTP's headers; using a length prefix, like the size field in our header; or closing the connection (like HTTP/1.0 did). In this case, we actually used a fixed-size header, which contains the length of the variable-size part.

I actually asked the students to implement this first version entirely by hand, and I told them in advance that it should feel disgusting, because the point was to feel the pain before we abstract it away.

Connections and Listeners

One nuisance that has been haunting us from the very beginning of this project is our networking code. It's a bit of a hassle to set up a socket that listens to some port, and even more of a hassle to receive exactly n bytes — and all of this is mixed in with our business logic of uploading thoughts. I'd like to remedy this by wrapping it up in a couple classes.

The first one, Connection, is pretty simple:

# connection.py
import socket


class Connection:

    def __init__(self, sock):
        self.socket = sock

    def __repr__(self):
        host, port = self.socket.getsockname()[:2]
        peer_host, peer_port = self.socket.getpeername()[:2]
        return f'<Connection from {host}:{port} to {peer_host}:{peer_port}>'

    def __enter__(self):
        return self

    def __exit__(self, exception, error, traceback):
        self.close()

    @classmethod
    def connect(cls, host, port):
        return cls(socket.create_connection((host, port)))

    def send(self, data):
        self.socket.sendall(data)

    def receive(self, size):
        chunks = []
        while size > 0:
            chunk = self.socket.recv(size)
            if not chunk:
                raise ConnectionError('incomplete data')
            chunks.append(chunk)
            size -= len(chunk)
        return b''.join(chunks)

    def close(self):
        self.socket.close()

It's a wrapper around a connected socket; send(data) sends all of it, receive(size) receives exactly size bytes or raises an exception if the connection is closed first (finally, our recv_all in its rightful place), and close() closes it, obviously. The representation is rather nice, too; it follows the same convention we established for our magic methods: since a connection can't be recreated from code, it's enclosed in angle brackets; and if you remember the nuance from our lifecycle chapter, we can see it here, too: __enter__ returns the connection itself, while __exit__ closes it. As for connect, it's a class method (the so-called factory method we saw in our attributes and methods chapter), which we can use as an alternative constructor: given a host and a port, it creates a socket, connects it and passes it in for the initialization.

Now, let's consider Listener:

# listener.py
import socket

from connection import Connection


class Listener:

    def __init__(self, port, *, host='0.0.0.0', backlog=1000, reuseaddr=True):
        self.port = port
        self.host = host
        self.backlog = backlog
        self.reuseaddr = reuseaddr
        self.socket = None

    def __repr__(self):
        return (f'{self.__class__.__name__}(port={self.port!r}, host={self.host!r}, '
                f'backlog={self.backlog!r}, reuseaddr={self.reuseaddr!r})')

    def __enter__(self):
        self.start()
        return self

    def __exit__(self, exception, error, traceback):
        self.stop()

    def start(self):
        self.socket = socket.socket()
        if self.reuseaddr:
            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.socket.bind((self.host, self.port))
        self.socket.listen(self.backlog)

    def stop(self):
        self.socket.close()
        self.socket = None

    def accept(self):
        sock, address = self.socket.accept()
        return Connection(sock)

We can pass in a port (and optionally, a host, a backlog and whether to reuse the address, with sensible defaults: all interfaces, 1000, and yes), and then use start and stop to begin and end listening, and accept to receive a connection — except, it's not a raw socket and an address, but a Connection object.

A few things to notice: the options are keyword-only, in accordance with our lesson from create_server, and the representation is rather similar to how we'd create such an object (so it follows the other convention); but more importantly, this class can be used as a context manager, where entering the context starts listening and exiting it stops, so we can do with Listener(8000) as listener: time.sleep(2) to listen for exactly two seconds. In fact, we can do this:

>>> from connection import Connection
>>> from listener import Listener
>>> listener = Listener(6789, host='127.0.0.1')
>>> listener
Listener(port=6789, host='127.0.0.1', backlog=1000, reuseaddr=True)
>>> with listener:
...     with Connection.connect('127.0.0.1', 6789) as client:
...         client.send(b'hello')
...         with listener.accept() as connection:
...             print(connection)
...             print(connection.receive(5))
...         print(client)
<Connection from 127.0.0.1:6789 to 127.0.0.1:56543>
b'hello'
<Connection from 127.0.0.1:56543 to 127.0.0.1:6789>
>>> print(listener.socket)
None
>>> Listener(6789, '127.0.0.1')
TypeError: Listener.__init__() takes 2 positional arguments but 3 were given

Two smaller design decisions: the accept method returns a single Connection object, not a pair of socket and address like before, since the address can be obtained from the connection object anyway (even its representation already shows it, so returning a pair would be redundant); and the default host is 0.0.0.0 (which is accessible from the network, so it's a good default for a server that is meant to serve others), but in these examples I pass 127.0.0.1 explicitly.

You can see the two representations mirroring each other, from the server's port to the client's and back; and after the with block ends, you'll see that the listener's socket is gone. If we pass the host by position, it will refuse it, too.

Now, let's rewrite our thought server on top of these classes — and notice how the core loop becomes exceedingly simple: accept a connection, receive the header, receive the thought, print it. No sockets, no setsockopt, no recv loops; only the words of our domain, with all the noise abstracted away. What's more, notice how we didn't actually get rid of any code — we just moved it around, so that our business logic is simple and clean, and if I ever need to deal with the networking part, I can just go to connection.py and handle it there. This is basic modular programming: it's reusable, so we can use it in other projects, and if we decide to improve our server's logic, we can do so without having to touch the networking part, and vice versa.

This brings to mind an old story: the Ship of Theseus. If you keep replacing the planks of your ship, one by one, at what point does it stop being the same ship? A codebase is kind of like that, too: we keep throwing things away and reorganizing everything, all the time. In my course, the first version was written by hand; then, we moved it into our own utilities, and later we threw those utilities away in favor of real libraries (click and flask instead of our own CLI and web utilities, a database instead of files on disk). Often times, the code you write today goes into the trash two days later; what remains is the insight into what it's supposed to do. Don't feel bad about it, either; that's how it's supposed to be.

A Layer of Our Own

The idea of passing thoughts around is nice, but it gets a bit tedious to work with triplets of user IDs, timestamps and strings all the time, so let's encapsulate this in a class: a representation, a nice string, serialization to get the bytes that are ready to send over the wire, and a class method to deserialize them back into an object.

And there's a problem, because serializing a thought is easy, but deserializing it is not: to do the latter, we need all the data, which we don't have, and to get it, we need to deserialize the header first, so we know how much of it is left.

When I taught this, some students suggested that we might pass the connection into the Thought class, and let it call the receive method to read the header and the rest. But that would couple the Connection class to the Thought one, even though they have nothing to do with each other.

In fact, the problem is in our protocol: even at this level, we've already mixed networking metadata (that is, how long is our message) with the actual data (the user, the timestamp, and the thought). To fix it, we can use the same solution that the internet uses: different layers. On the message layer, we simply define that every message has a uint32 size, and then that many bytes. On the thought layer, we define what's in a message — and since the message already knows its own length, we can omit the size. That's thought.py:

# thought.py
import datetime as dt
import struct


class Thought:

    header = struct.Struct('<QQ')

    def __init__(self, user_id, timestamp, thought):
        self.user_id = user_id
        self.timestamp = timestamp
        self.thought = thought

    def __repr__(self):
        return (f'{self.__class__.__name__}(user_id={self.user_id!r}, '
                f'timestamp={self.timestamp!r}, thought={self.thought!r})')

    def __str__(self):
        return f'[{self.timestamp:%Y-%m-%d %H:%M:%S}] user {self.user_id}: {self.thought}'

    def serialize(self):
        header = self.header.pack(self.user_id, int(self.timestamp.timestamp()))
        return header + self.thought.encode()

    @classmethod
    def deserialize(cls, data):
        user_id, timestamp = cls.header.unpack(data[:cls.header.size])
        thought = data[cls.header.size:].decode()
        return cls(user_id, dt.datetime.fromtimestamp(timestamp), thought)

Notice the header struct is now <QQ — that is, just the user ID and the timestamp. The thought itself is whatever bytes follow, up to the end of the message. deserialize now works on the entire message, and doesn't touch a socket at all; being a class method, it constructs cls, so a subclass would deserialize into itself, and so on.

To complete the picture, we need to add send_message and receive_message to Connection:

# connection.py (added to the Connection class)
    def send_message(self, message):
        self.send(struct.pack('<I', len(message)) + message)

    def receive_message(self):
        size, = struct.unpack('<I', self.receive(4))
        return self.receive(size)

Notice that sending a message involves prepending its length, and receiving a message means reading exactly 4 bytes, unpacking them into a size, and then reading exactly that many more. In other words, the message layer doesn't care about thoughts, and thoughts don't care about sockets: each layer simply treats the other one's data as an opaque payload, just like we did with our parcel in the previous chapter.

And it works:

>>> from connection import Connection
>>> from listener import Listener
>>> from thought import Thought
>>> import datetime as dt
>>> thought = Thought(1, dt.datetime(2019, 10, 25, 15, 12, 5), "I'm hungry")
>>> thought
Thought(user_id=1, timestamp=datetime.datetime(2019, 10, 25, 15, 12, 5), thought="I'm hungry")
>>> print(thought)
[2019-10-25 15:12:05] user 1: I'm hungry
>>> thought.serialize()
b"\x01\x00\x00\x00\x00\x00\x00\x00\xa5\xf4\xb2]\x00\x00\x00\x00I'm hungry"
>>> Thought.deserialize(thought.serialize())
Thought(user_id=1, timestamp=datetime.datetime(2019, 10, 25, 15, 12, 5), thought="I'm hungry")

We can even write a server, which receives a message and deserializes it into a thought, and then replies; and a client, which sends something and receives a reply:

>>> def serve(listener):
...     while True:
...         with listener.accept() as connection:
...             thought = Thought.deserialize(connection.receive_message())
...             print('server:', thought)
...             connection.send_message(b'thanks')
>>> def upload(thought):
...     with Connection.connect('127.0.0.1', 7890) as connection:
...         connection.send_message(thought.serialize())
...         return connection.receive_message()
>>> listener = Listener(7890, host='127.0.0.1')
>>> listener.start()
>>> threading.Thread(target=serve, args=(listener,), daemon=True).start()
>>> upload(thought)
server: [2019-10-25 15:12:05] user 1: I'm hungry
b'thanks'

The server simply prints out the thought it received, and replies with a message of its own, thanks, which our upload function returns. Notice that this reply also serves to make the client wait until the server is done — and the entire thing is just a few lines on either side, and pretty much reads like the protocol itself.

As an aside, receive_message trusts the size it receives; if a client announces a 4 GB message, and then doesn't send anything, the server will wait forever. Real protocols always cap their message sizes, and use timeouts to avoid such situations — but this is beside the point.

In the actual project this started from, the entire thing grew to a session: the client would send a hello message with the user's information, the server would answer with a config message listing all the parts of a snapshot that it's able to handle, and the client would then send a snapshot with those parts only. But notice how, with messages as a separate layer, adding new types of messages is rather easy.

The Protocol Is an Implementation Detail

To give you an example from a later exercise of mine: "There are no changes to the protocol, and to be honest, there aren't going to be any, because it's an implementation detail of your system, not part of its requirements."

This means that you can change it whenever you want. Why make so much effort to serialize and deserialize everything yourself, when there are perfectly valid formats already out there?

Let's swap the thought layer to JSON instead:

# thought.py (the JSON version, with import json at the top)
    def serialize(self):
        return json.dumps({
            'user_id': self.user_id,
            'timestamp': int(self.timestamp.timestamp()),
            'thought': self.thought,
        }).encode()

    @classmethod
    def deserialize(cls, data):
        fields = json.loads(data)
        timestamp = dt.datetime.fromtimestamp(fields['timestamp'])
        return cls(fields['user_id'], timestamp, fields['thought'])

This is what a thought in JSON looks like, and what happens if we upload it to our server, still running from before:

>>> from thought import Thought
>>> thought = Thought(2, dt.datetime(2019, 10, 25, 15, 15, 2), "I'm sleepy")
>>> thought.serialize()
b'{"user_id": 2, "timestamp": 1572009302, "thought": "I\'m sleepy"}'
>>> upload(thought)
server: [2019-10-25 15:15:02] user 2: I'm sleepy
b'thanks'

We can see the bytes on the wire are now readable text, and considerably larger: the numbers are written out in decimal notation, and every field carries its name with it. The server was able to decode this new format, and print out the thought just like before. What's more, neither the upload function nor the serve function changed by a single character!

To be fair, it's a bit of a cheat: when either of them run, they need to look up the name Thought, so when I rebound it, it effectively changed both sides at once. In a real deployment, you would have to upgrade clients and servers in concert, or at least support both formats for a while; but the fact that it works at all is pretty cool, no?

Of course, there are always trade-offs: binary formats are more compact, and faster to parse — but they're also more opaque. If you're ever struggling to decipher them, good luck; evolving them is also your problem, because you need to do it in a way that won't break old versions. JSON, on the other hand, is readable, and you can even debug it with netcat; it's also self-describing, if a bit verbose. My rule of thumb is to use YAML when dealing with a human-machine interface (like configuration files), and JSON when dealing with a machine-machine interface (like APIs).

That being said, we can do even better: protobuf provides a compact binary encoding, but with schemas that allow it to keep old and new versions compatible. Or, we can use gRPC to generate the necessary stubs that handle the networking and the serialization altogether. Or, we can forget about this custom protocol entirely, and use something like HTTP — or even REST, for that matter: GET /config, POST /snapshot. We'll get there in the web chapters.

One final note, though: whatever format you choose, never use pickle to deserialize untrusted bytes. Unpickling can execute arbitrary code, so a pickle-based protocol is basically an open door to anyone who can connect to it.

But why would you go through all this trouble, when what we have already works? You don't have to — but whenever you choose not to upgrade your system, you accumulate what's known as technical debt: the price of not doing things right, which will have to be paid down the road. What's more, you can think of it not as rewriting something that works, but as writing something new that works even better.

And it was only possible because of our layering: the message layer didn't care about what was inside the message, and the business layer didn't care how the thought was encoded. Our ship of Theseus lost yet another plank, but it's still the same ship.

UDP: Fire and Forget

So far, everything we did was TCP — or, to use the terminology of the socket API, SOCK_STREAM. For UDP, or SOCK_DGRAM, we have to do things differently: because there's no connection, there's no need to listen, accept, or even connect. All we do is implement a server, in a thread that receives datagrams and answers in uppercase, and send it stuff, much like before:

>>> server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> server.bind(('127.0.0.1', 8765))
>>> def pong(server):
...     while True:
...         data, address = server.recvfrom(1024)
...         print('server:', data, 'from', address)
...         server.sendto(data.upper(), address)
>>> threading.Thread(target=pong, args=(server,), daemon=True).start()
>>> client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> client.sendto(b'ping', ('127.0.0.1', 8765))
4
>>> client.recvfrom(1024)
server: b'ping' from ('127.0.0.1', 54268)
(b'PING', ('127.0.0.1', 8765))

So the server binds, and uses recvfrom to receive a datagram, along with the sender's address. It then answers with sendto, which accepts such an address to direct the datagram to. The client, though, doesn't even bother binding: the OS assigns it an ephemeral port as soon as it attempts to send a datagram out — which happens when it uses sendto, which returns the number of bytes sent; and recvfrom receives the answer, along with the server's address.

Unlike TCP, datagram sockets actually preserve the message boundaries: each call to sendto sends one datagram, and each call to receive gets exactly one datagram — it's never chopped in half, or stuck together with another one:

>>> receiver = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> receiver.bind(('127.0.0.1', 8766))
>>> client.sendto(b'hello', ('127.0.0.1', 8766))
5
>>> client.sendto(b'world', ('127.0.0.1', 8766))
5
>>> receiver.recv(1024), receiver.recv(1024)
(b'hello', b'world')
>>> client.sendto(b'truncated', ('127.0.0.1', 8766))
9
>>> receiver.recv(4)
b'trun'
>>> receiver.setblocking(False)
>>> receiver.recv(1024)
BlockingIOError: [Errno 35] Resource temporarily unavailable
>>> client.sendto(b'anyone?', ('127.0.0.1', 8767))
7

hello and world arrive separately; and if the datagram doesn't fit in the buffer, we get its beginning — b'trun' — and the rest is discarded, so the next receive finds nothing. With blocking turned off, it raises a BlockingIOError (otherwise, it would just hang, waiting for something that will never arrive); and if we try sending something to a port with no one listening, it actually succeeds — seven bytes went out, and nobody will ever tell us that nobody received them.

Of course, on our computer, nothing will get lost or reordered; but on a real network, datagrams can very well be dropped, duplicated or arrive out of order, and it's up to the application to handle this. DNS clients simply ask again; games and calls simply move on.

As an aside, calling connect on a UDP socket doesn't actually send anything — it just records a default destination for it, so we can use send and recv instead of sendto and recvfrom.

Anyway, that's UDP: it's simpler to use, but much harder to trust — if you need all your data, and in the right order, you'd have to reimplement TCP on top of it, which you would probably do poorly. Or maybe very well: QUIC, the protocol that HTTP/3 runs on, is exactly that — a reliable transport built over UDP.

One Client at a Time

But in all our servers, the same problem remained: while we were handling a client, we weren't accepting anyone else. The next connection would linger in the backlog queue, and if the current client was taking its sweet time, the rest of the clients would wait with it.

Let's make this point more vivid: suppose our server takes a second to think of an answer:

# slow_server.py
import socket
import time

listener = socket.create_server(('127.0.0.1', 4568), backlog=100)
with listener:
    while True:
        connection, address = listener.accept()
        with connection:
            data = connection.recv(4096)
            time.sleep(1) # pretend to think
            connection.sendall(data.upper())

And a client sends it a word and prints the answer:

# ask.py
import socket
import sys

with socket.create_connection(('127.0.0.1', 4568)) as connection:
    connection.sendall(sys.argv[1].encode())
    print(connection.recv(4096).decode())

If we time the first one, and then two clients at once:

$ python slow_server.py &
$ time python ask.py hello
HELLO

real	0m1.069s
user	0m0.038s
sys	0m0.016s
$ time (python ask.py hello & python ask.py world; wait)
WORLD
HELLO

real	0m2.061s
user	0m0.063s
sys	0m0.025s

The first takes a second, like we might expect; but the two clients take two seconds, because the server actually handled them one after the other: which one of them went first was a race, and world came out on top this time. If we had ten clients, the last one would take ten seconds to get a reply; that was the second exercise in my course: make the server support multiple clients at once, so that two uploads, or even ten, would take one second, not two.

You could think of it like a bank, with one clerk and a neat, orderly queue of people that nobody cuts — it's fair, but it's also frustrating. And notice the waste: during that sleep (or, in a real situation, while waiting for a disk, a database or another server to do its thing), the machine is just sitting there, when it could be attending to someone else. That's why network servers are, by their very nature, I/O-bound: a web server, for example, spends most of its life in recv and send.

Luckily, we can fix this: we can handle each connection concurrently, dedicating a thread to each client, and immediately going back to the accept loop. This is what we'll do in the next chapter — with all the fun challenges that it presents, like races, locks and the GIL; two chapters later, we'll do something completely different, and serve multiple clients asynchronously from a single thread, using select and coroutines.

But before we do — let's take a moment to appreciate how far we've come. We started from a raw socket, a pipe with two ends, and realized that TCP gives us a stream, not messages, so we had to build recv_all. We then used it to create a framed, binary protocol, and eventually a Connection and Listener, a message layer, and a swappable encoding; every rewrite threw away some code, but kept the insight. The abstraction we've built is still paying off: in the threaded version, we'll only change the accept loop, and the connections, the messages and the thoughts we've developed won't even notice — the Ship of Theseus, indeed. We'll have to replace its planks, but it'll remain the same ship.