Aspect-Oriented Programming

That last trick we did — dynamically loading an extension defined by a client and integrating it into our flow — is actually part of a wider, fascinating paradigm that I'd like to explore some more. We've seen it before, when we injected a custom hook after an arbitrary line of code, but we didn't dwell on it all that much. In real life, however, solving challenges in such an aspect-oriented way is by far one of the most useful metaprogramming techniques I know — much more so than the difficult and error-prone endeavor of rewriting bytecode.

I say "aspect-oriented" because, historically, that's how this paradigm was known — although we'll assume a somewhat different perspective on it soon enough. Still, let's pay tribute to the original terminology: much like object-oriented programming uses the object as its core abstraction to help us organize our ideas, aspect-oriented programming uses the aspect as its primary organizational concept. It's not a literal language construct, but rather a principle: by thinking about different aspects of our problem, we can separate our code accordingly, maintaining cohesion and composing different aspects only on demand. Of course, there are similar design patterns for objects, functions, and other modular components — the difference is in how it's done.

In object-oriented code, we mix in functionality either with inheritance or composition; the latter — passing all the necessary information as arguments — is how functional and procedural code works as well. With aspects, however, we invert the pattern: using what's known as pointcuts, we effectively signal our role or intent, so some later weaver can imbue our code with additional advice at specific join points; as in, weave specific aspect capabilities into specific places. This works particularly well for what's known as cross-cutting concerns: irrelevant (or rather, orthogonal) considerations that would otherwise become a part of our business logic, thus burdening it with their implementation details and compounding its complexity.

AOP: with Decorators

This is all very abstract, and frankly, the nomenclature is terrible — so let's look at some practical examples, and rephrase our concepts in saner terms as we go along. The usual suspects for cross-cutting concerns are things like logging, security or performance profiling: you know, tasks that are important in and of themselves, but don't actually have anything to do with the rest of our logic. Embedding them inside our code makes everything more complicated, so it would be nice if we could apply them from without. Take this endpoint:

from flask import Flask, request

app = Flask(__name__)

@app.post("/add")
def add_handler():
    x = int(request.form["x"])
    y = int(request.form["y"])
    return str(x + y)
    
if __name__ == "__main__":
    app.run(port=8000)

If we want to log it, secure it and profile its performance, we end up with:

import time

@app.post("/add")
def add_handler():
    print(f"request: {request.method} {request.path}")
    authorization = request.headers.get("Authorization")
    # validate authentication token
    

    started = time.monotonic()
    x = int(request.form["x"])
    print(f"x = {x}")
    y = int(request.form["y"])
    print(f"y = {y}")
    response = str(x + y)
    print(f"response: {response}")
    elapsed = time.monotonic() - started
    print(f"request took {elapsed:.2f} seconds")
    return response

You can see how messy it gets; and these things tend to proliferate, muddying up our entire codebase. One way to get rid of (or rather, contain) them would be with decorators. We might define:

def log(handler):
    def wrapper(*args, **kwargs):
        print(f"request: {request.method} {request.path}")
        for key, value in request.form.items():
            print(f"{key} = {value}")
        response = handler(*args, **kwargs)
        print(f"response: {response}")
        return response
    return wrapper

To handle logging; then:

def secure(handler):
    def wrapper(*args, **kwargs):
        authorization = request.headers.get("Authorization")
        # validate authentication token
        return handler(*args, **kwargs)
    return wrapper

To handle security; and finally:

def profile(handler):
    def wrapper(*args, **kwargs):
        started = time.monotonic()
        response = handler(*args, **kwargs)
        elapsed = time.monotonic() - started
        print(f"request took {elapsed:.2f} seconds")
        return response
    return wrapper

To profile performance. Those decorators are in fact our aspects, and the effects they have is their advice. The join point is the target, our handler; and the pointcuts — that is, the declaration of what should happen where — are the decorations themselves:

@app.post("/add")
@log
@secure
@profile
def add_handler():
    x = int(request.form["x"])
    y = int(request.form["y"])
    return str(x + y)

Notice how we've managed to clean the code up, all the way back to its original form, while still reaping all the benefits. We didn't inherit from any base class and didn't pass any additional arguments. Thanks to Python's native support for decorators, what composition this entails (add = log(secure(profile(add)))) happens implicitly, behind the scenes, so everything looks tidy and declarative.

Weaving

So far, so good; nothing earth-shattering, really, except some strange terminology to describe otherwise familiar concepts. It's not even entirely fair to say that we did any clever weaving, since the decorators are applied to our handler one after the other at the moment of its definition, simply enriching it into an ever more sophisticated function.

There is, however, a fourth pointcut hiding in that example, which I glibly skipped; and it also takes the shape of a decorator: @app.post("/add"). The aspect it addresses is that of a web application: if you think about it, we didn't bind any sockets or parse any HTTP, and yet when we do app.run(), our code is magically integrated into a full-fledged server that does all of this for us. In other words, Flask itself is the weaver, applying the various aspects of web applications — connection handling, request parsing, response formatting, etc. — to our uninterested business logic, expressed in decorated handlers.

This is also known as inversion of control: instead of our code calling into a third-party library to get something done, which would qualify as regular control flow, we actually define pieces of code to be called by a third-party library. When we do app.run(), we effectively run a Flask program, not our program; it calls us, not the other way around. Admittedly, this is exactly what we want: to run a webserver, not a custom script, and just "fill in the gaps", defining what exactly this webserver does when – so a simple way to define handlers is precisely what we expect from a web framework.

The key word here, I think, is just that: framework – and the reason metaprogramming lies at its heart is that frameworks are, in essence, code that takes other code, written by someone else, and does something with it. As such, it can't know in advance what this other code is going to be like — so it needs to establish conventions and interfaces for future integrations to be possible (and ideally, smooth).

A different web framework, like the standard http.server module, might provide similar functionality through inheritance. Want to see what it looks like? Take a deep breath:

from http.server import BaseHTTPRequestHandler, HTTPServer
import urllib.parse

class WebApp(BaseHTTPRequestHandler):
    def do_POST(self):
        content_length = int(self.headers["Content-Length"])
        post_data = self.rfile.read(content_length).decode()
        params = urllib.parse.parse_qs(post_data)
        x = int(params["x"][0])
        y = int(params["y"][0])
        result = x + y
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(str(result).encode())
        
if __name__ == "__main__":
    server = HTTPServer(("", 8000), WebApp)
    server.serve_forever()

This gets significantly worse when we want to handle errors or support multiple paths, but let's not go there. Flask's genius was that it managed to drastically reduce such clutter; it distilled the aspects so well that a single decoration and a friendly request object became all we need. What's more, Python's native decorators allowed it to do so in an exceedingly elegant fashion, with not much magic behind the scenes. As other web frameworks (Starlette, Sanic, FastAPI) adopted this syntax, we came to take it so much for granted that recasting it into the somewhat arcane idea of aspect-oriented programming might seem a bit forced — but bear with me.

AOP: with Code Analysis

Let's look at a different but no less popular example that doesn't use decorators, but reveals the exact same paradigm upon closer examination. The de-facto standard for testing in Python is the Pytest library; and since it employs much heavier metaprogramming than mere decorators, I hope it will elucidate the connection I'm trying to draw. To give an example of how it works, let's stick to our simple function, albeit no longer in handler form. In example.py, we write:

def add(x, y):
    return x + y

And to test it, we write test.py:

from example import add

def test_add():
    assert add(1, 2) == 3
    assert add(1, 0) == 1
    assert add(1, -1) == 0
    assert add(1, 0.5) == 1.5

Then we can run pytest test.py, which triggers our test and prints out whether it passed or failed. Once again, compare it to the standard unittest module:

import unittest
from example import add

class TestAdd(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add(1, 2), 3)
        self.assertEqual(add(1, 0), 1)
        self.assertEqual(add(1, -1), 0)
        self.assertEqual(add(1, 0.5), 1.5)
        
if __name__ == "__main__":
    unittest.main()

No doubt Pytest's version is cleaner — and when it comes to more complex scenarios, it gets exponentially more so. But before we discuss it, have you noticed something odd about our test file? If you look it over again, you'll see it doesn't have any invocation equivalent to unittest.main() — so how exactly did it run, if all we did was define a function?

Well, it didn't; it's Pytest that ran; and its main convention is that it collects and executes all the functions starting with test (from all the files starting with test); try renaming the function check_add and see it disappear from the tests. In other words, its pointcuts — the signals that indicate we expect some advice to be woven in — are different from those in Flask; instead of a decorator, they are detected by inspecting function (and file) names. Of course, it could have used decorators:

from example import add
from pytest import test

@test
def add_test():
    # … same as before

But seeing as real test files contain dozens of such functions, all of which would have to be decorated, why add this hassle? Better rely on a naming convention, especially since most of these functions would be called test_something anyway; and that's just the tip of the iceberg when it comes to Pytest's innovations.

Dependency Injection

One of the key challenges of framework design, when we use inversion of control, is parameterization: when it's us calling other code, we can pass whatever arguments we want — but when other code calls us, how does it know what parameters we might need to perform our task? In Flask, we've seen it encapsulated by the request object:

@app.post("/add")
def post_add():
    x = int(request.form["x"])
    y = int(request.form["y"])
    return str(x + y)

Although it's somewhat irregular that it's a global variable (from flask import request); a rather dubious design decision on Flask's part, if you ask me, and one that only works because different handlers run in different threads, so thread-local storage lets it present as different requests at once. A more standard approach would be to require each handler to have a fixed signature, an agreed-upon interface — like, accepting a request:

@app.post("/add")
def post_add(request):
    # … same as before

Incidentally, this is exactly what happens in newer web frameworks (Starlette, Sanic), so we can write it off to Flask being a pioneer in its domain. Anyway, this technique is known as dependency injection: since we've inverted control, whatever dependencies we might need (like the request's body or headers) are injected into our code by the framework, and we should account for that when filling in our snippets.

Still, the request object is cheating a bit, isn't it? We actually depend on x and y, but since it can't know that in advance, it provides a generic object with soft mappings, which might or might not contain our keys; a more responsible implementation would do:

@app.post("/add")
def post_add():
    try:
        x = int(request.form["x"])
        y = int(request.form["y"])
    except (KeyError, ValueError): # key is missing or not a number
        return "expected x and y as numbers", 400
    return str(x + y)

FastAPI solves this more elegantly: it lets us use type annotations to define what we expect to receive, and takes care of the parsing aspect for us:

from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn

app = FastAPI()

class AddRequest(BaseModel):
    x: int
    y: int
    
@app.post("/add")
async def post_add(request: AddRequest) -> str:
    return str(request.x + request.y)
    
if __name__ == "__main__":
    uvicorn.run(app, port=8000)

This can be achieved by inspecting the handler's __annotations__, pre-parsing the request body according to the structures defined therein, and passing them in instead of the "raw" request object. If we'd want to implement something similar in Flask, all we'd have to do is:

def post(app, url):
    def decorator(handler):
        @app.post(url)
        def parser_handler():
            args = []
            for name, annotation in handler.__annotations__.items():
                if name == "return":
                    continue
                arg = annotation(request.form[name])
                args.append(arg)
            return handler(*args)
        return parser_handler
    return decorator
    
# use @post(app, url) with annotations instead of @app.post(url) with request
@post(app, "/add")
def post_add(x: int, y: int) -> str:
    return str(x + y)

It's pretty neat; and what's neater still is that we can add informative errors if some values are missing or invalid without complicating our handler one bit – validation, if you will, is another cross-cutting concern. Pytest, however, does something infinitely cooler.

Dependency Request

Suppose we have a function that writes something to a file:

def hello(path):
    open(path, "w").write("Hello, world!")

And we'd like to test it. To do so properly, we'd have to create a temporary file, and make sure to clean it up whether the test succeeds or fails:

import os, tempfile
from example import hello

def test_hello():
    _, path = tempfile.mkstemp()
    try:
        hello(path)
        assert open(path).read() == "Hello, world!"
    finally:
        os.remove(path)

This is not terrible, but it's pretty annoying, especially since this pattern recurs a lot — it's really an aspect we'd expect a testing framework to take care of, wouldn't you agree? And indeed, Pytest does: all we have to do is add the tmp_path parameter to our function, and it will make sure to invoke it with an actual temporary directory as its argument, taking care of the allocation and cleanup for us:

def test_hello(tmp_path):
    path = tmp_path / "file.txt"
    hello(path)
    assert path.read_text() == "Hello, world!"

Now, this is just beautiful; and it's not simple dependency injection, either. A handler in a web framework always depends on the same request object, so it's reasonable to expect its signature to be handler(request) and simply pass it in. But this? This actually only applies to test functions with a tmp_path parameter; test functions, that is, that are explicitly asking for it. If we'd want to test a function that prints something to the standard output — another annoying scenario to orchestrate — Pytest similarly provides us with a utility, just by a different name:

def hello():
    print("Hello, world!")
    
def test_hello(capsys):
    hello()
    stdout, stderr = capsys.readouterr()
    assert stdout == "Hello, world!\n"

In other words, it's more of a dependency request: by the mere act of defining our signature, which we'd have to do anyway, we tell Pytest everything it needs to know – that is, what we expect to receive – simply by sticking to conventional names. And that's not all: we can create such utilities, known as fixtures, ourselves. Suppose we were testing a complex object that takes some effort to construct and configure; instead of repeating its preparation in each test, we could define it in a separate function once, and turn it into a reusable utility of our own:

@pytest.fixture
def o():
    o = Object()
    # prepare…
    return o

Then, simply adding the o parameter to a test's signature will prompt Pytest to call our o(), acquire a fresh copy and pass it in – matching the dependency to its injector by the fact that they are called by the same name. Pytest even supports dependency requests in fixtures, so if our tests expect a particular directory structure, we can do:

@pytest.fixture
def d(tmp_path):
    # create files and subdirectories in tmp_path…
    return tmp_path

Have Pytest allocate and clean up the temporary directory, but otherwise set it up ourselves. And the funny thing is — this utter magic is not even that difficult to implement; it's the idea that's so novel. To see the parameters of a function f, we can check f.__code__.co_varnames[:f.__code__.co_argcount]; or, if we're up to date on our standard library, use inspect.signature:

>>> import inspect

>>> def inject_dependencies(f, deps):
...     sig = inspect.signature(f)
...     kwargs = {}
...     for key, value in deps.items():
...         if key in sig.parameters:
...             kwargs[key] = value
...     return f(**kwargs)

So if we have the following functions:

>>> def f0():
...     print("f0")

>>> def f1x(x):
...     print("f1x", x)

>>> def f1y(y):
...     print("f1y", y)

>>> def f2(x, y):
...     print("f2", x, y)

And the following dependencies (which might as well have been callable fixtures):

>>> deps = {"x": 1, "y": 2}

What inject_dependencies does is filter out the values whose keys appear in the function's parameters, and pass them — and only them — on as arguments:

>>> inject_dependencies(f0, deps)
f0
>>> inject_dependencies(f1x, deps)
f1x 1
>>> inject_dependencies(f1y, deps)
f1y 2
>>> inject_dependencies(f2, deps)
f2 1 2

Setup and Teardown

Some fixtures, like tmp_path, need to run before and after the test — setting something up (like creating a temporary directory), and tearing it down when its over (removing that directory). It used to be that the way to configure such a fixture was:

import pathlib, pytest, tempfile, shutil

@pytest.fixture
def tmp_path(request):
    path = tempfile.mkdtemp()
    def remove_directory():
        shutil.rmtree(path)
    request.addfinalizer(remove_directory)
    return pathlib.Path(path)

That is, create a temporary directory, register a callback (using the special request dependency) to run at the end of the test and remove it, and return the actual dependency to inject. Later on this syntax was adopted instead:

@pytest.fixture
def tmp_path(request):
    path = tempfile.mkdtemp()
    yield pathlib.Path(path)
    shutil.rmtree(path)

Because a single yield-statement is the perfect syntax for a function that needs to run "before" and "after" — just like what contextlib.contextmanager does. If we'd want to revise inject_dependencies to work with fixtures and support this case, we'd have to tell apart regular callables, which we simply invoke to acquire the dependency they mean to inject, and such generators, which we'd have to start, run for a single step, and then resume after our target function has finished. In other words:

>>> def inject_dependencies(f, deps):
...     sig = inspect.signature(f)
...     kwargs = {}
...     finalizers = []
...     for key, value in deps.items():
...         if key not in sig.parameters:
...             continue 

...         if inspect.isgeneratorfunction(value):
...             generator = value()
...             dep = next(generator)
...             finalizers.append(generator)
...         else:
...             dep = value()
...         kwargs[key] = dep
...     try:
...         return f(**kwargs)
...     finally:
...         for generator in finalizers:
...             try:
...                 next(generator)
...             except StopIteration: # ignore generator's natural end
...                 pass

This way, if we have:

>>> def x():
...     return 1

>>> def y():
...     print("before")
...     yield 2
...     print("after")

>>> deps = {"x": x, "y": y}

We get:

>>> inject_dependencies(f2, deps)
before
f2 1 2
after

Just like we wanted.

Parsing Tracebacks

The reason I'm telling you all this is, well — it's just so delicious, and as we've already established, we are here first and foremost to have fun. But more than that, the point I'm trying to make in this chapter is that metaprogramming is not just a bunch of quirky, advanced topics; for aspect-oriented programming — that is, writing frameworks — it's a set of techniques that are quite indispensable, and I don't see it portrayed as such very often.

In fact, I don't know that even Pytest's core developers thought about it in these terms when they started. When the library just came out, its main feature, its core strength, was those simple assert statements we've seen in our very first example:

def test_add():
    assert add(1, 2) == 3

You see, there's a very good reason that tests before Pytest looked like this:

class TestAdd(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add(1, 2), 3)

It's not that unittest's developers were stupid and forgot that Python has a builtin assert-statement; it's that this statement was deemed insufficient, at least on its own. If we try using it outside Pytest, and deliberately break our test:

>>> assert add(1, 0) == 3
AssertionError

That's what we get: an empty assertion error. The reason, by the way, has to do with the precedence of evaluation: first, add(1, 0) is evaluated as 1, then 1 == 3 is evaluated as False, and then assert False raises an exception. It doesn't know what went wrong, just that it did. To counteract this inherent limitation, assert-statements actually support a second argument, intended to describe what exactly was it that incited their ire:

>>> assert add(1, 0) == 3, "add(1, 0) != 3"
AssertionError: add(1, 0) != 3

But writing tests like this would be extremely tedious. The alternative, self.assertEqual, at least lets the framework come up with an intelligible error on its own — its implementation might be something like:

class TestCase:
    def assertEqual(self, arg1, arg2):
        if arg1 != arg2:
            raise AssertionError(f"{arg1} != {arg2}")

Here, add(1, 0) is an argument, so it's evaluated before assertEqual is called; the reason for the test failing ends up as 2 != 1. It's not ideal, but it's better than nothing, or having to spell everything out ourselves, which were the only two conceivable options at the time. Not so, thought Pytest's developers; the people who wrote unittest came from much less dynamic languages like Java and C++, and they replicated the same design pattern used in those languages, in frameworks such as jUnit or GoogleTest. Python, after all, can do much more — its AssertionError, despite being empty, has a traceback object attached, which can help reconstruct what went wrong. If we step out of the interpreter for a moment, put everything in a file and run it:

def add(x, y):
    return x + y
    
def test_add():
    assert add(1, 0) == 3
    
if __name__ == "__main__":
    test_add()

$ python example.py
Traceback (most recent call last):
File ".../example.py", line 8, in <module>
    test_add()
File ".../example.py", line 5, in test_add
    assert add(1, 0) == 3
           ^^^^^^^^^^^^^^
AssertionError

We see that somehow, Python knows which line is the culprit; and like we've seen when we played with the ast module, this information can be retrieved and analyzed in runtime. If we were to write our own Pytest, then, I'd suggest:

import sys

def main():
    code = open(sys.argv[1]).read()
    tests = collect_tests(code)
    for test in tests:
        try:
            test()
            print(f"{test.__name__} passed")
        except AssertionError as error:
            print(f"{test.__name__} failed:")
            analyze_error(code, error)
            
if __name__ == "__main__":
    main()

That is, collect all the functions starting with test in a given file, run them, and if any of them fails, investigate further. Collecting the functions should be quite trivial by now:

def collect_tests(code):
    namespace = {}
    exec(code, namespace)
    return [test for name, test in namespace.items()
            if name.startswith("test")]

As for parsing their traceback, a few words are in order. Much like its first printed line suggests, it's organized with the most recent call (which we want) last. It's effectively an inverse of frame objects: where each frame has an f_back attribute pointing "outwards" to the frame that called it, tracebacks start from the outermost scope, where the exception was caught, and have a tb_next attribute pointing "inwards". So if we do:

def analyze_error(code, error):
    traceback = error.__traceback__
    while traceback.tb_next: # drill down to the innermost frame
        traceback = traceback.tb_next
    line_number = traceback.tb_lineno
    line = code.splitlines()[line_number - 1].strip()
    print(f"  on line {line_number}: {line}")

We'll get a message that's already considerably nicer than what was thought possible:

$ python test.py example.py
test_add failed:
  on line 5: assert add(1, 0) == 3

But Pytest did better — and we can, too. Tracebacks don't only hold a line number; they actually have a live frame object in tb_frame, which can give us its globals and locals, which lets us re-evaluate add(1, 0) and see why it didn't equal 3. Admittedly, I tailored the code to our particular use case – but you can easily imagine how it might be extended to accommodate other scenarios:

import ast

def analyze_error(code, error):
    # … same as before
    frame = traceback.tb_frame
    ErrorTransformer(frame.f_globals, frame.f_locals).visit(ast.parse(line))

class ErrorTransformer(ast.NodeTransformer):

    def __init__(self, globals_, locals_):
        self.globals = globals_
        self.locals = locals_
        
    def visit_Call(self, node):
        call = ast.unparse(node) # python 3.9+
        result = eval(call, self.globals, self.locals)
        print(f"    where {call} = {result}")
        return node

And we have:

$ python test.py example.py
test_add failed:
on line 5: assert add(1, 0) == 3
    where add(1, 0) = 1

Framework Design

Like I said, I don't think the Pytest developers were particularly preoccupied with aspect-oriented theory when they were writing their library: they just figured they can use Python's crazy dynamicity to do some really cool stuff, and seem to have had a knack for it, because they kept coming up with one creative solution after the other. The result is the most beautiful and versatile testing framework I've seen — and my feeling is that it has to do with the degrees of metaprogramming involved, so let's see if we can generalize our conclusions into a broader lesson in software design (and finally revise that awful terminology, like I promised).

My claim is this: when developing a framework (that is, "weaver"), we are primarily concerned with handling certain aspects (good term, let's keep it) of a common problem ("cross-cutting concern") and letting users fill in their business logic by defining what I like to call plugins ("join points") — because, well, it's pieces of code that we plug into the framework. In Flask, it's the decorated handlers; in Pytest, the test_ functions; in other cases, it might be classes (SQLAlchemy) or entire modules (Django) that are hoisted and integrated into some bigger whole.

However, because frameworks necessarily invert control, we have to come up with conventions that will handle plugin integration — first and foremost, some registration method ("pointcuts"): how we communicate to the framework which plugin should go where. In Flask, we use a decorator to mark them; in Pytest, we match them by name; in other cases, we might inherit from a base class (Twisted), or pass our plugins into a particular function (Starlette).

In any case, what makes a framework great is not just its featureset — that's what makes any third-party library great, so it's irrelevant to its quality as a framework. To assess the latter, we should ask how easy it is to develop plugins for it, and how seamlessly those plugins are integrated. After all, the framework is developed once, for some express purpose, by people who are well-versed in its problem domain; the plugins, on the other hand, are developed by a myriad people, for a myriad reasons, over and over again – for every line of code in Pytest, there are thousands of lines of tests that use it. So even if a framework solves a burning need, the degree to which it manages to abstract it away and provide friendly interfaces that translate what users might want into what has to happen — that, I think, is its mark of excellence.

And this is where metaprogramming comes in. Honestly, all of the above can be done without it — but this might lead to what's known as boilerplate code: tedious incantations we have to recite in our codebase, not to solve a particular problem, but to appease whatever framework we're using. And since Python is so delightfully introspective, we can use its reflective abilities to reduce such noise — effectively inferring what to do from subtle, indirect, and generally declarative signals, whether it's names, annotations, signatures, even docstrings.

This reminds me of a cute little framework I once used, called PLY: Python Lex-Yacc, intended to provide the lexical parsing functionality of the popular lex and yacc tools; essentially, a means for creating our own domain-specific language. In it, we'd use regular expressions to specify which tokens our language consists of (e.g. [0-9]+), and possibly define some post-processing for each (like converting a string of digits into an actual number). So in effect, we needed to pair regular expressions with functions – something a different framework might have done with a tedious mapping:

>>> number_pattern = r"[0-9]+"

>>> def parse_number(token):
...     return int(token.value)

>>> # and later:
>>> framework.register_tokens({
...     number_pattern: parse_number,
...     # … other tokens …
... })

Or, more elegantly, with a decorator:

>>> @framework.token(r"[0-9]+")
... def parse_number(token):
...     return int(token.value)

PLY, however, came up with a brilliant idea: since these small parsing functions don't really make use of their documentation string, why not put the regular expression there?

>>> def t_NUMBER(token):
...     r"[0-9]+"
...     return int(token.value)

>>> t_NUMBER.__doc__ # easy enough to access
'[0-9]+'

It also paid attention to names (functions starting with t_ would be used to parse tokens; whatever came next, like NUMBER, would be used later on to articulate the semantics in which they're to be used, etc.); and generally made use of Python's capabilities to their fullest extent, resulting in a neat little framework we'll revisit later.

A Note on Decorators

To be fair, if I had to choose just one of Python's features that is most invaluable for framework development, it would be decorators. At the end of the day, names and signatures have some constraints, whereas a decorator, being a function call, can pretty much accommodate any and all needs simply by receiving this or that argument. Of course, it's nicer when we don't have to add any extra marks — but it's also important not to overstretch the syntax of plugin definition into something too bizarre (regex-docstrings are about as far as I would go), and admit it when a clear mark is called for. Flask, for example, could have put the URL paths in the docstring:

>>> def add():
...     "/add"
...     # … same as before

But actually, the @app.post decoration also indicated which HTTP method is acceptable (as opposed to e.g. @app.get). We could account for that by requiring a specific prefix:

>>> def POST_add():
...     "/add"
...     # … same as before

But what about a handler that can serve both GET and POST requests? In Flask, we simply use @app.route("/add", methods=["GET", "POST"]); without it, we'd have to come up with some contrived convention:

>>> def GET_POST__add():
...     "/add"
...     # … same as before

Like expecting a handler name with two underscores, just so we can split the prefix itself by single ones. That's already getting ugly, which is what I meant by "overstretching the syntax".

But where do decorators fit within the aspect-oriented paradigm? Do they even qualify as metaprogramming, and if not, how does it sit with our conclusion on framework design? In my opinion, because of their native support and elegant syntax, we sort of take them for granted — but actually they're very much in line with everything we discussed. For starters, a decorator is literally code that works on other code: it's a function that receives a function and returns a function. But I would say something even stronger; when we do:

>>> def double(f):
...     def wrapper(*args, **kwargs):
...         return 2 * f(*args, **kwargs)
...     return wrapper

>>> @double
... def add(x, y):
...     return x + y

We actually invert control:

>>> add
<function double.<locals>.wrapper at 0x…>

That is, add(1, 2) no longer calls our original function, but something else, which calls our function at its discretion. It's also the reason wrappers have (*args, **kwargs) for a signature — effectively, a lazy way of coordinating interfaces, making sure that the wrapper can pass the original function whatever parameters it might expect.

So in a sense, a decorator is a miniature framework: it takes care of some aspect of a common problem by accepting a function as its plugin. Its registration method — adding a decoration line on top — just happens to be a native enough part of the language so as to make it feel like it isn't a big deal. The big deal in this case, though, is having a language so geared towards framework development that it makes it so seamless we don't even notice.