A Metaprogramming Framework

From 20 Lines to a Framework

In the previous chapter, we've seen how to implement a full-fledged templating engine in less than 20 lines of code. We've done so by separating our template into code lines (starting with !) and text lines, transpiling the latter into print(f"...") statements with a stack of indentation levels, and executing the result with print substituted for a function that collects the output — effectively, generation followed by execution, or code that generates code that generates text.

At the end, however, we noticed a problem: validate's body was indented incorrectly, and we said we'd fix it later. Well, later is now; and while it can certainly be patched, such an oversight can be indicative of deeper issues, which is why I'd like to take a different approach — namely, develop this tiny engine into a full-blown templating framework.

In fact, the one we're going to develop is the one I use most of the time: Auryn, a metaprogramming engine for templating and code generation (it's even on PyPI, pip install auryn), and over this chapter and the next, we'll cover all its key aspects, using the same names as in its API, so that by the end of it you'll be able to switch to the real thing and feel right at home. For now, though, let's look at what exactly is wrong with our current solution.

First, there's the indentation bug, which is actually quite trivial to reproduce: just a simple template that defines a function to generate code, and invokes it from an indented position. If you run it through chapter 11's execute function, you'll see the problem:

>>> from ch11 import execute
>>> template = """
... !def check_integer(name, min=None, max=None):
...     if not isinstance({name}, int):
...         raise ValueError(f"expected {{{name}=}} to be an integer")
...     !if min is not None:
...         if {name} < {min}:
...             raise ValueError(f"expected {{{name}=}} >= {min}")
...     !if max is not None:
...         if {name} > {max}:
...             raise ValueError(f"expected {{{name}=}} <= {max}")
... def validate(n):
...     !check_integer("n", 1, 10)
... """
>>> print(execute(template))
def validate(n):
if not isinstance(n, int):
    raise ValueError(f"expected {n=} to be an integer")
if n < 1:
    raise ValueError(f"expected {n=} >= 1")
if n > 10:
    raise ValueError(f"expected {n=} <= 10")

That is, the if statements are indented flush left instead of inside validate. The issue here is that while generating the text, we baked in the indentation of each line relative to the position at which it was written; but in the case of check_integer, we wrote it at the top of the template, and called it from somewhere else, which we had no way of knowing in advance. This means that in order to support code generation, we can't consider indentation a generation-time constant, and need to account for it as an execution-time state.

The second problem is with braces. We actually did something quite clever by delegating the interpolation to Python's f-strings, but it comes with the unfortunate side-effect that every curly brace in our template is considered interpolation. This means that if we try to generate something that has a lot of curly braces, like JavaScript, CSS, C or JSON, it's going to break immediately:

>>> print(execute("""
... function hello() {
...     return "Hello, world!";
... }
... """))
  File "<string>", line 1
    print(f'function hello() {')
                              ^
SyntaxError: f-string: expecting '}'

In this case, print(f'function hello() {') is not even a valid f-string, so we get a SyntaxError before anything even happens — and the only workaround is doubling up on the braces ourselves, which rather defeats the purpose of having a template that looks like its output. Still, it's a solvable problem: we can come up with our own interpolation scheme that doesn't clash with so many languages, and implement it ourselves.

The third problem, and the most important one, is that our tiny templating engine is closed: there's no way for us to include one template in another, to compose pages from a shared layout, or to add any new syntax to it at all; everything is hardcoded in one function, so every new feature means editing the engine itself. Recall how we talked about framework design in the aspect-oriented programming chapter: such a project is developed once, but its plugins are written by many people, many times — so the quality of such a project is not so much in how it solves problems, but how easy it is for its users to write their own plugins and seamlessly integrate them into the system. In this case, the "plugins" are of two types: templates and extensions of the template language itself — and this is precisely the core design principle behind Auryn: the whole point of two-phase generation is to make it as easy as possible to extend the syntax of its meta-language, so that most of its builtin features are actually implemented as such plugins, and they are quite trivial — somewhere between 2 and 20 lines of code.

To give you a taste of what we're going to accomplish, and the fun we're going to have along the way, here's a short outline. First, we'll consider how to parse templates into trees of lines; then, we'll implement our own generation/execution object — GX, for short; then, we'll see how to add our own interpolation, with custom delimiters; then, we'll tackle a third kind of line that runs during generation — what's known as macros; then, we'll see how to compose templates, and how to add plugins with macros and hooks; we'll also address errors that can intelligently point to the offending template line, and conclude with a few words on how little of all this is actually fixed. What's more, at every step, we'll reuse a trick from earlier in the book — from frames, to execing into namespaces, to context managers and naming conventions, to tracebacks and more.

Templates as Trees

So let's make the first of several changes. Our 20-line engine processed the template in one pass: it read a line, decided what to do with it, and moved on. In a framework, however, we might want to look ahead, so we'll parse it first, and generate it second. A macro might want to repeat its nested lines, or skip them, or store them for later, or emit them raw — so let's arrange them in a tree, where every line knows its children, and we'll be able to generate them as we please. This is the same separation of concerns that compilers make, between parsing and code generation.

Indentation as a structural device isn't even our invention: it's how Python itself is parsed, and its own tokenizer converts it to INDENT and DEDENT tokens. In our case, since we're dealing with a line-based format, our tree is a tree of lines:

# auryn.py:
class Line:

    def __init__(self, number, indent, content):
        self.number = number
        self.indent = indent
        self.content = content
        self.children = Lines()
        
    def __repr__(self):
        return f"<line {self.number}: {self.indent} | {self.content}>"
        
    def shift(self, offset):
        self.indent += offset
        for child in self.children:
            child.shift(offset)
            
class Lines(list):

    def snap(self, to=0):
        for line in self:
            line.shift(to - line.indent)
        return self
        
    def to_string(self):
        output = []
        for line in self:
            output.append(" " * line.indent + line.content)
            if line.children:
                output.append(line.children.to_string())
        return "\n".join(output)

Let's walk through it. A line has a number (which we'll need for error messages), an indentation, its content, and its children. The shift method simply moves a line and its entire subtree by the specified offset. The Lines class is just a list with two utilities, to_string to turn lines back to text, and snap — which we'll get to in a moment. As for parse:

# auryn.py:
def parse(text):
    lines, stack = Lines(), []
    for number, raw in enumerate(text.strip("\n").splitlines(), 1):
        content = raw.strip()
        indent = len(raw) - len(raw.lstrip())
        if not content:  # empty lines stick to the previous line's level
            indent = stack[-1].indent if stack else 0
        while stack and stack[-1].indent >= indent:
            stack.pop()
        line = Line(number, indent, content)
        (stack[-1].children if stack else lines).append(line)
        stack.append(line)
    return lines

It's the same stack trick we saw in the previous chapter, only simpler. For every line, we pop the stack until its top is less indented than the current line; this top is then its parent (or, if the stack is empty, the line is a root line). We then push the line, since it can be the parent of the next one — and note that empty lines take their previous line's indentation, so they don't break a block apart. As for the leading newlines, we just strip them first, so that a triple-quoted template that starts with a newline will still have its first real line as line 1. Now, let's parse a <ul> template with a for-loop, and see how it looks:

>>> from auryn import parse
>>> lines = parse("""
... <ul>
...     !for i in range(n):
...         <li>{i}</li>
... </ul>
... """)
>>> lines
[<line 1: 0 | <ul>>, <line 4: 0 | </ul>>]
>>> lines[0].children
[<line 2: 4 | !for i in range(n):>]
>>> lines[0].children[0].children
[<line 3: 8 | <li>{i}</li>>]

We see that there are two root lines, <ul> and </ul>, the loop is a child of <ul>, and <li> is a child of the loop; indentations are absolute, so <li> is indeed indented by 8 spaces, just like it is in the template. Now, let's get to the subtle part.

You see, in a template, indentation can mean two things at once: it defines the structure, in that one line is nested inside another — and it contributes to the output, in that this line should be indented this much. But when a block is transformed, we often want to discard its structural indentation, because it's not meant to be part of the final result. The body of the !for line, for example, is only indented by 4 spaces so that it's nested under it; these spaces don't need to make it to the output. So let's snap this loop to 0, then to 8:

>>> loop = lines[0].children
>>> print(loop.to_string())
    !for i in range(n):
        <li>{i}</li>
>>> print(loop.snap(0).to_string())
!for i in range(n):
    <li>{i}</li>
>>> loop[0].children
[<line 3: 4 | <li>{i}</li>>]
>>> print(loop.snap(8).to_string())
        !for i in range(n):
            <li>{i}</li>

snap(to) aligns the first level of lines to the specified indentation, and shifts each subtree by the same offset, so as to preserve relative indentation. This way, after snap(0), the loop is indeed at 0, and <li> is at 4, just like we want; it also changes the lines in-place and returns them, so we can chain it in a call.

The way I like to think about snap (and it takes a moment to get used to, as admitted in Auryn's own README) is this: transform these lines as if they had been written right here. It's an interesting idea, and we'll see it often — in fact, nearly every macro we'll write will use it in exactly this way: transform(line.children.snap(line.indent)); that is, continue with my children, but without the indentation that was necessary to make them my children. For now, having organized our template in a convenient data structure, we're ready to move on to the object that will walk it.

Generation/Execution

The centerpiece of Auryn is the GX object, which stands for generation/execution: it holds the lines of our template, the code generated from them, and eventually the output of executing this code. Our two-phase approach lends itself to a pretty obvious division: we can have a generate method and an execute method.

Instead of the simple if-statement that differentiated between ! and everything else, we'll use a dispatch table: a dictionary of line transforms, keyed by their prefix, with the empty prefix being the default one (that is, text). The longest matching prefix wins — and this table is our very first extension point, since whoever can change it can decide what lines mean.

So here's the generation part of GX:

# auryn.py:
class GX:

    def __init__(self, template):
        self.lines = parse(template)
        self.line = None
        self.code = []
        self.code_indent = 0
        self.transforms = {"": transform_text, "!": transform_code}
        self.x_globals = {"emit": self.emit, "indent": self.indent}
        self.output = []
        self.output_indent = 0
        
    def generate(self):
        self.transform(self.lines)
        return self
        
    def transform(self, lines=None):
        if lines is None:
            lines = self.line.children
        for line in lines:
            previous, self.line = self.line, line
            for prefix in sorted(self.transforms, key=len, reverse=True):
                if line.content.startswith(prefix):
                    content = line.content.removeprefix(prefix).strip()
                    self.transforms[prefix](self, content)
                    break
            self.line = previous
            
    def add_code(self, code):
        self.code.append((self.code_indent, code))
        
    @contextlib.contextmanager
    def increased_code_indent(self):
        self.code_indent += 4
        try:
            yield
        finally:
            self.code_indent -= 4
            
    def add_text(self, indent, text):
        self.add_code(f"emit({indent}, f{text!r})")
        
    def to_string(self):
        return "\n".join(" " * indent + content for indent, content in self.code)

Let's go over it; it's pretty straightforward, really. transform iterates over lines, and sets self.line to the line currently being transformed, so that any transforms can inspect it (and its children) without having it passed in explicitly. Then, it peels off the prefix and calls the appropriate transform on the rest, restoring the previous line afterwards, since transforms can recurse. add_code appends a line of code at the current code indentation, and increased_code_indent is a context manager for nesting. add_text is still somewhat naive, in that it generates f-strings, but there's already a twist: it generates emit(indent, ...), passing the text's indentation as an argument rather than baking it in. Here's transform_text and transform_code:

# auryn.py:
def transform_text(gx, content):
    if content:
        gx.add_text(gx.line.indent, content)
    gx.transform()
    
def transform_code(gx, content):
    indent = gx.line.indent
    if indent:
        gx.add_code(f"with indent({indent}):")
        gx.code_indent += 4
    gx.add_code(content)
    with gx.increased_code_indent():
        gx.transform(gx.line.children.snap(0))
    if indent:
        gx.code_indent -= 4

A text line simply emits its content at its indentation, and then transforms its children (preserving their indentation). A code line is where our fix lives: if it's indented (like a loop inside <ul>), we wrap it in with indent(N):, so everything that happens inside it — even if it's defined in some function that's called from elsewhere — is shifted by N at execution time. Then, we add the code itself, and transform its children snapped to 0 (as we don't want their indentation to grow), one code indent further in. So really, it's Python's job to keep track of the code's nesting; and it's the indent context manager's job to keep track of the text's, which we'll implement as part of the execution half:

# auryn.py (continued):
class GX:

    def execute(self, **context):
        self.x_globals.update(context)
        exec(self.to_string(), self.x_globals)
        return "".join(self.output).rstrip("\n")
        
    def emit(self, indent, text):
        self.output.append(" " * (self.output_indent + indent) + text + "\n")
        
    @contextlib.contextmanager
    def indent(self, indent):
        self.output_indent += indent
        try:
            yield
        finally:
            self.output_indent -= indent

execute runs the generated code in x_globals, an execution namespace where emit and indent are bound methods of the GX, so the code calls back into the object that generated it. The output is collected in a list, like the stolen print we used in the previous chapter; and indent is a contextlib.contextmanager (the yield-in-the-middle shape we've already seen in Pytest's fixtures) that increments output_indent upon entry and decrements it upon exit, even if an exception is raised.

So, generating code for our <ul> template now produces this:

>>> from auryn import GX
>>> gx = GX("""
... <ul>
...     !for i in range(n):
...         <li>{i}</li>
... </ul>
... """)
>>> print(gx.generate().to_string())
emit(0, f'<ul>')
with indent(4):
    for i in range(n):
        emit(0, f'<li>{i}</li>')
emit(0, f'</ul>')

Notice how the loop is wrapped in with indent(4):, and the <li> inside it emits at 0. The generated code is short and sweet — a quality that will become important later, when we have to debug it, and when we want to ship it. In the meantime, let's execute it with n=3:

>>> print(gx.execute(n=3))
<ul>
    <li>0</li>
    <li>1</li>
    <li>2</li>
</ul>

And see if our check_integer/validate template from the previous section works now:

>>> gx = GX("""
... !def check_integer(name, min=None, max=None):
...     if not isinstance({name}, int):
...         raise ValueError(f"expected {{{name}=}} to be an integer")
...     !if min is not None:
...         if {name} < {min}:
...             raise ValueError(f"expected {{{name}=}} >= {min}")
...     !if max is not None:
...         if {name} > {max}:
...             raise ValueError(f"expected {{{name}=}} <= {max}")
... def validate(n):
...     !check_integer("n", 1, 10)
... """)
>>> print(gx.generate().to_string())
def check_integer(name, min=None, max=None):
    emit(0, f'if not isinstance({name}, int):')
    emit(4, f'raise ValueError(f"expected {{{name}=}} to be an integer")')
    if min is not None:
        emit(0, f'if {name} < {min}:')
        emit(4, f'raise ValueError(f"expected {{{name}=}} >= {min}")')
    if max is not None:
        emit(0, f'if {name} > {max}:')
        emit(4, f'raise ValueError(f"expected {{{name}=}} <= {max}")')
emit(0, f'def validate(n):')
with indent(4):
    check_integer("n", 1, 10)
>>> print(gx.execute())
def validate(n):
    if not isinstance(n, int):
        raise ValueError(f"expected {n=} to be an integer")
    if n < 1:
        raise ValueError(f"expected {n=} >= 1")
    if n > 10:
        raise ValueError(f"expected {n=} <= 10")

It does: the body of check_integer now emits at 0 and 4 relative to wherever it's called, and the call itself is under with indent(4):, so its output ends up in validate. The bug is gone, and we didn't make the engine smarter at generation time; we simply made one decision later, at execution time, at a point where we have more information. That's the general lesson of our two-phase design: anything we can't know at generation time can be deferred to the runtime, because we own it, too — we decide what's in the execution namespace, so the generated code can rely on it.

As an aside, notice that the triple braces {{{name}=}} still work here, but only because add_text is still producing f-strings; it's next on our list to go.

Interpolation

So far, everything's been happening in f-strings; and if we're honest, it's served us quite well. Why, then, give up such a nice syntax, and resort to our own interpolation?

Well, for starters, the f-strings come with a few problems of their own: their use of curly braces is one — any output we have that contains curly braces of its own (and we'll encounter quite a few) will break our formatting. But there's also the fact that they are, quite frankly, horrible to read, at least when dealing with code — all these nested f-strings and quotes are not easy on the eyes, let alone to debug. And if we're already implementing our own interpolation scheme, we can actually support different delimiters, so we can choose them per template, or even per block.

So instead, let's parse them ourselves, during generation, and generate emit calls whose arguments will be the pieces — literal text as string literals, and expressions as code:

# auryn.py:
def interpolate(text, delimiters="{ }"):
    start, end = delimiters.split()
    i = 0
    while (j := text.find(start, i)) != -1:
        if j > i:
            yield text[i:j], False
        depth, k = 1, j + len(start)
        while depth:
            if k >= len(text):
                raise ValueError(f"unmatched {start!r} in {text!r}")
            if text.startswith(start, k):
                depth, k = depth + 1, k + len(start)
            elif text.startswith(end, k):
                depth, k = depth - 1, k + len(end)
            else:
                k += 1
        yield text[j + len(start) : k - len(end)].strip(), True
        i = k
    if i < len(text):
        yield text[i:], False
        
# … same as before
class GX:

    def add_text(self, indent, text):
        args = []
        for snippet, is_code in interpolate(text, self.interpolation):
            args.append(snippet if is_code else repr(snippet))
        self.add_code(f"emit({indent}, {', '.join(args)})")
        
    # … same as before
    
    def emit(self, indent, *args):
        text = "".join(map(str, args))
        self.output.append(" " * (self.output_indent + indent) + text + "\n")

To break it down, interpolate is a generator that yields (snippet, is_code) pairs. It starts by looking for the start delimiter, then scans forward with a depth counter to account for nested delimiters (like a dictionary literal within braces), and raises an error if it encounters one without finding its corresponding end. add_text turns code snippets into arguments as-is, but wraps literal ones in their repr, and emit is now able to take any number of arguments and join their strs. This should result in the same snippets, either with the default delimiters or with <% %>:

>>> from auryn import GX, interpolate
>>> list(interpolate("<li>{i}: {items[i]['name']}</li>"))
[('<li>', False), ('i', True), (': ', False), ("items[i]['name']", True), ('</li>', False)]
>>> list(interpolate("<li><% i %>: <% items[i]['name'] %></li>", "<% %>"))
[('<li>', False), ('i', True), (': ', False), ("items[i]['name']", True), ('</li>', False)]

And indeed, both result in the same snippets, including the brackets and the quotes of items[i]['name']. This, then, is the code generated for our <ul> template:

>>> gx = GX("""
... <ul>
...     !for i in range(n):
...         <li>{i}</li>
... </ul>
... """)
>>> print(gx.generate().to_string())
emit(0, '<ul>')
with indent(4):
    for i in range(n):
        emit(0, '<li>', i, '</li>')
emit(0, '</ul>')

That is, emit(0, '<li>', i, '</li>'): the expression i is evaluated by Python as a regular argument, and there's no parsing happening at execution time. This is character for character what the real Auryn generates.

As for the trade-offs we'll have to accept to stay small, our scanner doesn't understand Python strings, so a closing delimiter inside an expression's string will confuse it (Auryn's skips over quoted strings), and there's no escaping it (Auryn's does that, too, treating a doubled delimiter as a literal one).

But the whole point of configurable delimiters is that we can have code that is otherwise full of braces — and the classic example here is JavaScript, whose blocks are braces, and whose template literals are ${...}, which is interpolation of its own:

>>> gx = GX("""
... function greet(name) {
...     return `Hello, ${name}! I am <% me %>.`;
... }
... """)
>>> gx.interpolation = "<% %>"
>>> print(gx.generate().to_string())
emit(0, 'function greet(name) {')
emit(4, 'return `Hello, ${name}! I am ', me, '.`;')
emit(0, '}')
>>> print(gx.execute(me="Auryn"))
function greet(name) {
    return `Hello, ${name}! I am Auryn.`;
}

That is, ${name} is left untouched for JavaScript to interpolate in the browser, while <% me %> is interpolated by us, to be resolved during execution; for now, we just set gx.interpolation manually, but we'll make it a line in our template in the next section. We can also rewrite our check_integer template to use <% %>:

>>> gx = GX("""
... !def check_integer(name, min=None, max=None):
...     if not isinstance(<% name %>, int):
...         raise ValueError(f"expected {<% name %>=} to be an integer")
...     !if min is not None:
...         if <% name %> < <% min %>:
...             raise ValueError(f"expected {<% name %>=} >= <% min %>")
...     !if max is not None:
...         if <% name %> > <% max %>:
...             raise ValueError(f"expected {<% name %>=} <= <% max %>")
... def validate(n):
...     !check_integer("n", 1, 10)
... """)
>>> gx.interpolation = "<% %>"
>>> print(gx.generate().execute())
def validate(n):
    if not isinstance(n, int):
        raise ValueError(f"expected {n=} to be an integer")
    if n < 1:
        raise ValueError(f"expected {n=} >= 1")
    if n > 10:
        raise ValueError(f"expected {n=} <= 10")

Notice how our generated validation code has f-strings of its own, {n=}; and notice how they are just text now, not code — this is actually how Auryn's own README writes its typechecking example. Compare them to the triple braces in the previous chapter, {{{name}=}}, where one pair of braces was used to escape a brace for the outer f-string, and another to interpolate. In other words, when generating code that itself has templates, f-strings, or interpolation, it's not some edge case, it's the norm; so having the delimiters be ours to decide was a must.

Macros

Our next type of line will be one that starts with %, which we'll call a macro line. Such lines will be evaluated during generation: text lines will become output, code lines will become generated code, and macro lines will become code that writes code — which means we're now dealing with two namespaces: g_globals, for generation, where our macros will live, and x_globals, for execution. The syntax is simple enough: %name will invoke the name(gx) function, and %name argument will invoke name(gx, 'argument'). If you want to pass a line as-is as Python arguments, just add a colon: %name: arguments, which will invoke name(gx, arguments), passing them verbatim. The real Auryn actually has a third form, using : to split arguments by whitespace and :: to pass them as-is, but for us two should be enough:

# auryn.py:
MACRO = re.compile(r"^(\w+)(?:(:?)\s+(.*))?$")

# … same as before

def transform_macro(gx, content):
    name, colon, arg = MACRO.match(content).groups()
    if not arg:
        call = f"{name}(gx)"
    elif not colon:
        call = f"{name}(gx, {arg!r})"
    else:
        call = f"{name}(gx, {arg})"
    exec(call, gx.g_globals)

Walking through it, the MACRO regex captures the name, an optional colon and the argument. Then, we create a call string and exec it in our generation namespace, which we've prepared with the gx object. This is dynamic code execution, just like we saw in the chapter of that name, where a dictionary was used to represent a module: here, we do the same for our generation namespace. Note that the macros expect the GX as their first argument, much like methods expect their object via self.

For now, we'll assume that all our macros are defined in auryn.py, and that GX.__init__ collects all the module-level functions starting with g_ into its generation namespace (minus the prefix). This is a rather unceremonious way of registration, relying on a naming convention — much like Pytest expects functions starting with test_ to be tested. We'll generalize it to proper plugins soon enough, but for now, let's check out the derive and extend functions, as well as the %include macro:

# auryn.py:
class GX:

    def derive(self, template):
        gx = GX(template, root=self.root)
        gx.transforms = self.transforms.copy()
        gx.g_globals = {**self.g_globals, "gx": gx}
        gx.interpolation = self.interpolation
        return gx
        
    def extend(self, gx):
        for indent, content in gx.code:
            self.code.append((self.code_indent + indent, content))
            
    # … same as before
    
def g_include(gx, template):
    included = gx.derive(template)
    included.lines.snap(gx.line.indent)
    included.generate()
    gx.extend(included)

Let's go over it: derive creates a GX for another template, but prepares it for continuing the current generation: the same transforms, the same macros, the same delimiters. Then, %include derives a GX, snaps its lines to the indentation of the %include line, generates it, and uses extend to merge its code into ours, at the current code indentation. That is to say, it includes it on the level of generated code, not on the level of the output: the included template is effectively incorporated into the including one, becoming part of the same program. If we have our loop in loop.aur, we can include it inside a <ul>:

>>> from auryn import GX, execute
>>> print(execute("""
... <ul>
...     %include loop.aur
... </ul>
... """, n=3))
<ul>
    <li>0</li>
    <li>1</li>
    <li>2</li>
</ul>

And this will generate the following code:

>>> gx = GX("""
... <ul>
...     %include loop.aur
... </ul>
... """)
>>> print(gx.generate().to_string())
emit(0, '<ul>')
with indent(4):
    for i in range(n):
        emit(0, '<li>', i, '</li>')
emit(0, '</ul>')

Notice how it's pretty much indistinguishable from writing the same loop inline: at the moment of execution, there's no trace of our include statement. This is what the C preprocessor's #include directive does, as we've seen in the previous chapter's history of code generation — and it does so with raw text. We're doing the same, but for generated code.

Now, where do we find loop.aur? It's relative to the including template's file, or if the template was passed in as a string, then relative to the module that called us. To find that module, we can use the same trick as in dynamic code execution: start from the current frame, and keep going "up" with f_back until the code object no longer resides in auryn.py itself. In the interpreter, the caller's "file" is a pseudo-path like <stdin>, which doesn't actually exist — so we fall back to the current directory (the real Auryn actually stores all this as part of the GX's origin):

# auryn.py:
def caller_directory():
    frame = inspect.currentframe()
    while frame and frame.f_code.co_filename == __file__:
        frame = frame.f_back
    path = pathlib.Path(frame.f_code.co_filename) if frame else None
    return path.parent if path and path.exists() else pathlib.Path.cwd()

Next, let's add the %interpolate macro. When used with children, it changes the delimiters to the specified ones, but only for that block; and when used without, it changes them for the rest of the template. The block version uses a context manager called patch, which we can use to temporarily set attributes on an object and restore them afterwards.

>>> print(execute("""
... <script>
...     %interpolate <% %>
...         function hello() {
...             alert("Hello, <% name %>!");
...         }
... </script>
... <p>Hello, {name}!</p>
... """, name="world"))
<script>
    function hello() {
        alert("Hello, world!");
    }
</script>
<p>Hello, world!</p>

Let's see how it works: inside the block, the JavaScript can use <% name %>, but after it {name} works again; and since the children are snapped to the same indentation as the %interpolate line itself, it leaves no trace in the output. If we were to use a macro that didn't exist:

>>> gx = GX("""
... %hello world
... """)
>>> gx.generate()
NameError: name 'hello' is not defined
>>> sorted(gx.g_globals)
['__builtins__', 'gx', 'include', 'interpolate']

We'd get a NameError, but note that it doesn't complain about a non-existent macro — just a non-existent name in the generation namespace, which actually gives us a pretty good idea of how to add one: we just need to put a function there. This is exactly what plugins will do; but first, let's add a few more macros — ones that will make our templates composable.

Template Composition

Websites, in their essence, are layouts with holes: the head, the header and the footer are the same across all the pages, which only define their own titles and bodies. In Jinja and Django, this is known as template inheritance: the concept of a block, which we can both define and insert.

To do this, we'll use three macros: %define, %insert and %extend. The way it works is this: somewhere, we define a block; then, elsewhere, we insert it into some layout; and finally, we extend this layout, defining its blocks in a particular way. Of course, this implies some way of communicating between different macros, because a block is defined in one template and inserted in another; so, we add a state dictionary, which we share between a GX and all its derivatives. We also need %extend to happen last, after the entire template has been seen, so we add on_complete — callbacks that are invoked when generation is complete:

# auryn.py:
class GX:

    def on_complete(self, postprocessor):
        self.postprocessors.append(postprocessor)
        
    # … same as before
    
def g_define(gx, name):
    gx.state.setdefault("blocks", {})[name] = gx.line.children
    
def g_insert(gx, name):
    blocks = gx.state.get("blocks", {})
    if name in blocks:
        gx.transform(blocks[name].snap(gx.line.indent))
    else:  # default content
        gx.transform(gx.line.children.snap(gx.line.indent))
        
def g_extend(gx, template):
    def replace_code(gx):
        base = gx.derive(template)
        base.generate()
        gx.code = []
        gx.extend(base)
    gx.on_complete(replace_code)

Let's go over it: %define stores the line's children in the state, but note that these are unprocessed template lines, not generated code. This way, when we generate a block, we can do so anywhere it's inserted, just by transforming those lines snapped to its indentation, as if they had been written there. If the block is not defined, then %insert transforms its own children instead, which we can use as default content.

As for %extend, it's a reverse inclusion: it generates the base template instead of the current one — but only after all its blocks have been defined, and these definitions happen further down, in the very template that we're generating. So, it registers a callback that generates the base template and replaces our code with its code — pytest's request.addfinalizer from the aspect-oriented programming chapter has a similar shape, where we ask to be called back at the end. Anyway, here's how we can define and insert a block in one template:

>>> from auryn import GX, execute
>>> print(execute("""
... %define greeting
...     <p>Hello, {name}!</p>
... <div>
...     %insert greeting
... </div>
... """, name="world"))
<div>
    <p>Hello, world!</p>
</div>

Now, suppose base.aur looks like this:

# base.aur:
<html>
    <head>
        <title>
            %insert title
                Untitled
        </title>
    </head>
    <body>
        %insert body
    </body>
</html>

So it defines a title block, which it defaults to Untitled, and a body block. If we were to extend it like so:

>>> print(execute("""
... %extend base.aur
... %define body
...     <h1>{title}</h1>
...     <ul>
...         %include loop.aur
...     </ul>
... """, title="Numbers", n=2))
<html>
    <head>
        <title>
            Untitled
        </title>
    </head>
    <body>
        <h1>Numbers</h1>
        <ul>
            <li>0</li>
            <li>1</li>
        </ul>
    </body>
</html>

We get a page with our heading and a loop. Notice how the title falls back to Untitled, and how the body's %include works, despite being buried inside a block that's inserted into another template. This is because we pass the state and the root directory to any derived GX, so they can orient themselves accordingly.

And that's it; this is the code generated for such an extending template:

>>> gx = GX("""
... %extend base.aur
... %define title
...     {title}
... %define body
...     <h1>{title}</h1>
... """)
>>> print(gx.generate().to_string())
emit(0, '<html>')
emit(4, '<head>')
emit(8, '<title>')
emit(12, title)
emit(8, '</title>')
emit(4, '</head>')
emit(4, '<body>')
emit(8, '<h1>', title, '</h1>')
emit(4, '</body>')
emit(0, '</html>')

That is, a flat sequence of emit calls, with absolute indentations. Blocks, inheritance and includes are all resolved at generation time, so when we actually execute the code there's nothing left to look up and nothing left to merge — and that's why the compilation approach of Jinja and Mako, as opposed to Django's interpretation we saw in the previous chapter, renders fast; and we did all this in some 20 lines of Python.

Auryn's core plugin actually adds %ifdef/%ifndef for conditional blocks, required inserts, and %bookmark/%append for backtracking (as in, "wait; I need to add something to the head in the body of the page, even though I've already emitted it"); and there's an interesting side effect to the fact that we're storing the lines rather than the code: we can actually insert the same block multiple times, in different places with different indentations, and each %insert will generate it anew, snapped to where it stands.

Still, these are all hardcoded in auryn.py, so no user can add one without editing it. It's time to open the framework up.

Plugins and Hooks

At this point, we can open our framework to the world. All we need to do is move the macros out of auryn.py and into separate plugin modules — in fact, all the functionality we've developed so far will become plugins/core.py, which we'll load by default. Other plugins will be loaded either by path or by name (from the plugins directory), using a %load macro — which, conveniently, is also in the core plugin. In the terms of the aspect-oriented programming chapter, the registration method is a naming convention: any functions starting with g_ will be considered macros, and added to the generation namespace; any functions starting with x_ will be considered hooks, and added to the execution namespace; and we'll also allow an on_load(gx) to run when the plugin is loaded, in which it can reconfigure the GX as necessary. It's the same dynamic extensions trick from the autocompletion chapter: execute a module, harvest its functions.

# auryn.py:
class GX:

    def load(self, plugin):
        if isinstance(plugin, dict):
            namespace = plugin
        else:
            path = self.root / plugin
            if not path.is_file():
                path = PLUGINS / f"{plugin}.py"
            namespace = {}
            exec(compile(path.read_text(), str(path), "exec"), namespace)
        for name, value in namespace.items():
            if name.startswith("g_"):
                self.g_globals[name.removeprefix("g_")] = value
            if name.startswith("x_"):
                self.x_globals[name.removeprefix("x_")] = value.__get__(self)
        if "on_load" in namespace:
            namespace["on_load"](self)

Let's go over it: we accept a plugin that can either be a dictionary, in which case we're dealing with a ready namespace — or a path, which we resolve relative to the template's directory (falling back to the builtin plugins directory). We compile the module with its file name, so its tracebacks are accurate, then execute it into a fresh namespace; and then we peel off the prefixes and register the functions. To bind the hooks we use value.__get__(self): it's a little known fact that functions are also descriptors, and that's exactly what __get__ does — binds them to a particular object, which is precisely what makes methods aware of their self. So a hook can be written as a plain module function, accepting the GX as its first parameter — and when it's invoked by the generated code, it won't be passed in explicitly. Here's an example, with a %hello macro:

# greetings.py
def g_hello(gx, name):
    gx.add_text(gx.line.indent, f"Hello, {name}!")

We can now load it with:

>>> from auryn import GX, execute
>>> print(execute("""
... %load greetings.py
... <p>
...     %hello world
... </p>
... """))
<p>
    Hello, world!
</p>

Macros, then, happen during generation; hooks are what we call from the generated code during execution. There's a nice principle in Auryn's README, which I'll paraphrase: macros generate code, but any complexity that needs to happen during execution is better encapsulated in a hook, rather than generated inline; this way, the generated code remains short and declarative, and we only need to write the necessary logic once, in plain Python.

Let's take a concrete example that requires both: we want a macro like %assign name, which would capture its output into a variable, instead of emitting it. This might be useful if we want to post-process it somehow; but capturing the output can only be achieved at execution time. Here's how the %assign macro and its x_assign hook are defined in the core plugin:

# plugins/core.py:
def g_assign(gx, name):
    gx.add_code("with assign() as _:")
    with gx.increased_code_indent():
        gx.transform(gx.line.children.snap(0))
    gx.add_code(f"{name} = ''.join(_).rstrip()")
    
@contextlib.contextmanager
def x_assign(gx):
    output = []
    with gx.patch(output=output, output_indent=0):
        yield output

Let's go over it: the macro generates with assign() as _:, transforms its children inside it, and then assigns the joined result. The hook is a context manager that patches gx.output with a fresh list (and resets output_indent), for the duration of the block, then restores them. emit is none the wiser: it simply appends to whatever self.output is. It's the same patch we've seen in %interpolate already, during generation; it's our universal trick of swapping an attribute temporarily, for the duration of a block. Here's how we can use it:

>>> print(execute("""
... %assign items
...     !for i in range(n):
...         item {i}
... <p>{", ".join(items.splitlines())}</p>
... <p>{len(items.splitlines())} items</p>
... """, n=3))
<p>item 0, item 1, item 2</p>
<p>3 items</p>

And here's what this generates (and the hook, bound to the execution namespace):

>>> gx = GX("""
... %assign items
...     !for i in range(n):
...         item {i}
... <p>{", ".join(items.splitlines())}</p>
... """)
>>> print(gx.generate().to_string())
with assign() as _:
    for i in range(n):
        emit(0, 'item ', i)
items = ''.join(_).rstrip()
emit(0, '<p>', ", ".join(items.splitlines()), '</p>')
>>> gx.x_globals["assign"]
<bound method x_assign of <GX of <string>>>

Note that assign is actually a bound method of our GX; the same pattern again, of a macro generating with hook():, which in turn redirects the output. We'll see it again in the next chapter, when we learn how files are written. For now, just note that plugins can also reconfigure the GX when they are loaded, via on_load: we'll see what this can mean at the end of this chapter.

Where Did It Go Wrong?

All this power comes at a price, though: errors happen in code we never actually wrote. A line in a template becomes code that invokes a hook, which was defined in a plugin — so if an error occurs, Python's traceback will point to line 4 of <string>, which is about as descriptive as ERROR: unknown error.

To address this, real Auryn devotes a whole module to error reporting: it provides the context, the template traceback and the code traceback. Let's cover the essential parts.

There are two phases, and two kinds of errors to consider. Those that happen during generation are pretty straightforward, since they happen in macros; and we actually know where we are, because gx.line isn't reset when an exception propagates.

The execution errors, however, happen in generated code — so we need to map the failing line of that code back to the template line that produced it. This is pretty much all the bookkeeping we need: in add_code, we record, for every line of generated code, its GX and the template line number it came from. In extend, we copy these records over, so that if some code is generated by an included template, it keeps pointing to it.

# auryn.py:
class GX:

    def execute(self, **context):
        self.x_globals.update(context)
        code = self.to_string()
        filename = f"<generated by {self!r}>"
        linecache.cache[filename] = (len(code), None, code.splitlines(True), filename)
        try:
            exec(compile(code, filename, "exec"), self.x_globals)
        except Exception as error:
            traceback = error.__traceback__
            line_number = None
            while traceback:
                if traceback.tb_frame.f_code.co_filename == filename:
                    line_number = traceback.tb_lineno
                traceback = traceback.tb_next
            if line_number:
                _, content, source, number = self.code[line_number - 1]
                template_line = source.text.strip("\n").splitlines()[number - 1].strip()
                error.add_note(f"  in {source.name}, line {number}: {template_line}")
            raise
        return "".join(self.output).rstrip("\n")

Let's see how it works. First, we register the generated code in linecache, using a made-up file name — that's where Python's traceback machinery looks for the corresponding source lines, so it can show them as part of the traceback (an entry with None for modification time is never invalidated as stale). Then, we compile the code with that file name; and if it fails, we walk the traceback's tb_next chain inwards, like we did when we were dealing with parsing tracebacks in the aspect-oriented programming chapter, until we get to the innermost frame that's actually running our code — whose tb_lineno attribute indexes our records.

Now, instead of wrapping the exception, we just add a note to it using add_note (Python 3.11+), which will be printed after the error message. This way, the exception keeps its type — so if some code expects a ZeroDivisionError, it will still catch it, but it'll also be told where exactly in the template this happened:

>>> from auryn import execute
>>> print(execute("""
... <ul>
...     !for i in range(n):
...         <li>{100 // i}</li>
... </ul>
... """, n=3))
ZeroDivisionError: integer division or modulo by zero
  in <string>, line 3: <li>{100 // i}</li>

Note how the note points to line 3, <li>{100 // i}</li>, and not to the generated emit call.

Generation errors are treated similarly: in generate, we catch anything that slips through the cracks, add a note with the current template line, and raise it again. Since each derived GX does the same, we can actually collect one note per level of inclusion if an error happens in an included template, like broken.aur, which uses an unknown macro:

# broken.aur:
<p>
    %hello world
</p>

And if we include it:

>>> print(execute("""
... <div>
...     %include broken.aur
... </div>
... """))
NameError: name 'hello' is not defined
  in broken.aur, line 2: %hello world
  in <string>, line 2: %include broken.aur

Note how we actually get two notes: one for broken.aur, line 2, and one for the string we're including it in, line 2.

If we put the whole picture together in a script, it might look like this: a page that extends base.aur, defines a body that includes items.aur, and passes in some items that don't have the name attribute necessary to render them:

# page.py
import auryn

print(auryn.execute("""
%extend base.aur
%define body
    <h1>{title}</h1>
    %include items.aur
""", title="Items", items=[1, 2, 3]))

And we run it like so:

$ python page.py
Traceback (most recent call last):
  File ".../page.py", line 4, in <module>
    print(auryn.execute("""
          ~~~~~~~~~~~~~^^^^
    %extend base.aur
    ^^^^^^^^^^^^^^^^
    ...<2 lines>...
        %include items.aur
        ^^^^^^^^^^^^^^^^^^
    """, title="Items", items=[1, 2, 3]))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../auryn.py", line 281, in execute
    return GX(template).generate().execute(**context)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File ".../auryn.py", line 208, in execute
    exec(compile(code, filename, "exec"), self.x_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<generated by <GX of <string>>>", line 12, in <module>
    emit(0, '<li>', item.name, '</li>')
                    ^^^^^^^^^
AttributeError: 'int' object has no attribute 'name'
  in items.aur, line 3: <li>{item.name}</li>

Once again, walk with me: the frame <generated by <GX of <string>>>, line 12, shows us the generated line emit(0, '<li>', item.name, '</li>'), thanks to linecache; and the note actually maps it to line 3 of items.aur, even though we got there via %extend, %define and %include. Tracebacks, like everything else in Python, are just data — and as such, they can be enriched however we like.

A Framework for Frameworks

Let's take a step back, and consider what we have: it's a pretty nifty framework, all things considered, and it clocks in at a few hundred lines. What's more, most of its features are tucked away in plugins — so what, exactly, is fixed? To find out, let's implement text lines and code lines as macros — something suggested in Auryn's own README:

# lines.py
def g_text(gx, text):
    gx.add_text(gx.line.indent, text)
    gx.transform()
    
    
def g_code(gx, code):
    gx.add_code(f"with indent({gx.line.indent}):")
    with gx.increased_code_indent():
        gx.add_code(code)
        with gx.increased_code_indent():
            gx.transform(gx.line.children.snap(0))

And use it like so:

>>> from auryn import GX, execute
>>> gx = GX("""
... %load lines.py
... %text <ul>
...     %code for i in range(n):
...         %text <li>{i}</li>
... %text </ul>
... """)
>>> print(gx.generate().to_string())
emit(0, '<ul>')
with indent(4):
    for i in range(n):
        emit(0, '<li>', i, '</li>')
emit(0, '</ul>')
>>> print(gx.execute(n=2))
<ul>
    <li>0</li>
    <li>1</li>
</ul>

If you walk it through, you'll find that the generated code is exactly the same as what we had in our native syntax: ! and plain lines are just convenient spellings for two particular entries in the dispatch table. What's actually baked into the framework, then, is not a lot: a tree of lines, a table of transforms to apply to each according to its prefix, two namespaces, a buffer of generated code and a buffer of output. And really, the deeper move here is on_load: a plugin can replace the default transformation, and thus change the meaning of a plain line.

In paragraphs.py, for example, we can wrap every line in a paragraph:

# paragraphs.py
def on_load(gx):
    gx.transforms[""] = transform_paragraph
    
    
def transform_paragraph(gx, content):
    if content:
        gx.add_text(gx.line.indent, f"<p>{content}</p>")
    gx.transform()

And use it thusly:

>>> print(execute("""
... %load paragraphs.py
... Hello, world!
... !for i in range(n):
...     This is paragraph #{i + 1}.
... """, n=2))
<p>Hello, world!</p>
<p>This is paragraph #1.</p>
<p>This is paragraph #2.</p>

That's the entire idea in five lines: the template language is not fixed, so a plugin can decide that indented lines are paragraphs (or paths on disk, or anything else), while still accommodating code lines, macros and interpolation around it.

We've come a long way: in the domain-specific languages chapter, we designed languages — now we built a machine to make them, from within. And we did so with macros: user-defined code generation, just like we had imagined in the metaprogramming layer chapter, where users literally write functions that generate code.

If we map our little engine to Auryn, we'd have GX with parse, generate and execute; Template, Lines and Line; transform, add_code, add_text, snap, state, on_complete, patch, derive and extend; g_ macros, x_ hooks and on_load; and a core plugin that is loaded by default. What it adds, among other things, are origins and full error reports; %! lines that run code during generation, with %eval and %emit to pass values from generation to execution; %raw; %inline and %strip for emitting several lines as one; %bookmark and %append; %param for declaring parameters; a CLI; and standalone code, which we'll build in the next chapter.

The reason I'm telling you all this is to show you just how much we can get away with — and why we can get away with so much. Because Python does most of the heavy lifting for us — the insight from the previous chapter was that our intermediate representation is, in fact, Python, so all our flow control constructs (loops, functions, recursion, exceptions) are already taken care of. More than that, our "bytecode" is not some obscure, binary mess, but a high-level language that we can actually read, introspect and patch.

So far, we generated text; in the next and final chapter, we'll see lines becoming files and directories, shell commands, web pages, components and React; and how, eventually, our generated code can leave the engine behind entirely — as well as what all of this has to do with the book you're currently reading.