Dan Gittik Let’s talk

Writing/Metaprogramming: An Adventure

Chapter 13 · Draft · 40 min read

Code Generation

Directory Structures

In the last chapter, we built ourselves a little metaprogramming engine. In this one, we're going to put it to work.

We'll tackle a different problem in every section, pointing our shiny new tool at it and seeing what we can do. We'll scaffold projects, write scripts, generate web pages and components and even React; we'll cache our generated code, figure out a way to ship it without the engine, and render this very book, all with the same basic approach — and without any new theory, really, just different plugins.

One of the most common tasks a programmer has to do is to create a directory and put some files in it, which themselves have to contain some boilerplate. Whether it's django-admin startproject or npm create, cookiecutter or copying the last project and renaming a few things — it's always the same, really: a tree of directories and files with some text at its leaves. And a tree of directories and files with text at its leaves is pretty much an indentation-based template waiting to happen.

Auryn actually ships with a filesystem plugin for this purpose, but we'll write our own, because it's not that difficult. Once it's loaded, all we need to do is change what plain lines mean: any line ending with / should be considered a directory, and any other line should be considered a file; lines nested beneath a file should be considered its content. Code lines and interpolation should work as usual, even in paths. You might wonder why we don't just write a Python script with os.makedirs calls; but when we use a script to build a tree, we are effectively turning the tree inside out, aren't we? In this case, the template looks like the result — so we can see what the project will look like before we even create it.

Here's what such a project template might look like, and the code it generates: 

>>> import auryn
>>> template = """
... %load filesystem
... {name}/
...     README.md
...     {name}/
...         __init__.py
...     tests/
...         !for i in range(n):
...             test_{i}.py
... """
>>> print(auryn.generate(template))
with directory(s(name, '/')):
    with file(s('README.md')):
        pass
    with directory(s(name, '/')):
        with file(s('__init__.py')):
            pass
    with directory(s('tests/')):
        for i in range(n):
            with file(s('test_', i, '.py')):
                pass
>>> auryn.execute(template, name="proj", n=2)
''

Let's walk it through: the nesting structure in the template is translated to nesting structure in the code, which is translated to a nesting structure on the filesystem. The s(...) is a small core hook that concatenates the interpolated pieces into a string, which is useful for macros that need an interpolated value rather than an emitted line (gx.interpolated is what builds such expressions); and to keep Python happy, empty files have a pass. This should result in: 

$ find proj | sort
proj
proj/proj
proj/proj/__init__.py
proj/README.md
proj/tests
proj/tests/test_0.py
proj/tests/test_1.py

Here's what the plugin's on_load, path transform and directory hook look like: 

# plugins/filesystem.py:
def on_load(gx):
    gx.state["text_transforms"] = gx.transforms.copy()
    gx.transforms[""] = transform_path
    gx.transforms["$"] = transform_shell
    
def transform_path(gx, content):
    path, _, source = content.partition(" ")
    if path.endswith("/"):
        add_directory(gx, path, source, gx.line.children)
    else:
        add_file(gx, path, source, gx.line.children)
        
def add_directory(gx, path, source, children):
    gx.add_code(f"with directory({gx.interpolated(path)}):")
    with gx.increased_code_indent():
        before = len(gx.code)
        if source:
            for entry in sorted((gx.root / source).iterdir()):
                if entry.is_dir():
                    add_directory(gx, f"{entry.name}/", entry, [])
                else:
                    add_file(gx, entry.name, entry, [])
        if children:
            gx.transform(children.snap(0))
        if len(gx.code) == before:
            gx.add_code("pass")
            
@contextlib.contextmanager
def x_directory(gx, path):
    cwd = os.getcwd()
    pathlib.Path(path).mkdir(parents=True, exist_ok=True)
    os.chdir(path)
    try:
        yield
    finally:
        os.chdir(cwd)

Let's walk it through again: on_load stashes the original transforms in the state, because we'll need them when we're generating file contents, and then installs transform_path as the default, and transform_shell for any lines starting with $. A path line should be either a directory or a file. add_directory generates a with-statement on the directory hook, and then transforms the children inside it, snapped to 0, since their nesting is now carried by the code's indentation (that source part is for the next section). x_directory is a context manager that creates the directory, changes into it, and changes back in a finally clause — that setup-and-teardown shape we've seen before in Pytest's yield fixtures. Paths are relative to the current directory at the time of execution, so the tree should appear wherever we run this.

Compare this to the naive alternative suggested in Auryn's README: a macro that generates os.mkdir and os.chdir calls inline. It works, but it doesn't have the same vibe, does it? The hook lets us keep the generated code declarative, while putting all the necessary logic in one place, in plain Python. And as for the plugin itself, it hijacks the default line transform with the same few lines we used in the paragraphs plugin to close out the previous chapter — except here, we use it for something else: to turn plain lines into paths.

Files and Shells

Of course, directories alone are not enough: we need some files, too. Those are where our content actually goes — and the content, we said, is ordinary template text; so inside a file, our original text transforms are back. We can patch gx.transforms with the ones we stashed away, so they are available as the file's children are transformed; and let's also add the shell transform and hook while we're at it: 

# plugins/filesystem.py (continued):
def add_file(gx, path, source, children):
    gx.add_code(f"with file({gx.interpolated(path)}):")
    with gx.increased_code_indent():
        before = len(gx.code)
        with gx.patch(transforms=gx.state["text_transforms"]):
            if source:
                gx.extend(gx.derive(str(source)).generate())
            if children:
                gx.transform(children.snap(0))
        if len(gx.code) == before:
            gx.add_code("pass")
            
@contextlib.contextmanager
def x_file(gx, path):
    output = []
    with gx.patch(output=output, output_indent=0):
        yield
    pathlib.Path(path).write_text("".join(output))
    
def transform_shell(gx, content):
    command, _, options = content.partition(" # ")
    into = eval(f"dict({options})").get("into", "_")
    gx.add_code(f"{into} = shell({gx.interpolated(command)})")
    
def x_shell(gx, command):
    return subprocess.run(command, shell=True, capture_output=True, text=True).stdout

Let's walk through it. x_file is exactly like %assign — it replaces the output with a fresh list (and output_indent with 0), lets the block run, and writes the result to file. The text inside it doesn't even know it's going into a file: it's just a block, like any other.

And while we're here, we can add a source as well: another template, generated in place with derive and extend. We can similarly add a source to add_directory: given a real directory, we walk it and add its entries — every file as a template. That's a cookiecutter in a dozen lines.

But we can do better. Consider shell lines: $ command will run a command at execution time, and $ command # into="name" will store its standard output in an execution variable. This is just a thin wrapper around subprocess.run, as you can see in x_shell. And we can use all of this to generate a project: 

# project.aur:
%load filesystem
{name}/ skeleton
    README.md
        # {name.title()}
        Generated with a mini-Auryn.
    {name}/
        __init__.py
    tests/
        !for i in range(n):
            test_{i}.py
                def test_{i}():
                    assert {i} + 1 == {i + 1}
    run_tests.sh
        python -m pytest tests
    $ chmod +x run_tests.sh

And here are the skeleton directory's templates: 

# skeleton/pyproject.toml:
[project]
name = "{name}"
version = "0.1.0"
requires-python = ">=3.13"

# skeleton/docs/index.md:
# {name.title()}
!for i in range(n):
    - [Test {i}](../tests/test_{i}.py)

And this is why I like it — because it's so simple, but so powerful. The fact that the source is a template makes all the difference: now, it's not just a directory of files we can copy over, but a scaffolding tool. The skeleton can be edited, versioned and reviewed just like any other code — and the language used to describe it is exactly the same as the one we've already been using. All we need to do is execute it: 

>>> import auryn
>>> os.chdir("out")
>>> auryn.execute("../project.aur", name="proj", n=2)
''

And we get (including the MANIFEST.txt we'll generate in a moment): 

$ find proj -not -path "*/.git*" | sort
proj
proj/docs
proj/docs/index.md
proj/MANIFEST.txt
proj/proj
proj/proj/__init__.py
proj/pyproject.toml
proj/README.md
proj/run_tests.sh
proj/tests
proj/tests/test_0.py
proj/tests/test_1.py
$ cat proj/tests/test_1.py
def test_1():
    assert 1 + 1 == 2
$ cat proj/docs/index.md
# Proj
- [Test 0](../tests/test_0.py)
- [Test 1](../tests/test_1.py)
$ ls -l proj/run_tests.sh | cut -c1-10
-rwxr-xr-x

The pyproject.toml and the docs/index.md came from the skeleton, interpolated with the project's name; in fact, the latter even has a loop, because skeleton files are full templates. And run_tests.sh is executable, because of the $ chmod line. (Empty lines are removed by our text transform, by the way; in the actual Auryn, they are only emitted when asked, by an empty % line.)

This is already pretty cool, but it gets cooler still when you consider what shell lines, combined with code lines, allow us to do: I'd even call it generative scripting. The shell is good for what it's good for — running programs, finding files — and the Python code is good for what the shell is bad for, namely loops, data and string manipulations. The template, then, describes a result — so we can use this to generate, say, a MANIFEST.txt based on the output of find: 

>>> auryn.execute("""
... %load filesystem
... {name}/
...     $ find . -name "*.py" | sort # into="paths"
...     MANIFEST.txt
...         !for path in paths.split():
...             {path.removeprefix("./")}
... """, name="proj")
''

Which results in: 

$ cat proj/MANIFEST.txt
proj/__init__.py
tests/test_0.py
tests/test_1.py

That is, find is run inside the project directory, because that's where its shell line is nested; its output is stored in paths; and then a code line iterates over it to generate the desired file. As a bonus of having two phases, print(generate(template)) will show us exactly what our script is going to do before doing anything, a dry run for free — something most deployment scripts can't do.

Web Pages

We've come full circle: two chapters ago, we started with Jinja2 rendering a list of HTML elements, the kind of thing that web servers do all day. So let's put our engine where templating engines go: behind a web framework. To keep everything ours, I'll even write this framework, and it won't take more than 50 lines. I know what you're thinking: "why bother, when we have Flask?" — but hear me out. First of all, I'm not trying to compete with Flask. Second, every piece of this framework is a trick we've already seen before, so it's a nice way to review everything we've learned — and third, in two sections, we'll actually want to introspect this framework's routes. For now, though, behold web.py: 

# web.py: a tiny web framework
import http.server
import inspect
import json
import re


class App:

    def __init__(self):
        self.routes = {}
        
    def get(self, path):
        def decorator(handler):
            self.routes[path] = handler
            return handler
        return decorator
        
    def dispatch(self, url):
        for path, handler in self.routes.items():
            pattern = re.sub(r"\{(\w+)\}", r"(?P<\1>[^/]+)", path)
            if match := re.fullmatch(pattern, url):
                kwargs = {}
                for name, parameter in inspect.signature(handler).parameters.items():
                    kwargs[name] = parameter.annotation(match[name])
                return handler(**kwargs)
        raise LookupError(url)
        
    def serve(self, port):
        app = self
        
        class Handler(http.server.BaseHTTPRequestHandler):
            def do_GET(self):
                try:
                    result = app.dispatch(self.path)
                except LookupError:
                    return self.send_error(404)
                if isinstance(result, str):
                    body, content_type = result, "text/html"
                else:
                    body, content_type = json.dumps(result), "application/json"
                self.send_response(200)
                self.send_header("Content-Type", content_type)
                self.end_headers()
                self.wfile.write(body.encode())
                
        http.server.HTTPServer(("localhost", port), Handler).serve_forever()

So let's go over it: @app.get(path) is used to register handlers, and we're using the decorator-as-registration-method trick from the aspect-oriented programming chapter. dispatch is used to parse paths (turning {post_id} placeholders into named regular expression groups) and inject path parameters by name (converting them with the handler's annotation, post_id: int), which is the FastAPI-style dependency injection we implemented in that chapter; and serve is used to wrap everything up in one BaseHTTPRequestHandler, that verbose standard-library class we took a deep breath before, and which we only have to write once — returning strings as HTML, and everything else as JSON.

So far, so good; but let's try it on something more substantial, like a small blog. In app.py, let's have: 

# app.py
import dataclasses

import auryn
from web import App


@dataclasses.dataclass
class Post:
    id: int
    title: str
    content: str
    
    
posts = [
    Post(1, "Hello, world", "My first post."),
    Post(2, "Metaprogramming", "It's an adventure."),
]
app = App()


def page(template, **context):
    return auryn.execute(f"pages/{template}", **context)
    
    
@app.get("/")
def index():
    return page("index.aur", posts=posts)
    
    
@app.get("/posts/{post_id}")
def post(post_id: int):
    return page("post.aur", post=posts[post_id - 1])
    
    
if __name__ == "__main__":
    app.serve(8000)

page is actually our entire view layer, distilled down to one call to auryn.execute — and note how the template paths are relative to app.py, which the engine figures out by walking the stack, so this server can run from any directory. Here are our base layout, our index page and our post page: 

# pages/base.aur:
<!DOCTYPE html>
<html>
    <head>
        <title>
            %insert title
                My Blog
        </title>
    </head>
    <body>
        %insert body
    </body>
</html>

# pages/index.aur:
%extend base.aur
%define body
    <h1>Posts</h1>
    <ul>
        !for post in posts:
            <li><a href="/posts/{post.id}">{post.title}</a></li>
    </ul>
    
# pages/post.aur:
%extend base.aur
%define title
    {post.title}
%define body
    <h1>{post.title}</h1>
    <p>{post.content}</p>

Note how %extend, %define and %insert are used to compose the pages, and how interpolation reaches into attributes like {post.title}; because really, it's just Python code evaluated in a namespace we control, which means there's no separate expression language to learn, like we complained about templating languages before. At this point, we're ready to serve our blog on localhost: 

$ curl -s localhost:8000/
<!DOCTYPE html>
<html>
    <head>
        <title>
            My Blog
        </title>
    </head>
    <body>
        <h1>Posts</h1>
        <ul>
            <li><a href="/posts/1">Hello, world</a></li>
            <li><a href="/posts/2">Metaprogramming</a></li>
        </ul>
    </body>
</html>
$ curl -s localhost:8000/posts/2
<!DOCTYPE html>
<html>
    <head>
        <title>
            Metaprogramming
        </title>
    </head>
    <body>
        <h1>Metaprogramming</h1>
        <p>It's an adventure.</p>
    </body>
</html>
$ curl -s -o /dev/null -w "%{http_code}\n" localhost:8000/nope
404

Notice the 404 we get for a path that doesn't match any of our handlers — and notice, too, that every request does the following: parses the template, loads the plugins, runs the macros, compiles the code, and only then executes it. We'll come back to that point in a moment.

For now, I want to draw a lesson about frameworks: when you have your own templating engine, the templating part of a web framework is actually quite tiny; Flask's integration with Jinja, for example, is basically our page function, and everything else (layouts, includes, custom delimiters for script-heavy pages) is part of the engine. What you can't do with server-side templates, though, is react to the user after the page has been sent: and that's what today's web is all about, with its components and single-page applications (usually written in React).

Components

Of course, modern frontends don't think in pages, but in components — reusable pieces of user interface that can be customized by passing in properties (or props) and nesting content inside them. Templating engines have includes and blocks; a component is their more structured cousin, a template with parameters and a slot.

To add support for such components, we'll make a new plugin. Every file in a components/ directory will be turned into a macro with that name, so using a component will look like using a tag: %Card title=post.title, with the nested lines as its children; and in the component, %content will be used to put them in place. That's how vynil, the renderer that typesets this book, does it: 

# plugins/components.py
import contextlib


def on_load(gx):
    for path in sorted((gx.root / "components").resolve().glob("*.aur")):
        gx.g_globals[path.stem] = component(path)
        
        
def component(path):
    def g_component(gx, props=""):
        gx.state.setdefault("content", []).append(gx.line.children)
        rendered = gx.derive(str(path))
        rendered.lines.snap(gx.line.indent)
        rendered.generate()
        gx.add_code(f"with props({props}):")
        with gx.increased_code_indent():
            gx.extend(rendered)
        gx.state["content"].pop()
    return g_component
    
    
def g_content(gx):
    content = gx.state["content"][-1]
    gx.transform(content.snap(gx.line.indent))
    
    
@contextlib.contextmanager
def x_props(gx, **props):
    previous = {name: gx.x_globals[name] for name in props if name in gx.x_globals}
    gx.x_globals.update(props)
    try:
        yield
    finally:
        for name in props:
            del gx.x_globals[name]
        gx.x_globals.update(previous)

And that's how it works: on_load globs the directory and registers a macro for each file by calling component, which returns a function that closes over the path — so we're making macros at load time, because functions are values. The macro itself pushes the caller's children onto a stack in the state (so that components can nest), derives a GX for the component's file, snaps it to the caller's position, generates it, and wraps its code in a with props(...):.

Since props are passed in as raw text that goes into this with-statement, they are evaluated at execution time — so post.title is not going to be available until then. To make them available, x_props makes them global names in the execution namespace for the duration of the block, restoring whatever they might have shadowed after it's over — dynamic scoping, like in Lisp.

As for the stack, it's a bit more subtle: a component's content might very well contain other components — for example, a Link inside a Card, or a card inside a card; so for every call, its children are pushed, and then popped when it's done, so %content always refers to the innermost component's caller. For example: 

# pages/components/Card.aur:
<div class="card">
    <h2>{title}</h2>
    <div class="card-body">
        %content
    </div>
</div>

# pages/components/Link.aur:
<a href="{href}">{text}</a>

And an index page can use them like so: 

# pages/index.aur:
%load components
%extend base.aur
%define body
    <h1>Posts</h1>
    !for post in posts:
        %Card title=post.title
            <p>{post.content}</p>
            %Link href=f"/posts/{post.id}", text="Read more"

Note the %Link props: href=f"/posts/{post.id}", text="Read more"; since the entire argument string is going to end up in a Python call, they are actually keyword arguments, f-strings included. Rendering it, we get: 

>>> from app import page, posts
>>> print(page("index.aur", posts=posts))
<!DOCTYPE html>
<html>
    <head>
        <title>
            My Blog
        </title>
    </head>
    <body>
        <h1>Posts</h1>
        <div class="card">
            <h2>Hello, world</h2>
            <div class="card-body">
                <p>My first post.</p>
                <a href="/posts/1">Read more</a>
            </div>
        </div>
        <div class="card">
            <h2>Metaprogramming</h2>
            <div class="card-body">
                <p>It's an adventure.</p>
                <a href="/posts/2">Read more</a>
            </div>
        </div>
    </body>
</html>

And the code generated for a card is: 

>>> import auryn, pathlib
>>> gx = auryn.GX("""
... %load components
... !for post in posts:
...     %Card title=post.title
...         <p>{post.content}</p>
... """, root=pathlib.Path("pages"))
>>> print(gx.generate().to_string())
for post in posts:
    with props(title=post.title):
        emit(0, '<div class="card">')
        emit(4, '<h2>', title, '</h2>')
        emit(4, '<div class="card-body">')
        emit(8, '<p>', post.content, '</p>')
        emit(4, '</div>')
        emit(0, '</div>')

Like you can see, the component's lines are inlined under with props(title=post.title):, and the content <p> is emitted right where %content was, at its indentation. This is quite an interesting transformation, if you think about it, since there are no component objects at execution time: components are compiled away, in a way that's reminiscent of how Svelte compiles its components into plain code, rather than managing a virtual DOM.

These are, of course, server-side components, rendered once per request; the real thing happens in the browser, and it's coming up next.

A React Plugin

Now, say our posts have comments: those should load after the page does — so it's really a job for React. But there are two chores that make working with it together with Python somewhat awkward: creating a JavaScript client for our Python API (which is why Swagger Codegen and OpenAPI generators exist — people got tired of writing fetch wrappers by hand); and preparing a build step for our JSX. Luckily, both these problems are really code generation problems. Let's see if we can do it differently. 

# app.py:
comments = {
    1: [{"id": 1, "text": "Welcome!"}],
    2: [{"id": 2, "text": "So meta."}, {"id": 3, "text": "Wait, what?"}],
}

# … same as before

def page(template, **context):
    return auryn.execute(f"pages/{template}", app=app, **context)
    
# … same as before

@app.get("/api/posts/{post_id}/comments")
def get_comments(post_id: int):
    return comments[post_id]

That's the new endpoint, and the updated page function. Let's go over it: get_comments returns a list, so the framework will serve it as JSON; and page now passes in the app itself to the template context. The client, then, is auto-generated from the routes themselves — their handler names, their parameters (using inspect.signature, like in dependency injection), and their paths. It's a template, but it's full of JavaScript braces and ${...} template literals, so it's exactly what our custom delimiters are for: 

# pages/api.aur:
%interpolate <% %>
const api = {
    !for name, params, path in routes(app):
        <% name %>: async (<% ", ".join(params) %>) => {
            const response = await fetch(`<% path %>`);
            return await response.json();
        },
};

And then, a new hook: 

# plugins/react.py:
def x_routes(gx, app):
    for path, handler in app.routes.items():
        if path.startswith("/api/"):
            params = [camel_case(name) for name in inspect.signature(handler).parameters]
            path = re.sub(r"\{(\w+)\}", lambda match: f"${{{camel_case(match[1])}}}", path)
            yield camel_case(handler.__name__), params, path
            
def camel_case(name):
    first, *rest = name.split("_")
    return first + "".join(word.title() for word in rest)

Notice the phase: routes is a hook, so it's called while the page is executing, and it only depends on the app passed to it (and the same api.aur would describe any app built with our framework). Let's break it down: for every /api/ route, we want to yield its handler name in camel case, its parameters, and its path as a JavaScript template literal ({post_id} becomes ${postId}). The result is: 

>>> import auryn
>>> from app import app, page, posts
>>> print(auryn.execute("""
... %load react
... %include pages/api.aur
... """, app=app))
const api = {
    getComments: async (postId) => {
        const response = await fetch(`/api/posts/${postId}/comments`);
        return await response.json();
    },
};

And you know what — it can't drift from the server, because it's auto-generated from it; and it works (I checked it under node, with a fake fetch).

And now, the second chore: compiling JSX. To do that in the browser, Babel provides a standalone build that can transpile <script type="text/babel"> into JavaScript (this is actually how the web notebook's React frontend was implemented in the chapter on dynamic code execution). Consider: 

// pages/components/Comments.jsx:
function Comments({ api, postId }) {
    const [comments, setComments] = React.useState(null);
    React.useEffect(() => {
        api.getComments(postId).then(setComments);
    }, [postId]);
    if (comments === null) {
        return <p>Loading comments...</p>;
    }
    return (
        <ul>
            {comments.map((comment) => <li key={comment.id}>{comment.text}</li>)}
        </ul>
    );
}

And then, the React macros: 

# plugins/react.py (continued):
SCRIPTS = [
    "https://unpkg.com/react@18.3.1/umd/react.development.js",
    "https://unpkg.com/react-dom@18.3.1/umd/react-dom.development.js",
    "https://unpkg.com/@babel/standalone@7.26.4/babel.min.js",
]

def g_react_setup(gx):
    for url in SCRIPTS:
        gx.add_text(gx.line.indent, f'<script src="{url}"></script>', interpolate=False)
        
def g_react(gx, arg):
    name, _, props = arg.partition(" ")
    source = (gx.root / "components" / f"{name}.jsx").read_text()
    indent = gx.line.indent
    gx.add_text(indent, f'<div id="{name}"></div>', interpolate=False)
    gx.add_text(indent, '<script type="text/babel">', interpolate=False)
    for line in source.strip().splitlines():
        gx.add_text(indent + 4 if line else 0, line, interpolate=False)
    gx.add_code(f"emit({indent + 4}, 'const props = ', to_json({props}), ';')")
    root = f'ReactDOM.createRoot(document.getElementById("{name}"))'
    gx.add_text(indent + 4, f"{root}.render(<{name} api={{api}} {{...props}} />);", interpolate=False)
    gx.add_text(indent, "</script>", interpolate=False)
    
def x_to_json(gx, **props):
    return json.dumps(props)

Let's go over it: %react_setup emits the necessary script tags to load React, ReactDOM and Babel, with pinned versions. Then, %react Name props reads components/Name.jsx during generation and inlines it, as raw as it gets (since its braces are actually JSX, not interpolation). Next, we emit const props = ...;, serialized into JSON during execution by a to_json hook, so post.id is actually replaced with the value. And finally, we mount it with ReactDOM.createRoot(...).render(<Name api={api} {...props} />). Our post template is now: 

# pages/post.aur:
%load react
%extend base.aur
%define head
    %react_setup
    <script>
        %include api.aur
    </script>
%define title
    {post.title}
%define body
    <h1>{post.title}</h1>
    <p>{post.content}</p>
    %react Comments postId=post.id

And the result: 

>>> print(page("post.aur", post=posts[1]))
<!DOCTYPE html>
<html>
    <head>
        <script src="https://unpkg.com/react@18.3.1/umd/react.development.js"></script>
        <script src="https://unpkg.com/react-dom@18.3.1/umd/react-dom.development.js"></script>
        <script src="https://unpkg.com/@babel/standalone@7.26.4/babel.min.js"></script>
        <script>
            const api = {
                getComments: async (postId) => {
                    const response = await fetch(`/api/posts/${postId}/comments`);
                    return await response.json();
                },
            };
        </script>
        <title>
            Metaprogramming
        </title>
    </head>
    <body>
        <h1>Metaprogramming</h1>
        <p>It's an adventure.</p>
        <div id="Comments"></div>
        <script type="text/babel">
            function Comments({ api, postId }) {
                const [comments, setComments] = React.useState(null);
                React.useEffect(() => {
                    api.getComments(postId).then(setComments);
                }, [postId]);
                if (comments === null) {
                    return <p>Loading comments...</p>;
                }
                return (
                    <ul>
                        {comments.map((comment) => <li key={comment.id}>{comment.text}</li>)}
                    </ul>
                );
            }
            const props = {"postId": 2};
            ReactDOM.createRoot(document.getElementById("Comments")).render(<Comments api={api} {...props} />);
        </script>
    </body>
</html>

Take a moment to appreciate it: not only does Python data end up as JSON props in React, but the client itself is loaded into the head, and once the page makes it to the browser, the component actually fetches /api/posts/2/comments on its own! Admittedly, this isn't a framework (there's no shared state between the server and the client, no hydration) — but it's the shape of one: a Python server whose templates know about its routes, about its components, and can integrate the two together.

Pre-Generation

Let's get back to the problem we noticed before. Even with our nice plugin, every request still parses the template, loads the plugins (by executing their source code), runs the macros (which include reading Comments.jsx from disk), and finally compiles the result, all before executing a single emit. That was the trade-off we mentioned in our survey of templating engines: Django interprets its template on every render, whereas Jinja and Mako compile it to Python once, and render it multiple times. First, let's time it: 

>>> import timeit
>>> from app import app, page, posts
>>> timeit.timeit(lambda: page("post.aur", post=posts[1]), number=1000)
1.2749013747088611

In our case, since we've already separated the phases into different methods, this is easy to remedy: 

>>> import auryn
>>> gx = auryn.GX("pages/post.aur").generate()
>>> code = compile(gx.to_string(), "post.aur", "exec")
>>> def render(**context):
...     gx.output = []
...     exec(code, {**gx.x_globals, **context})
...     return "".join(gx.output).rstrip("\n")
>>> render(app=app, post=posts[1]) == page("post.aur", post=posts[1])
True
>>> timeit.timeit(lambda: render(app=app, post=posts[1]), number=1000)
0.021650417242199183

Let's walk through it: first, we generate the code once; then, we compile it into a code object once; and finally, for every request, we reset the output and exec the same code object in a fresh namespace, which is made up of the execution globals and the context of that request (so it's indeed fresh, and doesn't bleed over from one request to another). The output is exactly the same — but now, a thousand renders drop from about 1.3 seconds to about 0.02 seconds on my machine, which is some 60 times faster (your numbers may vary).

All that's left is a straight-line Python calling emit, so the bulk of the templating cost was really the template language, not its output. We might call this pre-generation, in the spirit of ahead-of-time compilation; and if you squint, the cache of generated code is our .pyc file, which is rather fitting for a book that started out as an attempt to understand what Python does with our code behind our backs.

But where do we actually put this cache? If we keep it in memory, a dictionary from template to code object will work fine — at least, as long as we're dealing with a single process. On disk, though, it gets interesting; you see, the generated code is not self-sufficient, because it calls emit, indent, props, to_json and routes — all of which are hooks bound to a live GX, which won't exist in the process that loads the file. Hand it to a bare exec, and it'll just die with a NameError; for generated code to stand on its own, then, it has to take its dependencies with it.

Standalone Code

Our goal is to generate code on one machine today, and execute it tomorrow on a machine that has never even heard of Auryn. We've already seen, in the runtime serialization chapter, that serialization of behavior is possible — and in fact, concluded that replaying recorded steps is superior to pickling frames. Here, instead of shipping the entire engine, all its plugins and the template — we only need to ship plain Python text, that we'll be able to execute with nothing more than a standard Python installation.

What does generated code need in order to run? emit, indent and the output list, which is the execution half of the GX, as well as whatever hooks it uses, and whatever imports and helpers those hooks use. So let's start with a small refactoring, splitting GX into a Runtime class (which holds the output, emit, indent and patch) and GX(Runtime) for generation. Then, the engine can ship the part of itself that's needed for execution, as source — simply by using inspect.getsource on it. As for figuring out which names the generated code uses — that's something we've been doing since the tracer and bytecode chapters: simply compiling it and looking at co_names, recursively so, since those might include nested code objects in co_consts (function bodies, comprehensions) — and ignoring the builtins: 

# auryn.py:
def global_names(code):
    if isinstance(code, str):
        code = compile(code, "<code>", "exec")
    names = set(code.co_names)
    for const in code.co_consts:
        if isinstance(const, types.CodeType):
            names |= global_names(const)
    return names - set(vars(builtins))
    
def collect_definitions(paths):
    definitions = {}
    for path in paths:
        for node in ast.parse(pathlib.Path(path).read_text()).body:
            if isinstance(node, (ast.Import, ast.ImportFrom)):
                for alias in node.names:
                    definitions[alias.asname or alias.name] = ast.unparse(node)
            elif isinstance(node, (ast.FunctionDef, ast.ClassDef)):
                if node.name.startswith("x_"):
                    node.name = node.name.removeprefix("x_")
                    node.args.args = node.args.args[1:]  # gx becomes a global
                definitions[node.name] = ast.unparse(node)
    return definitions

Let's walk through it: collect_definitions parses each plugin module with ast and maps every top-level import, function and class to its source, using ast.unparse. On the way, it rewrites the hooks, turning x_file to file and removing its first parameter, since in standalone code, gx is a module-level global and not an argument. It's the same kind of AST rewriting we've seen in the source code manipulation chapter. And then, to_standalone: 

# auryn.py:
class GX:

    def to_standalone(self):
        code = self.to_string()
        paths = set()
        for name in global_names(code):
            hook = self.x_globals.get(name)
            if inspect.ismethod(hook) and hook.__self__ is self and name not in ("emit", "indent"):
                paths.add(inspect.unwrap(hook.__func__).__code__.co_filename)
        definitions = collect_definitions(paths)
        needed, queue = [], list(global_names(code))
        while queue:
            name = queue.pop(0)
            if name in definitions and definitions[name] not in needed:
                needed.append(definitions[name])
                queue.extend(global_names(definitions[name]))
        imports = ["import contextlib", "import json", "import sys"]
        imports += [source for source in needed if source.startswith(("import", "from"))]
        others = [source for source in needed if source not in imports]
        return "\n\n".join([
            "\n".join(dict.fromkeys(imports)),
            inspect.getsource(Runtime).strip(),
            "gx = Runtime(globals())\nemit, indent = gx.emit, gx.indent",
            *others,
            "globals().update(json.loads(sys.argv[1]) if len(sys.argv) > 1 else {})",
            code,
            'print("".join(gx.output), end="")',
        ]) + "\n"

It finds the plugin files of the hooks used by the code (and note how inspect.unwrap peels off contextmanager's wrapper, thanks to the __wrapped__ attribute, to reveal the original function and its co_filename); then, collects their definitions; and finally, uses a worklist: starting from the names used by the code, it adds their definitions, and queues their names in turn — pretty much like a bundler's tree shaking algorithm would do.

Then, the assembly: imports first, so we have decorators like @contextlib.contextmanager when they're needed; then, the Runtime's source, gx = Runtime(globals()), the hooks, the context (which in standalone code we need to fill in any free names, like n, by updating the module's globals from a JSON argument we pass on the command line — the equivalent of passing the context to execute), and then the code and the printing of the output. Note that Runtime also gets the module's globals() as its execution namespace, so hooks like props continue to work. Here are the names used by our scaffold's generated code: 

>>> sorted(auryn.global_names(gx.to_string()))
['directory', 'emit', 'file', 'i', 'n', 'name', 's', 'title']
>>> definitions = auryn.collect_definitions([gx.x_globals["file"].__func__.__wrapped__.__code__.co_filename])
>>> print(definitions["file"])
@contextlib.contextmanager
def file(path):
    output = []
    with gx.patch(output=output, output_indent=0):
        yield
    pathlib.Path(path).write_text(''.join(output))

Notice the title: since co_names keeps attribute names as well (name.title()), it's a bit of an over-approximation, but that's OK, since only names that have definitions are going to be pulled in anyway. Here's how we'd generate the <ul> template as a standalone file: 

>>> import auryn
>>> gx = auryn.GX("""
... <ul>
...     !for i in range(n):
...         <li>{i}</li>
... </ul>
... """).generate()
>>> open("out/loop.py", "w").write(gx.to_standalone())
1177

And here's the standalone loop.py: 

# loop.py:
import contextlib
import json
import sys

class Runtime:

    def __init__(self, x_globals=None):
        self.x_globals = {} if x_globals is None else x_globals
        self.output = []
        self.output_indent = 0
        
    def emit(self, indent, *args):
        self.output.append(" " * (self.output_indent + indent) + "".join(map(str, args)) + "\n")
        
    @contextlib.contextmanager
    def indent(self, indent):
        self.output_indent += indent
        try:
            yield
        finally:
            self.output_indent -= indent
            
    @contextlib.contextmanager
    def patch(self, **attributes):
        previous = {name: getattr(self, name) for name in attributes}
        for name, value in attributes.items():
            setattr(self, name, value)
        try:
            yield
        finally:
            for name, value in previous.items():
                setattr(self, name, value)
                
gx = Runtime(globals())
emit, indent = gx.emit, gx.indent

globals().update(json.loads(sys.argv[1]) if len(sys.argv) > 1 else {})

emit(0, '<ul>')
with indent(4):
    for i in range(n):
        emit(0, '<li>', i, '</li>')
emit(0, '</ul>')

print("".join(gx.output), end="")

And we can run it with plain Python: 

$ python loop.py '{"n": 3}'
<ul>
    <li>0</li>
    <li>1</li>
    <li>2</li>
</ul>

Similarly, we can generate the project scaffold as standalone code: 

>>> gx = auryn.GX("""
... %load filesystem
... {name}/
...     README.md
...         # {name.title()}
...     tests/
...         !for i in range(n):
...             test_{i}.py
...                 def test_{i}():
...                     assert True
... """).generate()
>>> open("out/scaffold.py", "w").write(gx.to_standalone())
1814

And run it, then list the tree it generates: 

$ python scaffold.py '{"name": "proj", "n": 2}'
$ find proj | sort
proj
proj/README.md
proj/tests
proj/tests/test_0.py
proj/tests/test_1.py
$ cat proj/tests/test_1.py
def test_1():
    assert True

Notice how only directory, file and s came along, and not shell or the path transform — nothing else was needed. We have actually generated a project generator that requires nothing but Python itself! In the real Auryn, this standalone mode does the same collection, but also stores comments that map every generated line back to its template, so that errors can still be traced to the right line (it's restored with execute_standalone).

Rendering This Book

Finally, I have a confession to make. You've been reading a bunch of templates all along. Every chapter of this book is a .vyn file, rendered by vynil, a book renderer built on top of Auryn. All the %title, %section and %code are actually macros; the plain text lines are paragraphs; and any components, like tables, columns or images, are templates with %content.

Let's write a small version of it, then: the paragraphs plugin from the end of the previous chapter, all grown up: 

# plugins/book.py
import html
import re

STYLES = [
    (re.compile(r"`(.+?)`"), r"<code>\1</code>"),
    (re.compile(r"\*\*(.+?)\*\*"), r"<strong>\1</strong>"),
    (re.compile(r"\*(.+?)\*"), r"<em>\1</em>"),
]


def on_load(gx):
    gx.transforms[""] = transform_paragraph
    gx.on_complete(close_paragraph)
    
    
def transform_paragraph(gx, content):
    if not content:
        return close_paragraph(gx)
    if not gx.state.get("paragraph"):
        gx.add_text(0, "<p>", interpolate=False)
        gx.state["paragraph"] = True
    for pattern, replacement in STYLES:
        content = pattern.sub(replacement, content)
    gx.add_text(4, content, interpolate=False)
    
    
def close_paragraph(gx):
    if gx.state.get("paragraph"):
        gx.add_text(0, "</p>", interpolate=False)
        gx.state["paragraph"] = False
        
        
def g_title(gx, title):
    close_paragraph(gx)
    gx.add_text(0, f"<h1>{title}</h1>", interpolate=False)
    
    
def g_section(gx, title):
    close_paragraph(gx)
    slug = re.sub(r"\W+", "-", title.lower()).strip("-")
    gx.add_text(0, f'<h2 id="{slug}">{title}</h2>', interpolate=False)
    
    
def g_code(gx, language="python"):
    close_paragraph(gx)
    code = html.escape(gx.line.children.snap(0).to_string().strip())
    gx.add_text(0, f'<pre class="{language}"><code>{code}</code></pre>', interpolate=False)

Let's go over it together: the on_load function replaces the default transform with our own transform_paragraph and registers close_paragraph to run after generation is complete. The transform_paragraph function takes care of consecutive text lines, gluing them into a single paragraph (which we track with a flag in the state) and closing it with an empty line (which is why we kept those in the parser). As for inline styles, those are nothing more than regular expression substitutions: backticks for code, double asterisks for strong, and single asterisks for emphasis.

All the macros start by closing any open paragraph (that's why we needed the state flag), and then do something. %section creates an anchor out of its title, and %code treats its children as raw text (snap(0) followed by to_string), escaping its HTML and emitting it with no interpolation (since code samples are full of braces). If we look at the beginning of the source code of the chapter about the 20-line templating engine: 

# 11-a-templating-engine-in-20-lines.vyn:
%title A Templating Engine in 20 Lines

At this point, we've covered aspect-oriented programming, domain-specific
languages and user-defined code generation; we've manipulated source code as
plain text, and serialized the very act of execution. We're ready, at last, to
embark on the final leg of our journey: a metaprogramming framework, combining
all of these ideas into a particularly fun abstraction. Before we do, though,
I'd like to tell you the story of its conception – because the idea behind it
is quite simple, and literally fits in just a few lines of code.

In the book *The Architecture of Open-Source Applications: 500 Lines or Less*,
experienced programmers solve interesting problems from real-life projects. It's
an excellent read, full of practical insights into software design – and in one
of its chapters, Ned Batchelder describes the motivation, challenges and
solutions of writing a templating engine.

Here's the gist: most programs contain a lot of code, and only a little text, so
programming languages are designed for code — text is simply enclosed in quotes
and managed in strings:

%code
    >>> def hello():
    ...     print("Hello, world!")

And if we render it: 

>>> import auryn
>>> gx = auryn.GX("excerpt.vyn")
>>> gx.load("book")
>>> html = gx.generate().execute()
>>> print("\n".join(html.splitlines()[:16]))
<h1>A Templating Engine in 20 Lines</h1>
<p>
    At this point, we've covered aspect-oriented programming, domain-specific
    languages and user-defined code generation; we've manipulated source code as
    plain text, and serialized the very act of execution. We're ready, at last, to
    embark on the final leg of our journey: a metaprogramming framework, combining
    all of these ideas into a particularly fun abstraction. Before we do, though,
    I'd like to tell you the story of its conception – because the idea behind it
    is quite simple, and literally fits in just a few lines of code.
</p>
<p>
    In the book <em>The Architecture of Open-Source Applications: 500 Lines or Less</em>,
    experienced programmers solve interesting problems from real-life projects. It's
    an excellent read, full of practical insights into software design – and in one
    of its chapters, Ned Batchelder describes the motivation, challenges and
    solutions of writing a templating engine.

And the rendered code block: 

>>> start = html.index("<pre")
>>> print(html[start : html.index("</pre>") + len("</pre>")])
<pre class="python"><code>&gt;&gt;&gt; def hello():
...     print(&quot;Hello, world!&quot;)</code></pre>

Notice the escaped >>> and the quotes; and how the text from an earlier chapter was rendered into paragraphs. The same thing happens with vynil's real chapter plugin, which also does a little trick to prevent inline styles from being applied to code spans, and uses Pygments to highlight code and support markers for highlighting regions and breaking pages.

As a writer, this is exactly what I was looking for: the old promise of templating, to keep the source close to plain text, readable in any editor and easy to diff, while still generating nice typeset HTML and PDF. So the tool I wrote to write about metaprogramming is a metaprogram itself — and the HTML and PDF versions of these chapters are actually generated code's output. Whenever I needed something new — a table, two columns, a page break — I was able to do it myself, adding a component or a macro, not a feature request.

This is what I meant in the chapter about the 20-line templating engine, when I talked about convivial tools: tools that can be extended by their users, from within and in the same language they're already using, so as to foster their autonomy and creativity, not just their efficiency.

The End of the Adventure

And that's a wrap — on this chapter, and on this book. In two chapters, we grew a 20-line proof of concept to a mini-Auryn — trees of lines, a GX that defers indentation to execution, our own interpolation, macros, composition, plugins and hooks, errors that point at templates, and then, in a spectacular flourish, pointed it at directories and shells, web pages, components and React, caching, standalone code, and this very book. The real thing is just a pip install auryn away (it's pure Python, with no dependencies), and it's all there, with the same names — so it should feel pretty familiar: 

>>> import auryn
>>> template = """
...     <ul>
...         !for i in range(n):
...             <li>{i}</li>
...     </ul>
... """
>>> print(auryn.generate(template))
emit(0, '<ul>')
with indent(4):
    for i in range(n):
        emit(0, '<li>', i, '</li>')
emit(0, '</ul>')
>>> print(auryn.execute(template, n=3))
<ul>
    <li>0</li>
    <li>1</li>
    <li>2</li>
</ul>

Except, maybe, it looks a little too familiar; character for character, it's exactly what our mini-engine produced. The real Auryn adds all the things that make a library production-worthy: origins and full error reports, generation-time code lines, %raw, %inline and %strip, %bookmark and %append, %param, standalone code with source maps, and even a CLI. If you go look at its README, you'll also find a guide for writing your own plugins, which ends with an invitation that still stands: let me know what fun ideas you come up with.

Looking back, we've seen almost every chapter in the last two: frames were walked to find the caller's directory, as we did in dynamic code execution; co_names were read to find hooks, as in tracers and bytecode; exec was used to load plugins and run macros, as in dynamic extensions; naming conventions and decorators were employed for registration, like in aspect-oriented programming; tracebacks were enriched, like when we were parsing tracebacks; the ast module was used to rewrite hooks, like in source code manipulation; and code was shipped instead of state, like in runtime serialization.

If there's a moral to this adventure, I suppose it's that metaprogramming earns its keep when it disappears behind an honest interface; the person writing %Card title=post.title needn't know anything about emit or props. And unlike black-box generators like ProtoBuf, or an AI tool that we can't question — here, the code generated is short, and readable, and always a print(generate(...)) away. This, I think, is what makes the difference between convivial tools and opaque ones: that we can read the generated code, and reason about the generator; it's what keeps us fluent in the craft, and fluency in code is what lets us reason about what the job even is.

Anyway, this has been the adventure. I don't know about you, but I had fun; like I said in the introduction, it was never meant to be a theory of metaprogramming — just a hands-on exploration of uncharted territory, for the sake of exploration, to see what dragons be there; and when they could be tamed, all the better. And remember that Saint-Exupéry quote: "If you want to build a ship, don't drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea." I didn't want to build a ship; if any of that yearning stuck, if you feel that joy of making code do what it wasn't supposed to, just for the fun of it — then this book did its job.

So thanks for reading, and go play!