Introduction

Metaprogramming is the practice of writing code that operates on other code: programs that can inspect other programs, manipulate and modify them, even generate them from scratch. It's incredibly powerful, but somewhat underrated; incredibly sophisticated, but somewhat understudied; and most importantly, at least for this book – it's incredibly fun.

So why isn't it more popular? Why isn't there a single university with a course on metaprogramming, or a single company with a metaprogrammer job listing? There aren't even that many books on the subject – and what books there are all have some programming language in their title, like "C++ Metaprogramming" or "Metaprogramming in Ruby". Great though they may be, they're not really about the concept itself – rather, they teach advanced programming techniques, which happen to utilize metaprogramming ideas like templating or introspection.

Maybe it's because metaprogramming isn't a coherent paradigm: unlike functional or object-oriented programming, there isn't a body of theory articulating its principles, and not a programming language that implements them. But this just pushes the question further, doesn't it – I mean, why not? Concepts like monads or inheritance didn't exist before we invented them, either; so why can't we do the same here?

Maybe metaprogramming is too broad: when we have code that operates on other code, it begs the question of what other code it operates on: Python? Java? Haskell? And then, once we box it in an existing category, it just becomes a set of "advanced programming techniques" in that language, like we mentioned before. But that's the case for compilers too, isn't it? gcc only works for C, and rustc only works for Rust; yet there are compilation courses and compilation experts – so what gives?

Maybe it's too complex: mastering one programming paradigm well is difficult enough, so reasoning a level of abstraction above it is too mind-bending to carry out any useful work. But that's not the case, either: first of all, because metaprogramming is used – it's just not systematized; and second, because when did this ever stop us? High-level languages are also more abstract than low-level ones, and yet the industry has largely transitioned from Assembly to Javascript. So what's stopping us now?

Honestly, I don't know. I realize it's a bit of a frustrating conclusion, what with all this buildup – it really sounded like these questions were leading to some revelation, didn't it? But I think it's best to be open about it early on: I'm not a scholar who will lay out such a theory.

That said, I really like metaprogramming – I've been fascinated with this sorcery from the first time I saw it, and I've been playing with it for many years since. "Playing" is the key word here, because while I've written a lot of useful code that employed it, the moment it became "work", it shifted into the aforementioned realm of "advanced programming techniques". As for the "concept itself" – the abstractions and possibilities this way of thinking entails – I genuinely think it shouldn't be learned or taught for any serious purpose. Arguably, it can't be: when we are focused on achieving a concrete goal, we tend to narrow our field of vision and push forward to make it happen. It's only when we play that we can let our imagination run wild, experiment with new ideas, try new things – and then, usually through some lucky accident, stumble unto innovation. Sure, we can contemplate it afterwards, refine and adapt it into something more practical, effectively coming up with an "advanced programming technique" of our own. But such novelty is almost always preceded by a moment of uninhibited, irreverent joy; and maybe I'm getting old and bitter, but I feel like those moments are less and less common. They don't teach them in academia, nor practice them in industry; so maybe the reason we don't have a metaprogramming discipline is simply because we don't let ourselves have it. A failure of imagination, if you will – it seems to be a prevalent trend in many fields of our increasingly commercialized and alienated world.

That joy, then, is what I'd like to share; my contribution, such as it is, aims for that sense of wonder. There's a quote by Antoine de Saint-Exupéry that strikes a similar chord: "If you want to build a ship, don't drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea." Only I don't want to build a ship, nor presume I have anything substantial to teach you; like I said, I'm not a scholar, and this book is not a theory of metaprogramming, but rather an adventure in it – take from it what you will. However, I do believe that cultivating passion is the surest (not to mention, healthiest) way towards professional (and other) excellence – I find it's much more sustainable and profound when it's a byproduct of love, not grit or toil or this or that syllabus.

With that in mind, we're going to jump between a lot of different and somewhat unrelated topics, which I tried to weave into a narrative that will make sense, but also be fun. The bulk of the code is in Python, so maybe it wasn't entirely fair of me to criticize others for focusing on a particular language. In my defense, we'll also touch on Javascript, Bash and C – and anyway, what I aim to illustrate is more so a way of thinking than a way of doing. Python, with all its reflective and dynamic capabilities, simply happens to fit it particularly well; combined with its elegance and popularity, it seemed to me like the perfect scaffolding for building our cloud castles. That being said, this book is not a tutorial on advanced Python features – it assumes you're familiar enough with the language to follow along, or at least get the gist of what we're doing so as to apply it in your own context.

In fact, let me show you what I mean in a nutshell. If you've ever used Python, you probably encountered metaprogramming already, whether you thought about it in these terms or not. Even something as simple as:

>>> from dataclasses import dataclass

>>> @dataclass
... class A:
...     n: int
...     s: str

Is actually code that operates on other code: the dataclass decorator receives our class, inspects its annotations, and augments its code to auto-generate such standard considerations as its constructor and representation string, without us having to define any of it explicitly:

>>> a = A(n=1, s="hello world")
>>> a
A(n=1, s="hello world")

The thing is, doing so isn't even all that hard. If we were to write such a decorator ourselves, we could iterate over the annotations, collect their names, and format them into boilerplate __init__ and __repr__ code, which we would then evaluate and attach to the class dynamically:

>>> def dataclass(cls):
...     names = list(cls.__annotations__)
...     init = f"def __init__(self, {', '.join(names)}):\n    " + \
...            "\n    ".join(f"self.{name} = {name}" for name in names)
...     args = ", ".join(f"{name}={{self.{name}!r}}" for name in names)
...     repr = f"def __repr__(self):\n    return f'{cls.__name__}({args})'"
...     namespace = {}
...     exec(init, namespace)
...     exec(repr, namespace)
...     cls.__init__ = namespace["__init__"]
...     cls.__repr__ = namespace["__repr__"]
...     return cls

For A with annotations {"n": int, "s": str}, this will generate the code:

def __init__(self, n, s):
    self.n = n
    self.s = s
    
def __repr__(self):
    return f'A(n={self.n!r}, s={self.s!r})'

Which is pretty much what we'd write if we were doing it manually – except we'd have to do so for every attribute of every class, and here we solved the problem once and for all in some 10 lines. The catch is that these lines were somewhat more sophisticated than your garden variety source code, seeing as they had to do some source code gardening themselves.

Does this mean metaprogramming is really about productivity? – letting us create utilities that automate coding so that we don't have to do as much of it on our own? In some sense, yeah, maybe; but it's also much more than that. The popular Pydantic library, for example, solves a similar problem:

>>> from pydantic import BaseModel

>>> class A(BaseModel):
...     n: int
...     s: str

But also adds type safety checks, so if we try to assign anything other than a number to n, or anything other than a string to s, we get an error; and it gives us a bunch of other useful features, too, like dumping our instance state into a dictionary or a JSON.

Still, you might argue, this just saves us the hassle of writing validation or serialization code. And in a way, you'd be right: for any Turing-complete language, all that third-party libraries (or layers of abstraction in general) do is save us from implementing something we could otherwise implement ourselves. The difference lies in how they do it, which has a subtle but very real effect on our outlook on software development.

Consider serialization: if we were to implement it ourselves, we'd probably use the json library, right? It, too, saves us from having to worry about the details of the JSON format – but there's nothing particularly interesting about it, in that one can easily conceive of how such an implementation might look. Metaprogramming solutions like Pydantic's, on the other hand, can raise an eyebrow in curiosity – at least, if we approach them with the right mindset. I mean, it doesn't use a decorator to intercept our class object – so how does it do such dynamic customizations with plain inheritance?

This, in turn, might prompt us to look at its source code and discover it does so with a metaclass: that is, a class charged with instantiating other classes, since those are also objects in Python. And just like a regular class can define an __init__ method that runs when its objects are created – so can a metaclass define one to run when its classes are created, giving us the same opportunity as a decorator would have to inspect and augment our target.

But this realization – our initiation into metaclasses, if you will – might have a ripple effect; again, assuming a curious, exploratory attitude. If metaclasses can override class creation, what other behaviors can they override? Like, if a class implements __getitem__, its objects can be used as obj[...], right? So if a metaclass implements it, can we similarly do cls[...]?

The answer is yes; in fact, that's exactly what Python's core developers did when they introduced typing, and needed to come up with some syntax to define generics. If Queue contains items of a certain type, how do we tell apart a queue of integers from a queue of strings? The answer: Queue[int] and Queue[str], both of which use a class's __getitem__ to return a compound annotation, which tools like Mypy use to ascertain each queue item is treated according to its expected type.

Or, take a different scenario: suppose we want to keep track of all the instances of a given class, and be able to iterate over them at will. We can do so quite elegantly if we consider that object creation actually happens by invoking their class, so it's in fact implemented by the metaclass's __call__. If we override it, then, we can intercept and collect all the instances (and here we better use weak references, to not hinder garbage collection); to iterate over them, we similarly implement __iter__, which lets us stick the class itself in a for-loop:

>>> import weakref

>>> class M(type):
...     def __init__(cls, name, bases, attrs):
...         cls._instances = weakref.WeakSet()
...     def __call__(cls, *args, **kwargs):
...         instance = super().__call__(*args, **kwargs)
...         cls._instances.add(instance)
...         return instance
...     def __iter__(cls):
...         yield from cls._instances.copy()

Admittedly, this is a pretty sophisticated piece of code: it involves metaclasses, magic methods, weak references, sets, perfect forwarding, super(), yield from… But the result is exceedingly simple to use:

>>> @dataclass
... class A(metaclass=M):
...     n: int

>>> a1 = A(1)
>>> a2 = A(2)
>>> a3 = A(3)
>>> for a in A:
...     print(a)
A(n=1)
A(n=2)
A(n=3)

Now, there's something profoundly cool – or at least, very empowering – in knowing that we wield such a degree of control over the language. Again, one might argue that such "magic" is unnecessary, since we could achieve the same with:

>>> class A:
...     _instances = []
...     def __init__(self, n):
...         self.n = n
...         self._instances.append(self)
...     @classmethod
...     def get_all_instances(cls):
...         return cls._instances

>>> a1, a2, a3 = A(1), A(2), A(3)
>>> for a in A.get_all_instances():
...     print(a)

But it's not exactly the same, is it? – we have to reimplement it in every class we want to track, it keeps the objects alive in memory, it breaks if an object is created mid-loop... A generic solution would not only be reusable, but account for such considerations in a dedicated, cohesive fashion, instead of having them scattered among the unrelated logics of a particular class.

And besides, like I said, working at the meta level gives us such power over the language that it really does expand how we think – our horizon of possibilities, so to speak. In other circumstances, we might have a complicated inheritance tree, as often happens in large object-oriented projects: maybe we have a Settings class that should be savable either in JSON or in XML, but because those base classes don't play along, we have to do:

>>> class JSONSettings(Settings, JSONFile):
...     ...
>>> class XMLSettings(Settings, XMLFile):
...     ...

It seems like there are no two ways about it, since inheritance is "baked into" classes in a way that neither a metaclass nor a class decorator can help with. But what if we do:

>>> class Settings(Savable, metaclass=DynamicInheritance):
...     ...

Where Savable is just an abstract interface, to-be-resolved via a metaclass:

>>> class Savable:
...     def save(self, path):
...         raise NotImplementedError()

That metaclass can then implement __new__, which happens a moment before a class is created. This is effectively our only opportunity to modify the inheritance tree, but it's just enough: given some runtime state, we can decide which parents to mix into our classes in a dynamic, configurable way:

>>> settings_format = "json" # switch when necessary

>>> class DynamicInheritance(type):
...     def __new__(mcs, name, bases, attrs):
...         bases = list(bases)
...         if Savable in bases:
...             index = bases.index(Savable)
...             if settings_format == "json":
...                 bases[index] = JSONFile
...             if settings_format == "xml":
...                 bases[index] = XMLFile
...         return super().__new__(mcs, name, tuple(bases), attrs)

Now that's new; dynamic inheritance is not something they teach often, even in advanced object-oriented courses. And it can have far-reaching implications on our codebase: how clean it looks, how easy it is to extend. Of course, it has drawbacks, too – the "abnormality" of such a solution first and foremost among them, if other developers on our team can't follow along our weird machinations. Then again, job security 🤷

Anyway, that's not the point: whether you end up using any of the techniques we cover in a production environment is up to you. My goal – in this quick foray, as well as in the adventure that follows – is to explore such uncharted territories for exploration's sake, to see what dragons be there. And if those can be tamed, all the better – even if we still choose to plow our fields with a regular horse, it makes for a cool story, doesn't it?