Tracer Tricks

I'd like to start with something deliberately useless; bear with me, and I'll justify this decision soon enough. For now, the challenge we set out to solve will be the following: given a class that defines some attributes:

>>> class A:
...     x = 1
...     y = 2
...     z = 3

How can we determine their order of definition (as in: x, then y, then z)? In many cases it doesn't really matter — but in some, like database or serialization frameworks, it can be quite essential. The reason I say it's useless, then, is that since Python 3.8 (or even 3.6), the whole problem is moot: dictionaries are inherently ordered, and seeing as class attributes are stored in a dictionary, the order of their definition is invariably retained:

>>> [name for name in vars(A) if not name.startswith("_")]
['x', 'y', 'z']

Ordering Attributes: with Objects

So let's pretend this isn't the case — like, for some reason, we have to use Python 3.5. The way various libraries solved it back in the day was something along the lines of:

>>> class A(Model):
...     x = Field(1)
...     y = Field(2)
...     z = Field(3)

Where we introduce a base class, Model, wrap all the attributes in an auxiliary class, Field, and make them work in tandem to keep track of the order of definition:

>>> import itertools
>>> counter = itertools.count()

>>> class Field:
...     def __init__(self, value):
...         self.value = value
...         self.count = next(counter)

>>> class Model:
...     def __init_subclass__(cls):
...         fields = sorted((field.count, name) for name, field in
...                          vars(cls).items() if isinstance(field, Field))
...         cls.order = [name for count, name in fields]

That is, Field assigns each of its instances a unique, auto-incrementing integer (courtesy of itertool's infinite count generator); while Model collects the names of any such fields and stores them in its subclasses' order attribute, sorted by that number.

This way, even if we can't guarantee the order in which fields appear in each subclass dictionary, we don't need to. As long as fields are instantiated in order (which they are, since class bodies are evaluated sequentially), that information is preserved in each instance's count, which is used for sorting a moment later:

>>> A.order
['x', 'y', 'z']

This approach, however, is not all that elegant: it requires a lot of additional boilerplate to make things work, and one has to wonder whether the original, cleaner syntax of:

>>> class A:
...     x = 1
...     y = 2
...     z = 3

Can be similarly accommodated.

Ordering Attributes: with Metaclasses

As it turns out, it can be. Python 3 introduced the __prepare__ method for metaclasses, which is called before the bodies of their classes are evaluated, and should return the dictionary into which the results of such evaluation will go. Normally, it's a regular (and as such, we pretend, unordered) dictionary; but it's easy enough to swap with a decidedly ordered one, such as collections.OrderedDict. This way we can distill its sequence into a separate attribute once the class is created:

>>> import collections

>>> class OrderedMetaclass(type):
...
...     def __prepare__(mcs, name):
...         return collections.OrderedDict()
...
...     def __init__(cls, name, bases, attributes):
...         cls.order = [name for name in attributes # ← an OrderedDict
...                      if not name.startswith("_")]

All we need to do is make our original class derive from this metaclass:

>>> class A(metaclass=OrderedMetaclass):
...     x = 1
...     y = 2
...     z = 3

Or, more elegantly still, create a base class that derives from it:

>>> class Ordered(metaclass=OrderedMetaclass):
...     pass

So subclassing it is enough:

>>> class A(Ordered):
...     x = 1
...     y = 2
...     z = 3

>>> A.order
['x', 'y', 'z']

Tracing: an Interlude

But let's keep pushing: suppose we need to achieve the same in that archaic language, Python 2.7, that some people thought would last forever. Back then, metaclasses didn't have a __prepare__ method, and ordered dictionaries were all but a distant dream. Still, we can do it as simply as:

>>> @ordered()
... class A:
...     x = 1
...     y = 2
...     z = 3

To do that, however, we'd have to resort to somewhat darker magic: the Python tracer. If you're not familiar with it, I don't blame you — it's mostly used for advanced and rather low-level considerations, like implementing debuggers and coverage tools. But, like everything else in Python, if we want it (and know how to use it), it's right there for the taking. All we have to do is call sys.settrace, passing in some function — and every line of code evaluated henceforth, so long as our tracer is installed, will invoke that function with three special arguments. It's a bit harder to demonstrate in interpreter mode, so let's put everything in a file like example.py:

import sys

def trace(*args):
    print(args)
    
sys.settrace(trace)

def f():
    x = 1
    y = 2
    return x + y
    
f()

And when we run it we should get:

$ python example.py
(<frame at 0x…, file '/home/user/example.py', line 8, code f>, 'call', None)

That's actually a bit underwhelming; it only caught line 8, namely f(), when in fact it's capable of stepping into functions and tracing their internal execution as well. To do that, though, our tracer ought to return another tracer, which will be used to track whatever happens inside a line. Since we'd like it to be exactly the same, nothing stops us from making our tracer return itself, which will keep it active indefinitely:

import sys

def trace(*args):
    print(args)
    return trace
    
# … same as before

And produce a more satisfactory result:

$ python example.py
(<frame at 0x…, file '.../example.py', line 9, code f>, 'call', None)
(<frame at 0x…, file '.../example.py', line 10, code f>, 'line', None)
(<frame at 0x…, file '.../example.py', line 11, code f>, 'line', None)
(<frame at 0x…, file '.../example.py', line 12, code f>, 'line', None)
(<frame at 0x…, file '.../example.py', line 12, code f>, 'return', 3)

If we'd have even deeper nesting (say, f would call g), we'd see it invoked recursively and relentlessly on every step the interpreter makes; hence its utility when it comes to debuggers or code coverage tools.

As for its three arguments, we can see that the first one is a frame object, which contains the scope information of each traced line; the second is a string indicating what type of event was traced — and incidentally, we've seen all the three options it can take: call when entering a function, return when leaving it, and line for anything else; and the third is an argument, used only for return events to denote the value returned.

Ordering Attributes: with Tracers

Interesting though it may be, it's not immediately clear how all this is related to our original problem, of keeping track of the attribute definition order; so let's see what our tracer can tell us in that case:

# … same as before

sys.settrace(trace)

class A:
    x = 1
    y = 2
    z = 3

$ python example.py
(<frame at 0x…, file '.../example.py', line 9, code A>, 'call', None)
(<frame at 0x…, file '.../example.py', line 9, code A>, 'line', None)
(<frame at 0x…, file '.../example.py', line 10, code A>, 'line', None)
(<frame at 0x…, file '.../example.py', line 11, code A>, 'line', None)
(<frame at 0x…, file '.../example.py', line 12, code A>, 'line', None)
(<frame at 0x…, file '.../example.py', line 12, code A>, 'return', None)

Looks good – all of the lines are traced; and curiously enough, the class declaration is considered an invocation as well, with a call event at its beginning and a return event at its end.

However, further trickery is in order: even now that we effectively get a callback on each line, parsing them to infer which attributes they define is easier said than done. So instead, we'll focus on the first trace only: namely, the call event at which the class definition begins.

Much like the call event of a regular function, the frame it receives already has a code object, compiled in advance and containing all the necessary information pertaining to the scope of its execution. If you're familiar with code objects, you know that part of this information has to do with the local, non-local and global names used by this scope; and in case of class bodies, the global ones happen to be (or rather, become) exactly those attributes, the order of which we aim to reconstruct. Luckily for us, the code object provides them in the order in which they are defined — which we can glimpse if we would print frame.f_code.co_names:

import sys

def trace(frame, event, arg):
    print(frame.f_code.co_names)
    
# … same as before

And notice how I removed the return statement, since we don't actually care to keep tracing inside the class definition – the outermost call event is enough:

$ python example.py
('__name__', '__module__', '__qualname__', 'x', 'y', 'z')

Now all that's left is to attach this order to our class object (and maybe filter out the standard attributes, added to all classes by default). To do so, we can make use of a decorator, albeit a tricky one:

>>> def ordered():
...     order = []
...     def record_order(frame, event, arg):
...         order.extend(name for name in frame.f_code.co_names
...                      if not name.startswith("_"))
...         sys.settrace(None)
...     sys.settrace(record_order)
...     def attach_order(cls):
...         cls.order = order
...         return cls
...     return attach_order

What it does is the following: starting with an empty order list, it defines the record_order function, sets it as the active tracer, and returns an altogether different decorator, attach_order. Normally, the next thing that would happen would be the returned function being applied to the class — but since the class definition must happen before that, however implicit this step usually is, and since we've installed a tracer that gets invoked on exactly such an event, record_order is actually called first. As its name implies, it collects the names — the attribute names, that is, and in their order of definition, as we've established — into our list; and since we only ever wanted to capture this particular event for our rather unusual purpose, it also retires itself from its role as the active tracer. This way, when attach_order is finally called, it can decorate the class with our recently (and one might add, cleverly) recorded order, as in:

>>> @ordered()
... class A:
...     x = 1
...     y = 2
...     z = 3

>>> A.order
['x', 'y', 'z']

Do notice that this is a second-order decorator, so we mustn't forget the parentheses in the decoration: A is not passed into ordered, as omitting them would imply. Instead, we invoke ordered() as part of the decoration itself, so it gets a chance to install the tracer before the class definition; A is then passed into whatever it returns, in our case attach_order.

C-Style Enums: with Tracers

This is pretty exciting; or at least, it would be, if dictionaries weren't already ordered, and the whole ordeal wouldn't be entirely pointless. So let's use our newfound knowledge to do something else, something important: add support for C-style enums!

In C, the way we define an enumeration of values is thus:

enum Fruit {
    APPLE,
    LEMON,
    ORANGE,
};

That is, we list the names, and their values are auto-assigned to an incrementing integer, so APPLE ends up as 0, LEMON as 1 and ORANGE as 2. In Python, however, we have to do this:

>>> import enum
>>> class Fruit(enum.Enum):
...     APPLE = 0
...     LEMON = 1
...     ORANGE = 2

That is, provide the values explicitly and manage them manually. There is a trick to spare us the manual management, at least:

>>> class Fruit(enum.Enum):
...     APPLE = enum.auto()
...     LEMON = enum.auto()
...     ORANGE = enum.auto()

But there is something irritating in having a statement that is more elegant in C than it is in Python, isn't there? Surely, we can do better:

>>> @enum()
... class Fruit:
...     APPLE
...     LEMON
...     ORANGE

>>> Fruit.APPLE
0
>>> Fruit.LEMON
1
>>> Fruit.ORANGE
2

At first sight, it might look impossible; like, is this syntax even valid in Python? But if we think about it for a moment, of course it is — class bodies are just code blocks, and what this code block does is resolve the name APPLE, discard its value, and then similarly resolve and discard LEMON and ORANGE. Normally it would throw a NameError, because these names weren't defined, so their resolution naturally fails:

>>> class Fruit:
...     APPLE
NameError: name 'APPLE' is not defined

But there's nothing inherently wrong with the syntax, so long as we find a way to define the names — like, running a piece of code that inspects them in advance and injects an incrementing integer for each before the scope is executed. And it so happens that we've just seen such a trick:

>>> def enum():
...     def inject_values(frame, event, arg):
...         names = [name for name in frame.f_code.co_names
...                  if not name.startswith("_")]
...         for i, name in enumerate(names):
...             frame.f_locals[name] = i
...         sys.settrace(None)
...     sys.settrace(inject_values)
...     def do_nothing(cls):
...         return cls
...     return do_nothing

This decorator has the same flow as ordered: it registers inject_values as a tracer, and returns a passthrough decorator, do_nothing, to be applied to the class. And just like before, since the class definition happens before its decoration, the tracer is called first. In this case, it simply enumerates the names, prefilling the frame's local scope accordingly, and unregisters itself. This way, by the time the class body is executed, APPLE is already 0, LEMON is 1 and ORANGE is 2; the odd statements that resolve those names and discard their values pass flawlessly without doing anything. But then, their actual purpose was to declare the names, so inject_values knows what to prefill – we never needed anything from their execution other than to not crash.

And now, just like we wanted:

>>> @enum()
... class Fruit:
...     APPLE
...     LEMON
...     ORANGE

>>> Fruit.APPLE
0
>>> Fruit.LEMON
1
>>> Fruit.ORANGE
2

C-Style Enums: with Metaclasses

Admittedly, if we deign to update our Python from 2.7 to 3+, I'd say that this approach is a bit too elaborate for our purpose, and suggest reverting back to the more standard __prepare__ method. In this case, instead of providing an ordered dictionary for the class body evaluation, we'd have to implement our own – say, EnumDict – to be tolerant of missing names and know to assign them on demand:

>>> class EnumMetaclass(type):
...     def __prepare__(mcs, name):
...         return EnumDict()

>>> class EnumDict(dict):
...     def __init__(self):
...         self.counter = itertools.count()
...     def __missing__(self, key):
...         if not key.startswith("_"):
...             self[key] = value = next(self.counter)
...             return value
...         raise KeyError(key)

>>> class Enum(metaclass=EnumMetaclass):
...     pass

This way, any class that inherits from Enum will have its body evaluated into an EnumDict; and any odd resolve-and-discard statements will end up calling its __missing__ method (since, to be fair, they are missing), which would make up for this oversight by quickly allocating them and pretending like nothing happened. The result is the same, but with much less voodoo:

>>> class Fruit(Enum):
...     APPLE
...     LEMON
...     ORANGE

>>> Fruit.APPLE
0
>>> Fruit.LEMON
1
>>> Fruit.ORANGE
2

The Linking of Scopes

So, is our tracing trick useful in any real scenario with a modern version of Python? Yes and no. First, I'll give such an example — although, admittedly, it's pretty niche – and then I'll circle back to the point from the beginning of the chapter, on why I don't think it matters all that much.

Suppose we have a configuration defined in a class; the following is common to a lot of web applications:

>>> class config:
...     port = 8000
...     log_path = "/app/log.txt"
...     log_maxsize = 20 * 1024 * 1024
...     log_backlog = 5
...     db_path = "/app/db.sqlite"

And suppose this configuration grows enough that we want to break it into sections; a natural way to do it would be by using nested classes:

>>> class config:
...     port = 8000
...     class db:
...         path = "/app/db.sqlite"
...     class log:
...         path = "/app/log.txt"
...         maxsize = 20 * 1024 * 1024
...         backlog = 5

So far, so good: we run a server on config.port, access the database at config.db.path, and write logs to config.log.path; but there's an unfortunate duplication here that might lead to trouble down the road. The more files we have (located, presumably, in the same app directory), the more paths starting with app/ we'll end up with — and if we ever move our deployment elsewhere and fail to update all of them properly, our results will be written to different places. A natural way to prevent this would be to declare the root as a separate attribute, and reuse it where necessary:

>>> class config:
...     port = 8000
...     root = pathlib.Path("/app")
...     class db:
...         path = root / "db.sqlite"
...     class log:
...         path = root / "log.txt"
...         maxsize = 20 * 1024 * 1024
...         backlog = 5

Except, this doesn't work: try running it and you'll get a NameError, complaining that root is not defined in any of the nested subclasses. This is a peculiar limitation of Python classes — the intuitive lexical scoping we're used to from functions doesn't quite work as expected, because when a class body is evaluated, its scope is managed in a bit of a different way. It can access the global scope in which the class is defined (so if we put root outside config, our code will work) — but it doesn't have a notion of intermediary scopes with non-local variables like functions do. In other words, the namespaces of nested classes are not stringed together with those of their nesting classes, and we simply can't organize our configuration in this seemingly innocuous way. That is, unless:

>>> def link_scopes():
...     def copy_parent_scope(frame, event, arg):
...         if event == "call" and "__name__" in frame.f_code.co_names:
...             frame.f_locals.update(frame.f_back.f_locals)
...             return copy_parent_scope
...     sys.settrace(copy_parent_scope)
...     def remove_tracer(cls):
...         sys.settrace(None)
...         return cls
...     return remove_tracer

This code is similar to what we had before, but there's a subtle change: because we want to keep the tracer on for nested classes, we don't remove it immediately, but only in our second function, invoked as a decorator once the outermost class definition is complete. This way, the tracer stays alive (and notice how it returns itself, to keep "stepping into" any scopes it encounters), awaiting a call event that would signal a class definition, and making sure its frame gets all the information from the preceding scope; this works because while class namespaces might not be linked in the way we expect, frame objects most definitely are. We also want to avoid stepping into actual calls, like the pathlib.Path("/app") invocation, so I've added the "__name__" in frame.f_code.co_names heuristic, which should only hold for classes. And now:

>>> @link_scopes()
... class config:
...     port = 8000
...     root = pathlib.Path("/app")
...     class db:
...         path = root / "db.sqlite" 

...     class log:
...         path = root / "log.txt"
...         maxsize = 20 * 1024 * 1024
...         backlog = 5
>>> config.db.path
Path('/app/db.sqlite')
>>> config.log.path
Path('/app/log.txt')

Just like we wanted. As far as I know, there aren't many ways to achieve the same with other Python features; I guess we could use __prepare__ to return cleverly-linked dictionaries that respect the class nesting hierarchy and resolve names accordingly, but that would take considerably more code than our little trick.

And a final note on performance: some people may be understandably horrified at the notion of installing a tracer in a production environment, since executing a callback on every single line of code does slow it considerably. Bear in mind, however, that we only do so surgically and temporarily, for definitions that happen once, at load time – so the actual impact of such an intervention is actually quite negligible.

The Making of People

Of course, I'd be surprised if this code makes it into production for any number of reasons – but like I said before, this is besides the point. I'm not advocating for you to install tracers that patch code in non-trivial ways because it's useful, but because I believe that knowing it's possible is.

This is the most important thing about metaprogramming – at least, how I see it, and why I wrote this book. I said as much in the introduction, but now that I've had a chance to take you for a spin, I'd like to repeat it one more time – and then I'll shut up about it for the rest of the book, I promise. Metaprogramming is not so much about the making of code — or even, as one might be inclined to think, about the making of code that makes code. Rather, it's about the making of people: its value is not in writing a more useful program (at least, not directly), but in educating, challenging, and otherwise enriching the programmer (who might then also write more useful programs as a result).

Now, cultivating creativity and imagination is always a worthy goal; or, more accurately, is always worthy of lip service, because most of us find it desirable, at least in theory. In practice, however, it's easier said than done: in many situations there is an inherent (if somewhat short-sighted) conflict between a person's growth and their throughput — an exploration vs. exploitation kind of dilemma.

And like I said, a lot of what makes us learn can be called play: doing things for fun, dreaming up new ideas or reinventing old ones. Consequently, it often leads to dead-ends; best practices are "best" for a reason, so unless we are struck by unusual inspiration, we will probably not reimagine the problem so radically as to arrive at a genuinely better solution. Even so, just as play-hunting is nature's tried and true way of learning how to hunt, so is play-inventing the best method I know of learning how to invent; or, to put it differently, if we don't try to reimagine problems, we'll never arrive at better solutions anyway.

Unfortunately, this takes time, and its utility is indirect. So when we engage in the opposite activity, that of work, where we care about the results rather than the process, it's easy to focus on what's at hand, and forget just how much the process shapes us as people – or how little the results do. Or, what results we could achieve in the future, if only we'd taken the time to grow ourselves, too.

That, I think, is where the meta- prefix becomes relevant. When we write code about other code, we're not exactly solving a problem; rather, we're building a tool or a framework that will facilitate the solution of future problems. As such, we are one step removed from any concrete requirement or burning need, almost by definition; it lets us focus on play, stretch our imagination — and, one might hope, rekindle a passion to our unusual craft.