Writing/The Unbearable Lightness of Coding
Chapter 15 · 42 min read
Metaclasses (and Are They Even Useful?)
We've talked a lot about objects: what they are, what they do and where they come from. Since objects are defined in classes, we invariable talked about classes, too — but they were never our main focus. So in this chapter, it's time to give them central stage.
In other languages, like C++, classes don't actually exist in the final code: they are just directives for the compiler as to how objects should be laid out and wired. If class A defines attributes x, y and z and method f, it means that objects of type A are basically buffers with enough space to store these elements. a.x is translated to accessing the buffer at offset 0; a.y is translated to accessing it at offset sizeof(x); and a.z is translated to accessing it at offset sizeof(x) + sizeof(y). As for a.f(), it's translated to code that automatically pushes a pointer to a as the first argument, and then jumps to the definition of f, where any reference to this is replaced by access to that first argument. The details don't matter as much as the conclusion: in the end, no notion of the class itself remains in the compiled output — it's only relevant in preparing the code, to know what structures to expect and what constraints to enforce.
Even languages like Java, which support reflection — the ability of code to reflect on itself, like retrieving an object's class name or enumerating its attributes — doesn't really keep the classes around; it simply stores the metadata necessary to answer a limited subset of questions about them.
Python, on the other hand, makes no such discrimination: as we've seen, classes are objects, just like everything else:
>>> class A:
... pass
>>> A
<class 'A'>
They can be passed in as arguments to a function:
>>> def instantiate(cls):
... return cls()
>>> a = instantiate(A)
>>> a
<A object at 0x...>
Or passed back as return values:
>>> def create_class(x):
... class A:
... def f(self):
... return x
... return A
>>> A = create_class(1)
>>> a = A()
>>> a.f()
1
They can have attributes:
>>> A.x = 1
>>> A.x
1
>>> del A.x
And so on; and this has some pretty fascinating implications.
Decorating Classes
For example, it means we can have class decorators: just like a function decorator is placed on top of a function, gets invoked on it as soon as it's defined, and replaces it with its return value — so do class decorator operate, except they receive and return a class. It's not as common for them to define a brand new wrapper that delegates to the original — rather, they get a chance to inspect the newly defined class object and augment it as they see fit. If you recall our simple doubling (function) decorator:
>>> def double(f):
... def wrapper(*args, **kwargs):
... return 2 * f(*args, **kwargs)
... return wrapper
We can easily extend it to a decorator that doubles all the methods of a particular class:
>>> def double_all(cls):
... for key, value in cls.__dict__.items():
... if key.startswith('_') or not callable(value):
... continue
... doubled = double(value)
... setattr(cls, key, doubled)
... return cls
That is, we iterate over all the class __dict__, skipping any private or magic fields that might start with _ as well as simple, non-callable attributes — and for every public method we do find, replace it with its doubled version. Note that since we can't write directly to the class's __dict__, we need to use setattr; and don't forget to return the class at the end of the function — otherwise it'll be replaced with None! Here it is in action:
>>> @double_all
... class A:
... def inc(self, x):
... return x + 1
... def add(self, x, y):
... return x + y
>>> a = A()
>>> a.inc(1)
4
>>> a.add(1, 2)
6
To iilustrate some more useful functionality, consider a file class again:
>>> class File:
... def __init__(self, path):
... self.path = path
... def read(self):
... with open(self.path) as f:
... return self.read()
... def write(self, text):
... with open(self.path, 'w') as f:
... f.write(text)
If we have multiple threads working at the same time, we definitely don't want one of them to be reading a file while another one is writing to it; so to synchronize them, we might do something like this:
>>> def threadsafe(cls):
... cls._lock = threading.Lock()
... for key, value in cls.__dict__.items():
... if key.startswith('_') or not callable(value):
... continue
... synchronized = syncrhonize(value, cls._lock)
... setattr(cls, key, syncrhonized)
... return cls
>>> def synchronize(f, lock):
... def wrapper(*args, **kwargs):
... with lock:
... return f(*args, **kwargs)
... return wrapper
Stick it on top of the class, like so:
>>> @threadsafe
... class File:
... # same as before
And suddenly, all its methods are coordinated around the same lock, which our decorator adds as the class attribute _lock. This is known as coarse locking, because while it guarantees thread safety, it does so in the crudest way possible — any thread trying to use any method will have to wait until nobody else is using anything else, which will detract from the efficiency we might be trying to gain from this parallelism. Since the lock is defined on the class level, it's shared among all its objects — so even threads working on entirely different files will be waiting for each other; a finer resolution of locking might be achieved by adding it on the instance level:
>>> def threadsafe(cls):
... init = cls.__dict__.get('__init__')
... if init:
... def new_init(self, *args, **kwargs):
... init(self, *args, **kwargs)
... self._lock = threading.Lock()
... else:
... def new_init(self, *args, **kwargs):
... super(cls, self).__init__(*args, **kwargs)
... self._lock = threading.Lock()
... setattr(cls, '__init__', new_init)
... for key, value in cls.__dict__.items():
... if key.startswith('_') or not callable(value):
... continue
... synchronized = synchronize(value)
... setattr(cls, key, synchronized)
... return cls
With syncrhonized staying almost the same:
>>> def synchronize(f):
... def wrapper(self, *args, **kwargs):
... with self._lock:
... return f(self, *args, **kwargs)
... return wrapper
Here, we're doing something slightly more sophisticated: if the class defines __init__, we replace it with a function that calls the original, but also adds a _lock attribute to the instance; and if it doesn't, we add one that calls super (just to be sure) and then does the same — then, in the wrapper, we can safely assume that self._lock exists. Notice how because we're dealing with unbound methods here, we're passing self explicitly, since neither the original init nor the wrapped f will be bound to the instance. Also, while usually we can call super() with no arguments and Python will figure out what to do based on the class in which it's defined — here, we have no such class, and have to provide super with arguments explicitly, so it can orient itself accordingly.
Stick this on top of any class, and threads operating on the same instance will synchronize. If we wanted even finer resolution — for example, any number of threads can read a file at the same time, but if any one of them is writing to it everyone else should wait for it to finish — we'd have to add the locking logic to the class ourselves; as you might remember, the strength of Python's decorators is that they can be generic. So while our implementation is not the most efficient synchronization scheme imaginable, its value comes from that fact that you can apply it to any class — and with just one line, add a pretty substantial modification to all its methods.
Extending Classes
In fact, class decorators can even add logic that wasn't there before. Remember how implementing __lt__ was not enough to make its opposite, __ge__, work? Well, we can conceive of a decorator that fills in any such gaps itself: as long as __eq__ and at least one comparison operator is defined, the rest can be derived from them:
def total_ordering(cls):
eq = cls.__eq__ if cls.__eq__ is not object.__eq__ else None
ne = cls.__ne__ if cls.__ne__ is not object.__ne__ else None
if not ne:
ne = lambda self, other: not eq(self, other)
lt = cls.__lt__ if cls.__lt__ is not object.__lt__ else None
le = cls.__lt__ if cls.__le__ is not object.__le__ else None
gt = cls.__gt__ if cls.__gt__ is not object.__gt__ else None
ge = cls.__ge__ if cls.__ge__ is not object.__ge__ else None
if not any((lt, gt, le, ge)):
raise TypeError('must implement __lt__, __le__, __gt__ or __ge__')
if not lt:
if ge:
lt = lambda self, other: not ge(self, other)
elif gt:
lt = lambda self, other: not gt(self, other) and ne(self, other)
elif le:
lt = lambda self, other: le(self, other) and ne(self, other)
if not le:
if gt:
le = lambda self, other: not gt(self, other)
elif ge:
le = lambda self, other: not ge(self, other) or eq(self, other)
elif gt:
le = lambda self, other: gt(self, other) or eq(self, other)
# same for gt and ge
setattr(cls, '__lt__', lt)
setattr(cls, '__le__', le)
setattr(cls, '__gt__', gt)
setattr(cls, '__ge__', ge)
return cls
It's a bit tedious and confusing, but really, it doesn't do anything all the complicated. First, we need to acquire __eq__: unfortunately, cls.__dict__.get('__eq__') is not enough, since it might be defined in a superclass, and getattr(cls, '__eq__') is inconsequential as well, because all classes have a default __eq__ they inherit from object. So, to check for a custom __eq__, we make sure it's not the same as object.__eq__ (and the same for all the rest).
Next, if __ne__ is defined, we use it, and otherwise define it as the opposite of eq; we'll need it later. If none of the comparison operators is defined, we can't hope to implement the rest, so we raise an error — but otherwise, we go one by one and address its particularities. If lt () is missing, but we have ge (), they're just opposites; if not, we can make it the opposite of ge () as long as ne () holds, or the same as le () with ne — one of them is bound to be defined. The logic for le is slightly different, and then the logics for gt and ge mirror the same structure — until finally, we're ready to assign all the comparison operators back to the class, and get:
>>> @total_ordering
... class A:
... def __init__(self, x):
... self.x = x
... def __lt__(self, other):
... if not isinstance(other, A):
... return NotImplemented
... return self.x < other.x
>>> a1 = A(1)
>>> a2 = A(2)
>>> a1 < a2 # unsurprisingly, this works
True
>>> a2 >= a1 # but now, this works as well!
True
If we ever need to do something of this sort, though, we don't actually have to implement it ourselves; the standard library provides functools.total_ordering for us, I just wanted to show you the extent of surgery we can do on a class object — in this case, attaching any missing limbs, if you will.
(Standard) Class Creation
But if classes are objects — they must have a type, right? Their own class, in some sense, which is used to create them. And indeed they do; it's just a bit harder to see because of the way we normally instantiate them. If you look at this code:
>>> class A:
... pass
# A is created!
>>> A
<class 'A'>
That's the standard way classes are born — I marked the exact moment after which its object becomes available via its name. It's not your usual x = 1 or a = A(), but if you think about it, functions are created in a similar fashion: we write def f():, and they become available right after their code block is done, not through some explicit construction. In this sense, Python has four types of assignment: the regular one, function definition, class definition and the import statement, which binds a particular module to a name.
Anyway, we end up with an object that has three key attributes:
>>> A.__name__
'A'
>>> A.__bases__
(<class 'object'>,)
>>> A.__dict__
mappingproxy({'__module__': '__main__',
'__dict__': <attribute '__dict__' of 'A' objects>,
'__weakref__': <attribute '__weakref__' of 'A' objects>,
'__doc__': None})
That is: a name, a tuple of superclasses, and a dictionary of attributes; in this case just the standard ones, since we haven't defined anything in the class body. If we look at a more interesting example:
>>> class B(A):
... x = 1
... def f(self):
... return 2
# B is created!
>>> B.__name__
'B'
>>> B.__bases__
(<class 'A'>,)
>>> B.__dict__
mappingproxy({'x': 1, 'f': <function B.f(self)>, ...})
We'll see the name B, the superclass A, and both x and f in its dictionary, along with the same standard attributes from before. But how does Python know?
Unsurprisingly, __name__ is parsed from the name that appears right after the class keyword; and __bases__ are what appears in the parentheses right after that, or just object if no user-defined superclasses are provided. As for __dict__, Python goes ahead and executes the code in the class's body; but it does so in a special, separate scope. We've seen that scopes are little more than dictionaries — so all __dict__ is, in fact, is a snapshot of the class block's locals() after its execution is done.
This is a radically different paradigm to "classes as instructions for the compiler"; it's actual code being run, interacting with its scope as any code would — but with the result of its assignments and definitions preserved afterwards. It's easier to see here:
>>> class C:
... for i in range(3):
... print(i)
0
1
2
# C is created!
Bizarrely, having a loop in a class's body is perfectly legal — and it actually prints some stuff while the class is being evaluated! Now, think quickly: what class attributes does C have? We haven't explicitly defined any, and yet:
>>> C.i
2
Because at the end of the loop, the name i remains as part of the scope, still clinging to the value of its last iteration. A for-loop is not exactly an assignment statement — it's more of a side-effect; but I guess we could add it to our four ways of assignment (as well as the except-clause, maybe — although we saw that it's scope quickly expires).
Dynamic Class Creation
Still, what is the type of a class object? It's a bit confusing:
>>> C.__class__
<class 'type'>
Since this is exactly the same as the builtin type function, which we encountered when we were checking objects' types:
>>> type(1)
<class 'int'>
>>> C.__class__ is type
True # ?
>>> C.__class__(1)
<class 'int'> # ???
Turns out, type serves two purposes: when called with one argument, it returns its type (that is, its __class__ attribute); but when called with three arguments, it creates a class out of them. Our previous A and B, for example, could be created dynamically, like so:
>>> A = type('A', (), {})
>>> A
<class 'A'>
>>> B = type('B', (A,), {'x': 1, 'f': lambda self: 2})
>>> B
<class 'B'>
And it would work just as well as a standard class definition:
>>> b = B()
>>> b.x
1
>>> b.f()
2
>>> isinstance(b, A)
True
So in conclusion, all that Python does when creating a class is tease out its name and superclasses, execute its body, pluck its scope, and pass all three to the type function.
Incidentally, a standard function definition can also be done dynamically, although here we'd have to compile its code rather than execute it, figure out its global scope, and pass those to types.FunctionType:
>>> import types
>>> code = compile("print('hello world')", 'example.py', 'exec')
>>> f = types.FunctionType(code, globals())
>>> f()
hello world
Note that a function doesn't necessarily have a name (lambdas don't); but we can pass it in as an optional argument, together with its default arguments and closure.
So really, while there are several forms of assignment in Python, the way they work is always the same; it's just that some statements, like def, class or import, are syntactic sugar that does it for us — it lets us write out code in a more intuitive way, but then passes it in as arguments to a special-yet-regular object and binds it to a name.
Metaclasses
Interestingly, we can interfere in this process, just like with most anything else in Python. You see, since type is actually the class of classes, it's known as a metaclass — and we can define our own metaclasses.
The way this is done is simply by subclassing type — but there are a few nuances. To start, while regular classes' __init__ can have an arbitrary signature, a metaclass expects exactly three arguments to construct (or rather, initialize) a class: its name, bases and attributes. Also, since its first argument is this a new class object, we conventionally call it cls instead of self:
>>> class M(type):
... def __init__(cls, name, bases, attrs):
... print('created', cls)
The way to connect a class to a metaclass — that is, tell Python that this here should be created not with type, but with something else — is by passing it via the metaclass keyword argument in the class signature, where we'd normally pass superclasses:
>>> class A(metaclass=M):
... pass
created <class 'A'>
Once this relationship is established, it becomes an inherent part of the class — its type is no longer type, but M, and any subclasses of it are created accordingly:
>>> A.__class__
<class 'M'>
>>> class B(A):
... pass
created <class 'B'> # from M.__init__
>>> B.__class__
<class 'M'>
And since M is the class of A now, we can use it to define behaviors that will pertain to the class objects created from it:
>>> class M(type):
... def __repr__(cls):
... return 'M::' + cls.__name__
... def __getitem__(self, key):
... print('getting', key)
... return 1
>>> class A(metaclass=M):
... pass
>>> A
M::A
>>> A['x']
getting x
1
And note how this has nothing to do with A's objects:
>>> a = A()
>>> a
<A object at 0x...>
>>> a['x']
TypeError: 'A' object is not subscriptable
That is to say, M defines the behavior of its instances only — of which A is one; to define the behavior of A's instances, we need to do it in A's body.
It's weird, but it lets us dream up some pretty funky scenarios. We can, for example, override __call__ (which a class uses to instantiate its objects, as in a = A()), capture all the instances created by it, and then use __iter__ to produce an iterable thereof:
>>> class M(type):
... def __init__(cls, name, bases, attrs):
... cls._objs = []
... def __call__(cls, *args, **kwargs):
... obj = super().__call__(*args, **kwargs)
... cls._objs.append(obj)
... return obj
... def __iter__(cls):
... yield from cls._objs
And to A I'll add an x and a __repr__, just so we can tell its objects apart:
>>> class A(metaclass=M):
... def __init__(self, x):
... self.x = x
... def __repr__(self):
... return '%s(%r)' % (self.__class__.__name__, self.x)
>>> a1 = A(1)
>>> a2 = A(2)
>>> a3 = A(3)
>>> for a in A:
... print(a)
A(1)
A(2)
A(3)
Class Modification
From my experience, when people first hear about metaclasses — or rather, when they first understand them — they get pretty excited; it feels like we've just gained a new magical power, unprecedented in the degree of control it gives us over the language. And while in some sense it is true, the question of "what exactly should I use a metaclass for" remains elusive, leading either to needlessly complicated code or frustration and disappointment.
So, let's consider the two main scenarios that people have used metaclasses for, and see if we can shed light on this problem. The first one I call class modification: one of the opportunities a metaclass presents is to snatch a class as soon as it is defined and do something with it, augmenting or extending its functionality. I've seen frameworks, for example, that let you have type-safe Python classes like so:
>>> class A:
... x = int
>>> a = A()
>>> a.x
0
>>> a.x = 1
>>> a.x
1
>>> a.x = 'hello world'
TypeError: attribute x must be int
That is, they let you define an attribute and its type on the class level, and somehow enforce this constraint on its objects (even providing a default value for this attribute — namely, 0 is the result of calling int()).
And indeed, we can implement such a scheme with a metaclass:
>>> class TypeSafe(type):
... def __init__(cls, name, bases, attrs):
... for key, value in attrs.items():
... if isinstance(value, type):
... prop = TypeSafeProperty(key, value)
... setattr(cls, key, prop)
Provided that we also implement this property:
>>> class TypeSafeProperty:
... def __init__(self, name, type):
... self.name = name
... self.type = type
... def __get__(self, instance, cls):
... if instance is None:
... return self
... if self.name not in instance.__dict__:
... instance.__dict__[self.name] = self.type()
... return instance.__dict__[self.name]
... def __set__(self, instance, value):
... if not isinstance(value, self.type):
... raise AttributeError('attribute %s must be %s' %
... (self.name, self.type.__name__))
... instance.__dict__[self.name] = value
Simply put, we replace each class attribute bound to some type with a descriptor, which in turns uses this type to instantiate it upon first access. It also lets us to set it to something else, but only as long as the assigned value matches the expected type.
And it works; all we have to do is pass TypeSafe to A as a metaclass:
>>> class A(metaclass=TypeSafe):
... x = int
>>> a = A()
>>> a.x
0
>>> a.x = 1
>>> a.x
1
>>> a.x = 'hello world'
TypeError: attribute x must be int
And that's cool. We can even improve it a bit: writing metaclass=TypeSafe is a bit clunky, so let's call it TypeSafeMetaclass instead, and define an empty TypeSafe class to preestablish this connection:
>>> class TypeSafeMetaclass(type):
... # same as before
>>> class TypeSafe(metaclass=TypeSafeMetaclass):
... pass
Now, simple inheritance should be enough:
>>> class A(TypeSafe):
... x = int
And that's some darn elegant design for a pretty intrusive manipulation.
Class Annotations
As an aside, this was the old way of binding a class attribute to a type, even though x is not actually int — we just want to communicate to the framework that it should have an integer value, and don't really have any other way of linking the two together. In Python 3.6+, however, we can do so with annotations, similar to those we've seen in functions:
>>> class A:
... x: int = 1
... y: int # no value
Any annotated attributes end up in the class's __annotations__ dictionary; and in its __dict__ as well, if they have a value — although, in this case, they don't have to:
>>> A.__annotations__
{'x': <class 'int'>, 'y': <class 'int'>}
>>> A.x
1
>>> A.y
AttributeError: type object 'A' has no attribute 'y'
This means that we can improve our solution further: annotations seem like a more natural fit for our purposes, and we can even grab their values, if present, to be used as the initial, default ones:
>>> class TypeSafeMetaclass(type):
... def __init__(cls, name, bases, attrs):
... for key, value in cls.__annotations__.items():
... # same as before
>>> class TypeSafeProperty:
... # same as before
... def __get__(self, instance, cls):
... # same as before
... if self.name not in instance.__dict__:
... if hasattr(cls, self.name):
... default = getattr(cls, self.name)
... else:
... default = self.type()
... instance.__dict__[self.name] = default
... return instance.__dict__[self.name]
>>> class A(TypeSafe):
... x: int = 1
>>> a = A()
>>> a.x
1
>>> a.x = 'hello world'
TypeError: attribute x must be int
However, if we're honest — there's no real need to use a metaclass here; we've already seen a way to modify a class upon creation, namely with class decorators. Exactly the same can be achieved without inheriting from type and using all its magicky techniques — simply by receiving the class as an argument:
>>> def typesafe(cls):
... for key, value in cls.__annotations__.items():
... if isinstance(value, type):
... prop = TypeSafeProperty(key, value)
... setattr(cls, key, prop)
... return cls
>>> @typesafe
... class A:
... x: int = 1
>>> a = A()
>>> a.x
1
>>> a.x = 'hello world'
TypeError: attribute x must be int
So maybe we don't really need metaclasses for modification, after all. But maybe…
Class Registration
In some frameworks, the users get to define logics in functions or classes, which are then hoisted and integrated into a more complex flow — we've touched briefly on webservers and tests, which are two examples of this. To give another that involves classes rather than functions, consider a situation where we define commands like so:
>>> class AddCommand:
... name = 'add'
... arguments = {'x': int, 'y': int}
... def run(self, x, y):
... return x + y
>>> class HelloComand:
... name = 'hello'
... options = {('-n', '--name'): str}
... def run(self, name=None):
... print('hello', name or 'stranger')
Based on these definitions, we'd like to stick something like run_cli() at the end of our module (say, example.py), and have it translated into a command-line interface:
/home/user$ python example.py add 1 2
3
/home/user$ python example.py hello
hello stranger
/home/user$ python example.py hello --name=alice
hello alice
Nevermind how the command-line arguments and options are parsed, or how our commands are invoked; our task for now is to simply mark those classes as commands to be collected. We could do it with a decorator, which would simply pass the class through while retaining a reference to it:
>>> commands = []
>>> def command(cls):
... commands.append(cls)
... return cls
>>> @command
... class AddCommand:
... # same as before
>>> @command
... class HelloCommand:
... # same as before
>>> commands
[<class 'AddCommand'>, <class 'HelloCommand'>]
And then use the list of references, commands, to construct our CLI. It's a fine solution, but an argument can be made that it's lacking support for inheritance: if we'd want to isolate some common functionality in BinaryCommand and have AddCommand and others inherit from it:
>>> commands = []
>>> @command
... class BinaryCommand:
... arguments = {'x': int, 'y': int}
>>> class AddCommand(BinaryCommand):
... # same as before
>>> commands
[<class 'BinaryCommand'>]
Then we're not going to get any of the subclasses unless we decorate them, too. Had we used a metaclass, on the other hand:
>>> commands = []
>>> class Command(type):
... def __init__(cls, name, bases, attrs):
... commands.append(cls)
Then we would capture the class, as well as all its future subclasses:
... class BinaryCommand(metaclass=Command):
... # same as before
>>> class AddCommand(BinaryCommand):
... # same as before
>>> commands
[<class 'BinaryCommand'>, <class 'AddCommand'>]
In other words, metaclasses are subclass-aware in a way that decorators are not; so maybe they have an edge here?
And they did, once; until Python 3.6 introduced the __init_subclass__. It's an (implicit) class method that gets called whenever a new subclass is defined — so if we're interested in tracking our inheritance tree, we don't need to go much further than:
>>> commands = []
>>> class Command:
... def __init_subclass__(subclass):
... commands.append(subclass)
>>> class BinaryCommand(Command):
... # same as before
>>> class AddCommand(BinaryCommand):
... # same as before
>>> commands
[<class 'BinaryCommand'>, <class 'AddCommand'>]
To get exactly what we want — no metaclasses, no nothing.
Dynamic Inheritance
So what are metaclasses useful for? Let's tackle this question from a different angle: instead of what's popular, let's consider what they make feasible. Metaclasses let us intercept class creation, receiving its name, bases and attributes — but the name and the attributes we can change after the class is created, so that's not a big deal. Not so the bases: if we want to augment the inheritance tree, our only opportunity to do so is in the metaclass's __new__, before the class object is even created.
We can, for instance, conceive of superclass injection: rather than hardcoding Dog to subclass Animal, merely specify a list of intentions or expectations via dummy classes like Eats and Poops — and have a metaclass figure out what exactly superclasses will conform these requirements. It's similar to dependency injection, where we inspected a function's signature to decide which arguments to pass — and this way, we can apply sweeping changes to the inheritance tree across many classes, choosing dynamically which superclasses serve the specified interfaces best. Today, it might be Animal:
>>> class Eats:
... pass
>>> class Poops:
... pass
>>> class InheritanceManager(type):
... def __new__(mcs, name, bases, attrs):
... actual_bases = set()
... for base in bases:
... if base is Eats:
... actual_bases.add(Animal)
... elif base is Poops:
... actual_bases.add(Animal)
... else:
... raise ValueError('unfamiliar interface: %r' % base)
... return super().__new__(mcs, name, tuple(actual_bases), attrs)
And note how __new__ is an implicit metaclass-method, so its first argument is the metaclass mcs; just like in classes, it was an implicit class-method that receives cls — we're consistently jumping one level of abstraction "above" regular classes.
Anyway, see Animal getting injected:
>>> class Animal:
... def eat(self, food):
... print('slurp', food)
... def poop(self):
... print('💩')
>>> class Dog(Eats, Poops, metaclass=InheritanceManager):
... pass
>>> dog = Dog()
>>> dog.eat('snack')
slurp snack
>>> dog.poop()
💩
And if tomorrow we decide to split up this functionality:
>>> class InheritanceManager(type):
... def __new__(mcs, name, bases, attrs):
... actual_bases = set()
... for base in bases:
... if base is Eats:
... actual_bases.add(Eater)
... elif base is Poops:
... actual_bases.add(Pooper)
... else:
... raise ValueError('unfamiliar interface: %r' % base)
... return super().__new__(mcs, name, tuple(actual_bases), attrs)
>>> class Eater:
... def eat(self, food):
... print('slurp', food)
>>> class Pooper:
... def poop(self):
... print('💩')
Our Dog class (and presumably, many other animal subclasses) need not change at all. This might be an interesting approach to mitigate some of the drawbacks of inheritance we discussed — namely, that it commits to a fix taxonomy, which it turns out that we can change on the fly.
Metaclass Keywords
It's a cool idea — but if you cringed at the horrible coupling between the metaclass and the routing logic, I'm with you; it would be a nightmare to manage a complicated inheritance tree in an already strange __new__.
Luckily, we can pass additional keywords in a class signature. Broadly speaking, it has the form A(*bases, metaclass=type, **kwargs): superclasses are passed positionally, we have the special metaclass keyword that defaults to type, and anything else is captured and passed along to the metaclass's __new__ and __init__.
This means we can actually customize the metaclass's operation with parameters. The reason I didn't mention it before is that in our previous scenarios, it wouldn't really change our conclusions: decorators can also be parameterized, and as for __init_subclass__, it actually receives these keywords, too:
>>> class A:
... def __init_subclass__(subclass, **kwargs):
... print(kwargs)
>>> class B(A, x=1, y=1):
... pass
{'x': 1, 'y': 2}
So anything we can track in the metaclass, we can track there. In our case, however, we can isolate the routing logic in a resolve_animal function like this one:
>>> def resolve_animal(bases):
... for base in bases:
... if base is Eats:
... yield Animal
... elif base is Poops:
... yield Animal
... else:
... raise ValueError('unfamiliar interface: %r' % base)
In which case we can have a metaclass as simple as:
>>> class InheritanceManager(type):
... def __new__(mcs, name, bases, attrs, resolve_bases):
... actual_bases = tuple(set(resolve_bases(bases)))
... return super().__new__(mcs, name, actual_bases, attrs)
And pass it in as:
>>> class Dog(Eats, Poops, metaclass=InheritanceManager,
... resolve_bases=resolve_animal):
... pass
Which is admittedly cumbersome, but can be encapsulated in a single base class, as we've already seen:
>>> class That(metaclass=InheritanceManager, resolve_bases=resolve_animal):
... pass
So we have a simple and clean:
>>> class Dog(That, Eats, Poops):
... pass
This way, should we want to change the inheritance tree, we only need to edit resolve_animal, or just pass a different resolver as That's argument.
But even that, it turns out, we can do differently: this time, with the special method __mro_entries__. It's actually an instance method, which is called when some object appears in a class signature — and it's called on the classes bases, returning a tuple of superclasses that will replace it in the inheritance tree. This means we can drop all our dummy classes altogether and resort to simple strings, from which we construct a That object that resolves them to the appropriate superclasses:
>>> class That:
... def __init__(self, *interfaces):
... self.interfaces = interfaces
... def __mro_entries__(self, bases):
... bases = set()
... for interface in self.interfaces:
... if interface == 'eats':
... bases.add(Animal)
... elif interface == 'poops':
... bases.add(Animal)
... else:
... raise ValueError('unfamiliar interface: %r' % interface)
... return tuple(bases)
In which case we can define Dog like so:
>>> class Dog(That('eats', 'poops')):
... pass
And have:
>>> Dog.__bases__
(Animal,)
Leaving us free to change this routing as we please, without involving any metaclasses.
Special Class Behavior
But surely, you might think, metaclasses are the only way to define special behavior for classes — having something like A['x'] or for a in A, as we've seen; and here, you'd be correct. Just like objects are instances of a class, meaning all their behavior derives from its magic methods — so are classes instances of a metaclass, and the only way to implement something like class comparison or class division would be there.
However, this goes very strongly against the principle of least astonishment: users are going to be quite surprised to see A < B or A / B, so it might not be the best interface for whatever it is we're trying to do. I say "might not", because it really depends on the context — but unless you have a use case that really justifies such syntax, ask yourself if you're seriously solving the problem or just showing off.
To give an example of such a use case, consider typing annotations again. We've seen int and str, but what about more complex types, like a list or a dictionary thereof? The syntax popular in other languages is to use square brackets, as in List[int] for a list of numbers, or Dict[str, bool] for a dictionary mapping strings to booleans — and while we can create regular objects for List and Dict, conceptually they should be classes. And anyway, if we want a user-defined class to be used this way, like A[int], it seems we have to go poke at its metclass.
However, there's an interesting feedback loop happening in Python's design. As this new syntax became widespread and familiar, this particular scenario was made simpler, with version 3.7 introducing __class_getitem__:
>>> class A:
... def __class_getitem__(cls, key):
... print('getting', key)
... return 1
>>> A['x']
getting x
1
That is, another implicit class method, which lets us add support for bracket notation on any class that defines it, without having to resort to changing its metaclass. Note that since this syntax isn't widely used for assignment or deletion, there is no equivalent __class_setitem__ or __class_delitem__ — but if this or any other behavior ever becomes widely accepted enough to justify their addition, I expect to see them as well.
Namespace Preparation
Another special behavior that's worth mentioning is namespace preparation. We haven't seen it before because it doesn't make sense for regular objects; but in Python 3, classes gained a special behavior called __prepare__.
This method is run even before the class is created, and its purpose is rather special: it returns the dictionary into which the class body will be evaluated — effectively, its local scope. Normally, it's a regular dictionary, so everything happens just like you might expect — but in some cases, it gives us an opportunity to do some pretty unusual tricks.
The rationale behind its introduction was pretty specific: before Python 3.6, dictionary order was not guaranteed — and as a result, class attributes were unordered as well. Usually it doesn't matter, but sometimes it might, like if we want to serialize our objects into a format read by another program, and we want the order in which our fields are defined to be preserved. It would be nice to let the user write:
>>> class A:
... x = 1
... y = 2
And be able to retain the information that x comes before y; however, in a world of unordered dictionaries, this is impossible: all we're left with after the class is evaluated is {'x': 1, 'y': 2} (or maybe, {'y': 2, 'x': 1}), and no way of telling which key was set first. Various frameworks tackled this problem by having users write:
>>> class A(metaclass=Ordered):
... x = Field(1)
... y = Field(2)
In which case the Field class could keep track of its instantiation order:
>>> class Field:
... counter = itertools.count()
... def __init__(self, value):
... self.value = value
... self.order = next(self.counter)
Which the Ordered metaclass would then capture while peeling these Fields off:
>>> class Ordered(type):
... def __init__(cls, name, bases, attrs):
... fields = []
... for key, value in attrs.items():
... if not isinstance(value, Field):
... continue
... fields.append((value.order, key))
... setattr(cls, key, value.value) # peel fields off
... fields.sort()
... cls._fields = [name for _, name in fields] # retain fields order
This gives us a class with the original values, but also their order (e.g. in _fields):
>>> A.x
1
>>> A.y
2
>>> A._fields
['x', 'y']
This works, but it's not ideal; so __prepare__ was suggested as a hook that could replace the default dictionary with one that does preserve order, namely collections.OrderedDict. Since such preparation happens before the class body is evaluated, it's actually an implicit static method within the metaclass; cls doesn't exist yet, mcs is irrelevant, and attrs doesn't exist either, so it only receives name and bases. Afterwards, though, attrs arrives to __init__ as the dictionary we prepared — so if it's an OrderedDict, we can extract the definition order simply by listing its keys (and without having had to wrap everything in a Field object):
>>> class Ordered(type):
... def __prepare__(mcs, name):
... return collections.orderedDict()
... def __init__(cls, name, bases, attrs):
... cls._fields = list(attrs)
>>> class A(metaclass=Ordered):
... x = 1
... y = 2
>>> A._fields
['x', 'y']
This use case, however, was rendered moot when dictionaries became ordered in Python 3.6 due to a new, more efficient implementation; and when this new reality became so popular that ordering was ratified as a guaranteed part of their interface in Python 3.7, OrderedDict became somewhat redundant.
Implementing C-Like Enums
Since then, I haven't seen many useful applications for __prepare__, but it's still cool. Suppose you're coming from a language like C, where enums are defined like so:
enum Level {
LOW,
MEDIUM,
HIGH
};
In other words, there's no need to assign a value to each, like we'd have to do in a Python class:
>>> class Level:
... LOW = 0
... MEDIUM = 1
... HIGH = 2
In C, the numbering just starts from 0 and rolls on — and it's quite unacceptable, letting C get away with being more elegant than Python. As it turns out, though, we can actually support the same syntax in Python, although initially it might seem invalid. What we're going to do is provide our own dictionary implementation, called EnumDict; it will subclass the original dict, but augment its logic, so that when a key is missing, rather than throwing a KeyError, we'll assign it the next value from a counter:
>>> class EnumDict(dict):
... def __init__(self):
... self.counter = itertools.count()
... def __getitem__(self, key):
... if key not in self:
... self[key] = next(self.counter)
... return super().__getitem__(key)
>>> class EnumMetaclass(type):
... def __prepare__(mcs, name):
... return EnumDict()
This way, we can actually define a class like this:
>>> class Level(metaclass=EnumMetaclass):
... LOW
... MEDIUM
... HIGH
And it works:
>>> Level.LOW
1
>>> Level.MEDIUM
2
>>> Level.HIGH
3
To understand why, consider what's happening when the class body is running. The first statement is simply LOW; it's not a very interesting one, but it's still valid, provided Python can resolve this name into a value. Normally, since it wasn't defined, the scope dictionary would throw a KeyError, which would be translated into a NameError; but since our dictionary implements accounts for this absence by simply generating a number for it, the code works. LOW is thus resolved to 0, MEDIUM to 1 and HIGH to 2 — and by the time the class body is executed, EnumDict stores precisely those values, which become attributes. In other words, merely touching these names was enough to define them — and we've effectively augmented Python to support something most people consider invalid syntax, saving face in our competition with C.
Method Overloading
Another interesting example is method overloading: defining functions with a similar name but different signatures, so when the method is invoked, the call is routed to the version that matches its arguments. In a language like Java, it's quite useful if we want to provide similar functionality for different types:
class A {
public int add(int x, int y) {
return x + y;
}
public float add(float x, float y) {
return x + y;
}
}
Or if we want to support optional parameters:
class A {
public void hello() {
System.out.println("hello stranger");
public void hello(String name) {
System.out.println("hello " + name);
}
}
In Python, it's quite immaterial: we can use duck typing and default arguments:
>>> class A:
... def add(self, x, y):
... return x + y
... def hello(self, name=None):
... print('hello', name or 'stranger')
Which is cleaner and less confusing. Still, if we'd want to overload a method in Python (say, based on its number of arguments), we'd find it impossible:
>>> class A:
... def f(self):
... print(0)
... def f(self, x):
... print(1)
>>> a = A()
>>> a.f(None)
1
>>> a.f()
TypeError: A.f() missing 1 required positional argument: 'x'
Simply because of how dictionaries work. When the class body is executed, the first f is defined and added to its scope — but then the second f overrides it, and we end up with the last version of this name, like we would if we had x = 1 and then x = 2.
How annoying, that Java can do something that Python can't. So lets prepare us a MultiDict: a dictionary capable of holding multiple values under the same key. Rather than implementing the whole thing properly, I'll be lazy and just amend __setitem__ to store every value as a list with a single item, so if a key already exists, we just append the additional value to its list:
>>> class MultiDict(dict):
... def __setitem__(self, key, value):
... if key.startswith('_'): # don't screw with special attributes
... super().__setitem__(key, value)
... elif key in self:
... self[key].append(value)
... else:
... super().__setitem__(key, [value])
Then, in the metaclass, we'll replace all the one-item lists (meaning, values that were only defined once, as usual) with themselves; and all the longer lists we'll stick into a special router:
>>> class Overloaded(type):
... def __prepare__(mcs, name):
... return MultiDict()
... def __init__(cls, name, bases, attrs):
... for key, values in attrs.items():
... if key.startwith('_'):
... continue
... if len(values) == 1:
... setattr(cls, key, values[0])
... else:
... setattr(cls, key, OverloadRouter(values))
All this router has to do is map the methods to their number of arguments and, when called, redirect the call to the right one (accounting for the extra argument self):
>>> class OverloadRouter:
... def __init__(self, fs):
... self.fs = {f.__code__.co_argcount: f for f in fs}
... def __call__(self, *args):
... return self.fs[len(args) + 1](self, *args)
It's pretty obscene — but guess what, it works:
>>> class A(metaclass=Overloaded):
... def f(self):
... print(0)
... def f(self, x):
... print(1)
>>> a = A()
>>> a.f(None)
1
>>> a.f()
0
Although in real life, I'd probably just use f(self, *args). Anyway, take that, Java!
The Underwhelming but Beautiful Truth about Metaclasses
I hope it's been a fun ride — metaclasses are definitely cool. But the somewhat underwhelming truth is that unless you really know what you're doing, and you're absolutely sure you need a metaclass, then you can probably make do without one. Class modification is easier with decorators; class registration is easier with __init_subclass__; dynamic inheritance can be achieved with __mro_entries__; and even special behavior, at least the useful one, we can get via regular methods like __class_getitem__. And while we can implement class iteration, or funky syntax that tolerates undefined names or support overloading — other than pranking our colleagues or winning which-language-is-better contests driven by toxic masculinity, there's little value to be gained.
So why are metaclasses there? Because the language needs them — they're the machinery underlying class creation, and the fact that they're available to us doesn't mean we actually have to use them. It's the usual empowerment Python gives, but there's an even more beautiful attitude at play here — because whenever new use cases involving metaclasses do arise, whether it's inheritance tracking, ordered attributes or compound type annotations, they get assimilated into the language and made available via simpler means.
There's a practical reason for this, too: metaclasses are complicated. Not just because they're so meta, but because they effectively juxtapose an additional inheritance tree on top of whatever inheritance we might be dealing with. If A has metaclass MA, and B has metaclass MB, we can't inherit from both:
>>> class MA(type):
... pass
>>> class MB(type):
... pass
>>> class A(metaclass=MA):
... pass
>>> class B(metaclass=MB):
... pass
>>> class C(A, B):
... pass
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
Because Python can't decide which metaclass to use for C's creation; if MB was a subclass of MA, at least, then it would be the more specific of the two, and Python could choose it with confidence:
>>> class MB(MA):
... pass
>>> # same as before
>>> class C(A, B):
... pass
>>> C.__class__
<class 'MB'>
This might seem like an improbable situation, but consider what happens if we're using two frameworks that leverage metaclasses: we don't really know how they're implemented behind the scenes, so we go about our business using both — when suddenly we start getting opaque errors about advanced concepts. Even if we understand what they mean, they're not easy to ameliorate: we'd have to create a metaclass that reconciles the meta-inheritance tree of both frameworks and use it explicitly, which is probably not the seamless user experience either framework was aiming for.
So again, unless you're absolutely certain you should be using a metaclass, you probably shouldn't; and this certainty is only rewarded when Python's core developers come up with a way to address the same needs without metaclasses. Still, it's a fascinating language feature that generates a fascinating dialectics; so to this chapter's titular question, "are metaclasses even useful?", I would say: no — but they facilitate such an ecosystem and dialectic around language design, that I dare say they're meta-useful.