Writing/The Unbearable Lightness of Coding
Chapter 12 · 42 min read
Objects 3: Mastering Magic Methods
In our first chapter about objects, we saw how they're comprised of an ID, a type and a value — and how it's the type that determines their overall behavior, which is the most important and interesting part. This behavior is outlined as part of their class, in magic methods that start and end with two underscores: __repr__, __eq__, __getitem__ and the like. We touched on some of them briefly, but there are quite a lot, and there's quite some nuance to them: so this time, let's go over everything properly.
Representation Revisited
We saw that if we override __repr__, we can customize the object representation:
>>> class A:
... def __init__(self, x):
... self.x = x
... def __repr__(self):
... return 'A(%s)' % self.x
>>> a = A(1)
>>> a
A(1)
But what makes a good representation? By convention, if we can replicate the code used to construct the object, like in the case of A(1), that's what we should do; and if we can't, or want to include a more elaborate description, we should put it between < and > like Python does in <A object at 0x…>. So on the face of it, the implementation above is alright — except, it isn't; and we've already seen this problem pop up once, when we talked about inheritance. Just take a look:
>>> class B(A):
... pass
>>> b = B(1)
>>> b
A(1)
Our hardcoding of A as part of the representation string doesn't account for the fact that the same method might be called from a different subclass with a different name. Luckily, this is easy enough to solve: since the class name is available in __name__ of the class (which is available in __class__ to its instance), we can make __repr__ use this dynamic resolution (or polymorphism, if you'd like), like so:
>>> class A:
... # same as before
... def __repr__(self):
... return '%s(%s)' % (self.__class__.__name__, self.x)
>>> a = A(1)
>>> a
A(1)
>>> b = B(1)
>>> b
B(1)
String Representation
Another thing to consider is that __repr__ is not actually what Python uses when attempting to print an object: in that case, it first attempts to convert it to a string using __str__. If this method is not defined, its default implementation (inherited from object) falls back to __repr__, so this nuance might not be immediately obvious:
>>> print(a)
A(1) # looks the same to me
But it provides us with an interesting opportunity: we can have two representation strings per object — one that is as elaborate as necessary to make development and debugging easy, and one that is more user friendly:
>>> class User:
... def __init__(self, is_admin=False):
... self.name = name
... self.is_admin = is_admin
... def __str__(self):
... return self.name
... def __repr__(self):
... return '%s(name=%s, is_admin=%s)' % (self.__class__.__name__,
... self.name, self.is_admin)
>>> alice = User('alice')
>>> alice
User(name=alice, is_admin=False)
>>> print(alice)
alice
Sometimes we might want the first one, and sometimes the second — the point is, there's no need to choose, as we can have both. If in doubt which one will be used, we can always pass the object to the builtin repr or str functions, which will convert it respectively; and when formatting strings (as you might have noticed, since I've been using it sneakily all along to make the examples look nicer), you can do either:
>>> print('%s logged in' % alice)
alice logged in
>>> print('%r logged in' % alice)
User(name='alice', is_admin=False) logged in
# or, with new-style formatting:
>>> print('{alice} logged in'.format(alice=alice))
alice logged in
>>> print('{alice!r} logged in'.format(alice=alice))
User(name=alice, is_admin=False) logged in
This is particularly useful for recursive representation. We've mentioned already that a good __repr__ returns a string that, if evaluated, would ideally yield the same object. However, this is easier said than done — in fact, I've deliberately made a mistake in the code above; see if you can spot it. A better way to implement the same thing would be:
>>> class User:
... # same as before
... def __repr__(self):
... return '%s(name=%r, is_admin=%s)' % (self.__class__.__name__,
... self.name, self.is_admin)
>>> alice = User('alice')
>>> alice
User(name='alice', is_admin=False)
You see, a string should be enclosed in quotes, and its raw representation knows as much, while the stringy one removes them for tidiness' sake. As long as all our objects adhere to this principle (as native objects always do), we can have fully viable object creation code ready on hand.
Why Formatting Matters
The reason I'm dwelling so much on this point is that it's invaluable in debugging and maintaining code. Incidentally, that's the reason the interpreter uses __repr__ by default: a simple alice after the >>> prompt gives us the whole picture, not the short and readable version of __str__. This also means that implementing __str__ alone is not enough:
>>> class A:
... def __str__(self):
... return 'a-object'
>>> A()
<A object at 0x...>
Nah, we need to print it, or at least convert it to a string for this to show up:
>>> print(a)
a-object
>>> str(a)
'a-object'
The rationale behind it is that Python, being such a dynamic and interpretative language, is practically begging to be used interactively, whether to experiment or to investigate. If you have a long running program that writes to a log, than if anything goes wrong, it might get a chance to record the offending objects' representation into a file — which we can then copy and paste into an interpreter session, poking it in various ways until we figure out what was the problem.
On the other hand, since Python is perfectly suited to implement command-line interfaces, web interfaces and other user facing displays, __str__ is equally important: we wouldn't want to dump a mountain of low-level and irrelevant information on an unsuspecting person, to whom it might look like some horrible error. For that reason, Python includes a third, less familiar method, __format__, which overrides __str__ to support an even more detailed formatting string. This is supported only in new-style formatting; but whenever we center a string, or write a number in hexadecimal with padding zeroes, or specify how exactly a datetime object is to be displayed:
>>> print('{title:^76}'.format(title='The Unbearable Lightness of Coding'))
The Unbearable Lightness of Coding
>>> print('{id:016X}'.format(id=id(a))
000001512EB115D0
>>> print('{now:%d/%m/%Y %H:%M:%s}'.format(now=datetime.datetime.now())
01/01/2024 12:00:00
Then this is exactly __format__ that we're using. To wit, we can customize our a-object representation so that it can be uppercase (if the formatting string contains U) or use an underscore instead of a dash (if it contains an underscore):
>>> class A:
... def __format__(self, desc):
... string = 'a-object'
... if 'U' in desc:
... string = string.upper()
... if '_' in desc:
... string = string.replace('-', '_')
... return string
>>> a = A()
>>> print('{a}'.format(a=a))
a-object
>>> print('{a:U}'.format(a=a))
A-OBJECT
>>> print('{a:_}'.format(a=a))
a_object
>>> print('{a:U_}'.format(a=a))
A_OBJECT
Equality Revisited
Another behavior we had discussed is equality: for example, we mentioned how __eq__ lets us write objects that can equal each other when their values or states are such that we would like to deem them the same:
>>> class A:
... def __init__(self, x):
... self.x = x
... def __eq__(self, other):
... return self.x == other.x
>>> a = A(1)
>>> b = A(1)
>>> a == b
True
>>> c = A(2)
>>> a == c
False
But what happens when we compare our object to a value of a different type? In this example, the objects' values conceptually represent a simple number, so it's not unthinkable to end up with code like this:
>>> a == 1
AttributeError: 'int' object has no attribute 'x'
Which crashes: the number 1 is passed in a.__eq__(1) as other, and naturally doesn't have an x attribute; the same would have happened to any other object that doesn't have this attribute, really. The responsible thing to do, then, is to double-check that the other's type is A:
>>> class A:
... # same as before
... def __eq__(self, other):
... return type(other) is A and self.x == other.x
>>> a = A(1)
>>> a == 1
False
This works, but has a different drawback; consider what happens with:
>>> class B(A):
... pass
>>> b = B(1)
>>> b == a
False
>>> b == b # ?!
False
Because we've hardcoded the expected type as A, its subclasses don't strictly qualify — to the degree that b doesn't even equal itself! What we should have done is:
>>> class A:
... # same as before
... def __eq__(self, other):
... return isinstance(other, A) and self.x == other.x
That is, making sure that other is-an A, in the broad sense.
Pleading Ignorance
We might be tempted to do the same for other comparison operators, like __gt__; after all, if we get an other with an incompatible type, they're going to crash with an error when we try to access their x attribute just the same:
>>> class A:
... # same as before
... def __gt__(self, other):
... return isinstance(other, A) and self.x > other.x
>>> a1 = A(1)
>>> a2 = A(2)
>>> a2 > a1
True
>>> a1 > 0
False
It sort of works; for A objects it makes sense, but for everything else it returns False, which is confusing, since if we'd implement __le__ in a similar way, A(1) < 0 would return False, too. And there's a finer point — suppose we have a class like Epsilon that's smaller than anything, by definition:
>>> class Epsilon:
... def __lt__(self, other):
... return True
>>> e = Epsilon()
>>> e < a1
True
>>> a1 > e
False
That's even more confusing, until we remember that e < a1 calls e.__lt__(a1), which always returns True, while a1 < e calls a1.__gt__(e), which returns False, since e is not an instance of A. So instead, what we should have done is:
>>> class A:
... # same as before
... def __gt__(self, other):
... if not isinstance(other, A):
... return NotImplemented
... return self.x > other.x
>>> a1 = A(1)
>>> e < a1
True
>>> a1 > e
True
NotImplemented (not to be confused with NotImplementedError) is a special singleton value, which tells Python that we don't know how to perform the operation for a given argument — so Python goes ahead and checks whether the other side of the comparison has an idea, which Epsilon does. It also means that our previous case raises an error:
>>> a1 > 0
TypeError: '>' not supported between instances of 'A' and 'int'
But honestly, that's better than returning a misleading False: it's not that the A object is not greater than 0, implying that it might be less than — it's that they're incomparable.
In other words, Python doesn't just call one magic method; it really attempts to consider the behavior in context. If the right handside knows what to do, great; and if not, and it admits as much, then the inverse behavior of the left handside is also given a chance. Actually, it's even smarter than that:
>>> class B(A):
... def __lt__(self, other):
... print('B')
... return self.x < other.x
>>> b = B(0)
>>> b < a
B
True
>>> a > b
B
True
b < a calls b.__lt__(a), no surprise there; but the printed B indicates that a > b calls it too, even though it should have been a.__gt__(b). The reason is that subclasses tend to be more specific than their base classes, so Python actually notices this relationship in advance and starts from the lowest node in the inheritance tree.
Hashing
There's one last thing to address about equality. Suppose we have a simple class, and we want to use its objects as keys in a dictionary or elements in a set:
>>> class A:
... def __init__(self, x):
... self.x = x
>>> a1 = A(1)
>>> a2 = A(2)
>>> d = {a1: 1, a2: 2}
>>> d[a1]
1
>>> d[a2]
2
Everything works great; but if we add a perfectly valid __eq__, like before:
>>> class A:
... # same as before
... def __eq__(self, other):
... return isinstance(other, A) and self.x == other.x
>>> a1 = A()
>>> a2 = A()
>>> d = {a1: 1, a2: 2}
TypeError: unhashable type: 'A'
It suddenly stops working, and we get a weird error complaining that our class is unhashable.
To understand what it means, let's talk about hashing. When Python stores something as a dictionary key or a set element, it does so in an optimized data structure called a hash map, which is able to locate its values exceedingly quickly. The way it does so is by allocating an array, and reducing complex objects into simple indexes (or hashes) in which to store them — hopefully in such a way that two objects will not end up with the same hash and colide (in which case it can still handle it, but less efficiently).
You can get the hash of any object in Python simply by passing it to the builtin hash function:
>>> hash(1)
1
>>> hash('hello world')
2787772774853031892
As for objects, if we take our previous, hashable implementation:
>>> a = A()
>>> hash(a)
90511708677
You can see that this value is actually part of their ID:
>>> a = A()
>>> hex(hash(a))
'0x1512eb1205'
>>> hex(id(a))
'0x1512eb12050'
Which makes sense, doesn't it? Different objects have different IDs, so they will also have different hashes — and since the default behavior of __eq__ is to compare objects by ID, like the is keyword, its implementation conforms the same logic, and so objects are hashable by default.
The reason hashing and equality should work in tandem, is that if two objects are equal, they should be interchangeable as keys in a dictionary, and not appear in a set, whose elements are supposed to be unique, at the same time. That's why, when we implement a custom __eq__, we introduce a problem: unless we also implement __hash__, Python can't be sure two equal objects will have equal hashes, which might violate how we expect sets and dictionaries to work.
Luckily, it's easy enough to fix: whenever we implement __eq__, and also want our objects to be hashable, we can implement __hash__ as well. We probably shouldn't reinvent our own hashing algorithm — delegating to the builtin hash function is perfectly valid, and we'd usually do so precisely to those values with which we determine equality:
>>> class A:
... # same as before
... def __hash__(self)
... return hash(self.x)
>>> a1 = A()
>>> a2 = A()
>>> d = {a1: 1, a2: 2}
>>> d[a1]
1
>>> d[a2]
2
As an aside, if multiple attributes are involved in a comparison (e.g. both the x and the y attributes of our objects have to match for ithem to be equal), we can simply pass all of them to hash in a tuple (as in, hash((self.x, self.y))).
Truth Values
It might feel like I'm telling you about some really minute and esoteric details — but this book is not just about programming, it's about software design. And one of the simplest and most important ideas in design is the principle of least astonishment: the idea that things should behave intuitively, like users would expect them to. To give an example, consider an object's truth value: by default, all user-defined objects are true, so you can use them in an if-statement like this:
>>> o = object()
>>> if o:
... print('object exists')
object exists
That is, as opposed to None, the anti-object whose truth value is false. But if our object represents a data structure, like a priority queue (where the higher the priority, the more the item will "advanced in the queue", and the earlier it will leave it):
>>> class PriorityQueue:
... def __init__(self):
... self.items = []
... def enqueue(self, item, priority=0):
... self.items.append((priority, item))
... self.items.sort()
... def dequeue(self):
... if not self.items:
... raise LookupError('queue is empty')
... _, item = self.items.pop()
... return item
Then users might expect its truth value to behave like a list or a dictionary — namely, be true if the queue is full, and false when the queue is empty. However:
>>> q = PriorityQueue()
>>> q.enqueue(1)
>>> q.dequeue()
1
>>> if q:
... q.dequeue()
LookupError: queue is empty
Since q is an object, it's always true — which astonishes our users, no less. We can address this with the magic method __bool__, which lets us decide on an object's truth value whenever its passed to the builtin bool function (even implicitly, as in if-statements):
>>> class PriorityQueue:
... # same as before
... def __bool__(self):
... return len(self.items) > 0 # or even: return bool(self.items)
>>> q = PriorityQueue()
>>> if q:
... q.dequeue()
# nothing happens, since queue is empty
In other words, adding __bool__ that returns whether we have at least one item makes our object behave in a way that's much more aligned with Python — and as a result, much more natural for Python programmers, who will presumably use this object.
Scott Meyers actually has a great talk about it, called The Most Importat Design Guideline, in which he lays it out like so: make interfaces that are easy to use correctly, and hard to use incorrectly. His talk is not about Python — he's a pretty famous expert in C++ — but the points he makes are language agnostic. The first part is pretty obvious, right? Our interfaces should be understandable and work as intended. It's the second part that's so interesting: nobody is deliberately trying to use our code wrong. If users are astonished at what happens, despite our common sense and best intentions, then it's fair to say there's something wrong with our design. There's a common tendency to blame distant, amorphous users — "they are using the product wrong", so to speak. But like Scott says in his talk, it's pretty safe to assume that actual users are somewhat intelligent, willing to read some documentation, and genuinely want their code to work — so if they find it hard, let alone fail outright, the fault actually might be with the product. It's not the same as "the customer is always right" mentality, where we just "give people what they want" — if we're solving a novel or difficult problem, the solution might be original enough to require users to adjust and familiarize themselves with new concepts. After that, though — and especially in mundane aspects, like a container's truth value — whatever concepts we come up with should conform to a (reasonable) user's intuition.
That's part of the reason Python is so great: sure, you have to learn plenty of details to wield it properly, but it's incredibly consistent. Did you notice how all the magic method names match the names of the builtin functions that invoke them? repr and __repr__, str and __str__, format and __format__. In Python 2, the magic method used for an object's truth value was actually called __nonzero__ (since it returns whether the object is non-zero, that is, true) — but this didn't match the builtin bool function, so it was renamed to __bool__.
And that's why, when it comes to designing objects, all these tiny details actually matter. Especially in Python, where we can override native behaviors and fit our ideas so snugly into the syntax, it's important to bear in mind that other people who use them are going to expect this syntax to check out in every case where it generally does. If we have a class, and we subclass it, the default representation string changes accordingly; so if we have a custom representation string that contains the class name, and someone subclasses it, they will legitimately except the same. If we have an epsilon object that's supposed to be smaller than everything, it should work whether we test for it through its less-than method, or through another object's greater-than. And if two equal objects are used in a dictionary, they should be interchangeable when used as a key. It's really not just nice-to-have, cosmetic touch ups: it's about making a conscious effort, as software designers, to create interfaces that are easy to use correctly, but also hard to use incorrectly.
Arithmetic Operations
With that, let's talk about object arithmetic. I mentioned briefly that when we see code like 1 + 1, it's actually equivalent to (1).__add__(1) — so you might see where this is going:
>>> class A:
... def __init__(self, x):
... self.x = x
... def __repr__(self):
... return '%s(%r)' % (self.__class__.__name__, self.x)
... def __add__(self, other):
... return A(self.x + other.x)
>>> a1 = A(1)
>>> a2 = A(2)
>>> a1 + a2
A(3)
But immediately, you might spot a few errors: if we subclass A with B, summing B objects will result in an A object; if we add an x-less value to A, like a1 + 1, we'll get an obscure exception; and if we have a class like Infinity, which should remain infinite whatever it's added to, we'll end up with an inconsistency:
>>> class Infinity:
... def __repr__(self):
... return '∞'
... def __add__(self, other):
... return self
>>> inf = Infinity()
>>> inf + a1
∞
>>> a1 + inf
AttributeError: 'Infinity' object has no attribute 'x'
So once again, we should check for other's type, returning NotImplemented if it's not an instance of A, and then use self.__class__ rather than a hardcoded A to instantiate the result:
>>> class A:
... # same as before
... def __add__(self, other):
... if not isinstance(other, A):
... return NotImplemented
... return self.__class__(self.x + other.x)
Besides addition, we naturally have subtraction in __sub__, multiplication in __mul__, and division — which used to be in __div__, but changed a little in Python 3. This might seem counter to my previous praise about consistency — but there's actually a good reason. Python supports two types of division:
>>> 5 / 2
2.5
>>> 5 // 2
2
The first can be thought of as true division, and the second as integer division, discarding the fraction and effectively flooring the result to the nearest whole number. So the name __div__ turned out somewhat ambiguous: which division does it stand for?
Naturally, it stood for the first one (the other way around would be bizarre), but it was still needlessly confusing; so to remove this hurdle altogether, the name was changed to __truediv__, as opposed to __floordiv__.
Then, we also have the modulo operation (%) in __mod__, and a bunch of bitwise operations: bitwise-or (|) in __or__, bitwise-and (&) in __and__, bitwise exclusive-or (also known as xor, ^) in __xor__, and the bitwise left- and right-shift (>> and <<) in __lshift__ and __rshift__.
Creative Arithmetic
This wealth of operations is not just for numbers (whose operations are, after all, already implemented); it aims to reflect the language at its fullest, and a lot of beautiful ideas flourish as a result. Strings, for example, support addition (to concatenate), multiplication (to repeat), and modulo (to format):
>>> 'hello ' + 'world'
hello world
>>> 'la' * 5
lalalalala
>>> '%x' % 32
0x20
So other text-like objects we implement can be reasonably expected to do the same. Similarly, set operations bear a lot of resemblance to bitwise logic, so they utilize its operators accordingly (together with subtraction):
>>> x = {1, 2}
>>> y = {2, 3}
>>> x | y # union
{1, 2, 3}
>>> x & y # intersection
{2}
>>> x ^ y # symmetric difference
{1, 3}
>>> x - y # difference
{1}
The same holds true for other unordered collections of unique items, like a dictionary's keys; one more way to intersect dictionaries would be:
>>> d1 = {'x': 1, 'y': 2}
>>> d2 = {'y': 2, 'z': 3}
>>> {key: d1[key] for key in d1.keys() & d2.keys()}
{'y': 2}
In Python 3.9, they even added a similar operator to merge them:
>>> d1 | d2
{'x': 1, 'y': 2, 'z': 3}
And then, there are some truly creative use cases: path objects, for example, which we'll cover later, represent filesystem paths — you know, like /etc/passwd — and since those are delimited by slashes, the (true) division operator is overloaded to join them (like os.path.join would), even with plain strings:
>>> import pathlib
>>> path = pathlib.Path('/etc')
>>> path / 'passwd'
PosixPath('/etc/passwd')
And if you're familiar with input/output redirection in Linux, where adding < to a command makes it read from a file, > makes it write to a file, >> append to it and | pipes the output of one command input to another — I've seen the same behavior emulated in Python like so:
>>> sort() < 'input.txt' # __lt__
>>> sort() > 'output.txt' # __gt__
>>> sort() >> 'more-output.txt' # __rshift__
>>> sort() | uniq() # __or__
Niche Arithmetic
But in fact, even all these methods are not enough. The extremely popular numeric library, numpy, naturally overloads all these operations to deal with -dimensional arrays, known as a matrix; and when multiplying two such matrices, multiplication can mean either element-wise multiplication or matrix multiplication, which is a different operation with a different result. numpy solved this by providing two separate functions, multiply and matmul — but neither is nearly as elegant as A * B, so eventually Python actually incorporated this feedback into the language, and a matrix multiplication operator was added: @, implemented in __matmul__, so A @ B is now valid syntax that does the right thing. Needless to say, it can be overloaded in a variety of ways, just like the rest; one more symbol to play with, however it makes sense.
And there are more: the __abs__ method, for examples, matches the builtin abs function, which returns the absolute value of a number — but for -dimensional vectors is used to return their norm (a square root of the sum of their elements squared). Notice that this is a unary operator, whereas all the operators we've seen so far are binary — so it doesn't receive other, but operates on the object itself. More unary operators include bitwise inversion, ~, implemented in __invert__; integer negation, -, implemented in __neg__; and its opposite, +, implemented in __pos__.
Interestingly, that last one has no effect on numbers (+1 is the same as 1), but since it's equally valid Python syntax, overloading it is equally well-supported; and who knows, a vector can take it to mean that that all its negative elements should be made positive (as absolute value would do for numbers — but then, abs means something else for vectors). The standard collections.Counter class, for example, uses it to return a "slice" of whatever score it stores, keeping only the positive:
>>> c = collections.Counter(x=1, y=0, z=-1)
>>> c
Counter({'x': 1, 'y': 0, 'z': -1})
>>> +c
Counter({'x': 1})
>>> -c
Counter({'z': 1})
A few other magic methods to mention here is __pow__, which lets us overload the power operator ** (or the builtin pow function); __divmod__, which lets us overload the builtin divmod function; and __round__, __floor__, __ceil__ and __trunc__, which let us override various standard rounding functions: math.round (which rounds a fraction to a specific precision), math.floor (which rounds down), math.ceil (which rounds up), and math.trunc (which removes the fraction altogether, returning an integer).
There's also __int__, which happens when we convert our object to an integer with the builtin int function, as well as __float__ and __complex__, which similarly map to the builtin float and complex convertors. There's even __index__, which lets us decide how exactly our object will be converted to an integer when used in slicing, or passed to the builtin bin, oct or hex functions — that's the crazy resolution Python lets us have over our objects' behavior.
As an aside, a few of these functions accept some optional parameters, so be sure to double-check their signatures when implementing them: __pow__ should support a modulo (e.g. pow(2, 1000, 100) gives 76, which is 21000 % 100, without having to actually compute that huge number), which isn't provided in case of e.g. 2**1000; and __round__ should support an optional ndigits parameter, specifying the expected rounding precision, which should default to a reasonable value if one isn't provided.
Right-Side Operations
But wait, there's more; what if I told you we only covered a bit more than a third of the available arithmetic operators — crazy, right?
Don't worry; this section and the next will mostly repeat what we already know, but with a small twist. Remember our Infinity object from before? Suppose we make it support subtraction, too:
>>> class Infinity:
... # same as before
... def __sub__(self, other):
... return self
>>> inf = Infinity()
>>> inf - a1
∞
That's cool; subtracting any value from infinity is still infinite. But what about a1 - inf? According to standard math, this should result in negative infinity; but since this code calls a1.__sub__(inf), do we really expect A to account for that? And if it just returns NotImplemented — what is Python supposed to do? After all, it's not the same operation as inf.__sub__(a1).
The problem is that while addition (or multiplication), for example, are cummutative operations, meaning a + b = b + a, subtraction (or division) are not: a - b is not necessarily the same as b - a. To support these cases, too, Python provides magic methods to support right-side operations for all the (binary) operators we covered so far: that is, methods that are called when our object is the right operand, like in the case of a1 - inf:
>>> class Infinity:
... def __init__(self, positive=True):
... self._positive = positive
... def __repr__(self):
... return '∞' if self._positive else '-∞'
... # same as before
... def __rsub__(self, other): # other - self
... return Infinity(positive=False)
>>> inf = Infinity()
>>> a1 - inf
-∞
In fact, there are __r*__ versions even for operations that are normally commutative, like addition — and if they're implemented, they actually take precedence over the regular versions. So even if we wouldn't amend A.__add__ to return NotImplemented (although we should!), Infinity could handle all the cases pertaining to it on its own:
>>> class Infinity:
... # same as before
... def __radd__(self, other):
... return self
>>> class A:
... # same as before
... def __add__(self, other):
... return self.__class__(self.x + other.x) # wrong, but...
>>> A() + Infinity() # still works!
∞
Because Python actually tries Infinity.__radd__(A) first — since it's present and clearly accounts for more specific cases. Only if it's missing, or returns NotImplemented, does Python try A.__add__(Infinity); and only if that's missing, or returns NotImplemented, and the operation is cummutative (like addition), does Python try Infinity.__add__(A).
In-Place Operations
The second twist has to do with in-place operatorions; which is, once again, all the (binary) operators we covered, but this time merged with an assignment statement:
>>> a1 = A(1)
>>> a2 = A(2)
>>> a1 += a2
>>> a1
A(3)
This code shouldn't be surprising; Python naturally unpacks a1 += a2 into a1 = a1 + a2, and since the sum results in A(3), this is what a1 ends up with. But look what happens in the process:
>>> a1 = A(1)
>>> id(a1)
1971056690128
>>> a1 += a2
>>> id(a1)
1971056690384
Since our implementation of __add__ actually returns a new object, the original a1 is discarded, and its name is reassigned to the result. This behavior is to be expected, since __add__ should primarily account for a situation like a1 + a2, in which a new object has to be created. But when we write +=, we might expect the original object to grow — and if we're dealing with large, complex objects, creating copies so casually can be wildly inefficient. So, Python gives us a chance to intercept this with __iadd__:
>>> class A:
... # same as before
... def __iadd__(self, other):
... self.x += other.x
... return self
In which case:
>>> a1 = A(1)
>>> id(a1)
1971056690128
>>> a1 += a2
>>> id(a1)
1971056690128 # same as before, and yet…
>>> a1
A(3)
Similarly, we have __isub__ for -=, __imul__ for *=, and so on — I'm not going to enumerate all of them, you get the point.
As an aside — and maybe it's not the best design decision, but at least in theory — we can use some in-place operation without supporting its full-fledged equivalent at all. It's easy to imagine a file object to which we can append text, or a queue to which we can add elements:
>>> f += 'the end'
>>> q += 1, 2, 3
Without the expressions f + 'the end' or q + (1, 2, 3) making any sense on their own. It might go against the principle of least astonishment — I would be surprised if I can add something one way but not another — but in some situations it might make sense, and come in handy.
Revisiting Containment
We already covered some container behaviors — namely, __getitem__, __setitem__ and __delitem__ — but let's give them another go. Most containers that can be indexed also have a notion of whether they contain some element or not, using the in and not in keywords, which is implemented by __contains__:
>>> class PriorityQueue:
... # same as before
... def __contains__(self, item):
... return item in self.items
>>> q = PriorityQueue()
>>> 1 in q
False
>>> q.enqueue(1)
>>> 1 in q
True
As well as a certain notion of length, how many items they have in total, using the bulletin len function and implement in __len__:
>>> class PriorityQueue:
... # same as before
... def __len__(self, item):
... return len(self.items)
>>> q = PriorityQueue()
>>> len(q)
0
>>> q.enqueue(1)
>>> len(q)
1
So if some class of ours pretends to be a container, it better not forget those two (or three, if we also count __bool__) "secondary" concerns. There's also the question of iteration — if something is a collection of items, we should be able to go over them one by one — but before we address it, there's one more nuance to bracket notation that I should mention. Python supports slice notation, which can be extremely useful:
>>> x = [1, 2, 3, 4, 5]
>>> x[2:4] # slice elements from the second until the fourth
[2, 3]
>>> x[:-2] # slice elements from the start until the second to last
[1, 2, 3]
>>> x[1::2] # slice elements from the second on, in increments of 2
[2, 4]
>>> x[::-1] # slice all the elements in -1 increments (so, in reverse)
[5, 4, 3, 2, 1]
When : is present as part of an index, Python automatically wraps it in the builtin slice class, filling up its start, stop and step attributes accordingly; so if we want to support this behavior, we need to be ready to accept such an object:
>>> class A:
... def __getitem__(self, arg):
... print(arg)
>>> a = A()
>>> a[2:4]
slice(2, 4, None)
>>> a[:-2]
slice(None, -2, None)
>>> a[1::2]
slice(1, None, 2)
>>> a[::-1]
slice(None, None, -1)
What's more, if you've used numpy, you might have seen more sophisticated slicing, necessary when dealing with -dimensional arrays:
>>> import numpy
>>> x = numpy.array([[1, 2], [3, 4]])
To index a 2-dimensional matrix, one index is not enough, as it will return the th row; we need another index, to hone in on the th cell within it:
>>> x[0]
array([1, 2])
>>> x[0][0]
1
However, numpy lets us do it with a single pair of brackets:
>>> x[0, 0]
1
And if we want to slice the first column, for example, we can do:
>>> x[:, 0]
array([1, 3])
That is, slice "all the elements" (:) in the first dimension, but keep only the first value of each row, returning a column. And if we'd have an 5-dimensional array where we'd like to slice the first 4-dimensional part of it, rather than writing x[:, :, :, :, 0], you might see code like x[..., 0]; … is another special singleton object in Python, known as Ellipsis:
>>> ...
Ellipsis
The way numpy cuts and dices its -dimensional arrays is quite sophisticated, but immaterial for our purposes; what we're interested in is how it supports such notation — what does it mean in Python. And as you might've guessed, it's actually quite simple: multiple values, like in [0, 0], are grouped into a tuple; a single : is a valid way, in this context, to represent slice(None, None, None); and ... is, well, Ellipsis. So:
>>> a[0, 0]
(0, 0)
>>> a[:, 0]
(slice(None, None, None), 0)
>>> a[..., 0]
(Ellipsis, 0)
All use cases we might want to support — and can do so, simply by treating __getitem__'s argument accordingly.
Iteration
Then, there's the iteration protocol. We've covered it twice already, when we talked about for-loops and generators, so I'll be brief: an object is iterable when it can be passed to the builtin iter function to produce an iterator — which, in additional to being iterable itself, can also be passed to the builtin next function to produce values, until it raises a StopIteration.
If we'd want to iterate over our priority queue, for example, we can implement its __iter__, making it return a temporary, one-time pass over its items. Conventionally, this object is defined as a nested class, since it's really an auxiliary construct that doesn't stand on its own:
>>> class PriorityQueue:
... # same as before
... def __iter__(self):
... return self.PriorityQueueIterator(self.items)
... class PriorityQueueIterator:
... def __init__(self, items):
... self.items = items
... self.index = 0
... def __iter__(self):
... return self
... def __next__(self):
... if self.index >= len(self.items):
... raise StopIteration()
... _, item = self.items[self.index]
... self.index += 1
... return item
It's pretty cumbersome, but it works:
>>> q = PriorityQueue()
>>> q.enqueue(1, priority=10)
>>> q.enqueue(3, priority=0)
>>> q.enqueue(2, priority=5)
>>> for x in q:
... print(x)
1
2
3
However, there's a much simpler way to do exactly the same:
>>> class PriorityQueue:
... # same as before
... def __iter__(self):
... for _, item in self.items:
... yield item
>>> q = PriorityQueue()
>>> # same as before
>>> for x in q:
... print(x)
1
2
3
What does it mean, to implement __iter__ as a generator function? Passing a priority queue into iter returns PriorityQueue.__iter__(), which results in a generator object. But as we've seen, this is an iterator by its very nature: resuming whenever it's passed to next, raising StopIteration when it's over, even evaluating to itself when passed through iter again — literally everything we've implemented manually before, but in three clean lines instead of 14 clunky ones.
A Different Approach
So whenever we decide to implement an iterator, it's worth considering to do so with a generator. I say "whenever", because if I'm honest we didn't strictly need to do it with PriorityQueue at all; if we remove __iter__ and implement __getitem__ instead, it would work just as well:
>>> class PriorityQueue:
... # same as before
... def __getitem__(self, index):
... return self.items[index][1]
>>> q = PriorityQueue()
>>> # same as before
>>> for x in q:
... print(x)
1
2
3
The reason is that Python, given __getitem__ and __len__ (which we've already added), can infer a basic iteration protocol on its own, even when __iter__ is missing: go with some i from 0 to len(q) and yield q[i] at each step. If we have these two methods, it's actually better than implementation iteration ourselves, because it will also work in reverse:
>>> for x in reversed(q):
... print(x)
3
2
1
That is, because Python is equally capable of going from len(q) to 0. If we'd remove __getitem__ and add __iter__ back, though, we'll get this:
>>> for x in reversed(q):
... print(x)
TypeError: 'A' object is not reversible
That is, Python can't jump to the end of our iterator and go from there — it's simply not how iterators work. To support this scenario, we'd have to implement __reversed__ ourselves, similarly yielding items, but this time from the lowest priority to the highest.
Why would we ever implement __iter__ and __reversed__, then, if __getitem__ and __len__ actually give us more? Well, it depends: if we know our object's length and th item, then it's better to not implement it — but in some cases we simply don't. A file, for example, can be iterated line by line:
>>> open('file.txt', 'w').write('line 1\nline 2\nline 3\n')
>>> for line in open('file.txt'):
... print(line)
line 1
line 2
line 3
But this doesn't mean we can know its length, or index a particular line:
>>> len(open('file.txt'))
TypeError: object of type '_io.TextIOWrapper' has no len()
>>> open('file.txt')[0]
TypeError: '_io.TextIOWrapper' object is not subscriptable
And that's because supporting this would require loading all of it to memory in advance, which can be a problem for large files. Reading it character by character, however, until we encounter a newline, and yielding just this part — this is a perfectly viable and extremely useful scenario, so it makes sense to implement __iter__ only. And while __reversed__ is not supported, it could be implemented, at least in principle, by moving the file cursor to the end of the file and reading it character by character, backwards.
But as we've seen, some streams can be infinite, like a pseudorandom number generator, and some collections can be unordered, like a set; in these cases, it's perfectly understandable to not have __len__ (what's the length of an infinite sequence?), nor __getitem__ (what's the ith item of an unordered set?), nor __reversed__ (how do you go backwards in an infinite sequence, or in an arbitrarily ordered one?) — only __iter__ makes sense, and that's OK, too.
A similar thing happens with __contains__, by the way: if it's not defined, Python will still attempt to service the in keyword — it'll just do it by iterating over the object (either with __iter__ or with __len__ and __getitem__, called the sequence protocol), seeing if any of them match. In this case, it can be pretty inefficient — so providing __contains__ is usually a good idea, unless there really is no other way to check if it's true.
Invocation
One last behavior I want to cover is invocation. If implement __call__, it means we can call our object like a function — and here, as opposed to all the other behaviors we discussed, the signature isn't predefined; so whatever parameters it expects will determine what arguments this object-turned-function will expect.
This means we can easily encapsulate behaviors in objects, like having a task that stores some configuration, and gets executed when it's invoked:
>>> class Task:
... def __init__(self, message):
... self.message = message
... def __call__(self):
... print(self.message)
This way, we can have a list of callable tasks, which we then iterate over and invoke as if they were functions:
>>> tasks = Task('message 1'), Task('message 2'), Task('message 3')
>>> for task in tasks:
... task()
message 1
message 2
message 3
It's pretty cool; but arguably, it's also one of those cases where we get blinded by all the shiny features, and use them somewhat excessively. If we're honest, we could have just as easily put it in a dedicated run method:
>>> class Task:
... # same as before
... def run(self):
... print(self.message)
>>> tasks = Task('message 1'), Task('message 2'), Task('message 3')
>>> for task in tasks:
... task.run()
message 1
message 2
message 3
And it would have been just as clear. In fact, whenever we see a class with a single method, we should ask ourselves whether a simple function would have suffices — after all, in Python, we can keep a list of those, too:
>>> def task(message):
... def run_task():
... print(message)
... return run_task
>>> tasks = task('message 1'), task('message 2'), task('message 3')
>>> for task in tasks:
... task()
message 1
message 2
message 3
We've seen this before; it's a higher-order function, keeping state via its scope for some inner functions that it defines and returns — something we used in second-order decorators, for example.
Decorator Classes
If you recall, defining a second order decorator was a bit cumbersome:
>>> def multiply(m):
... def decorator(f):
... @functools.wraps(f)
... def wrapper(*args, **kwargs):
... return m * f(*args, **kwargs)
... return wrapper
... return decorator
>>> @multiply(2)
... def inc(x):
... return x + 1
>>> inc(1)
4
But classes, if you think about, actually have two callable entry points: first, we call the class to create an object (which effectively runs __init__), and then we can call that object (which runs __call__). This means we can break this triple-nested contraption into more sensible and isolated steps:
>>> class Multiply:
... def __init__(self, m):
... self.m = m
... def __call__(self, f):
... @functools.wraps(f)
... def wrapper(*args, **kwargs):
... return self.m * f(*args, **kwargs)
... return wrapper
>>> @Multiply(2)
... def inc(x):
... return x + 1
>>> inc(1)
4
Which gives us a decorator factory class: an object that captures some m, and is also callable, so it can be used to decorate some f with multiplication by m. Any second order decorator can be broken down this way: first, receiving its configuration, and then applying it when called — so for big, complicated use cases, it might be advisable to break them down a little.
But we can further: what is we use the class itself as a decorator? Sticking it in a decoration line above a function will pass that function into its __init__, where it can capture it and invoke it whenever it is called, effectively replacing the wrapper. This can be useful if we want to expose state, like we had with the Fibonacci cache:
>>> class Memoized:
... def __init__(self, f):
... self.f = f
... self.cache = {}
... def __call__(self, *args, **kwargs):
... token = args + tuple(kwargs.items())
... if token not in self.cache:
... self.cache[token] = self.f(*args, **kwargs)
... return self.cache[token]
>>> @Memoized
... def fib(n):
... return n if n <= 1 else fib(n - 1) + fib(n - 2)
>>> fib(10)
55
>>> fib.cache
{(1,): 1, (0,): 0, (2,): 1, (3,): 2, (4,): 3, (5,): 5, (6,): 8, (7,): 13, (8,): 21, (9,): 34, (10,): 55}
That is, instead of adding the cache as a haphazard function attribute, we can expose it as a proper attribute of our object — including methods that calibrate it, or how the callable object behaves, altogether. In short, we can use decoration to replace a function with an equally callable object that, besides pretending to be the original function and delegating work to it, can still have all the benefits of an object. "Pretending" is a key word, though; it's still an object, as we can see here:
>>> fib
<Memoized object at 0x...>
Which is a bit unfortunate, since we've lost the function's name, docstring and such, just like we did when we discussed wrapping issues and the benefits of functools.wraps. Luckily, we can also use it in a class, albeit a bit differently; if you recall, it receives the original function, and was then used to decorate the wrapper, copying everything unto it. The same can be done in __init__ manually:
>>> class Memoized:
... def __init__(self, f):
... # same as before
... functools.wraps(f)(self)
... # same as before
We still loose the representation string:
>>> @Memoized
... def fib(n):
... 'Returns the n-th Fibonacci number.'
... return n if n <= 1 else fib(n - 1) + fib(n - 2)
>>> fib
<Memoized object at 0x...>
But at least we retain all the attributes:
>>> print(fib.__name__)
fib
>>> print(fib.__doc__)
Returns the n-th Fibonacci number.
Besides second-order and stateful decorations, there's another case where we'd want to replace a function with a special object: when we want to turn it into a descriptor, like a property — but that's a can of worms we'll open in the next chapter.
Magic Methods: Conclusion
So yeah, magic methods. We went over a lot of them: from equality, comparison and hashing, through arithmetic and containers, to class callables and decorators. Along the way, I tried to point our various nuances and caveats, so that if you end up implementing this behavior, you'll know how to do it right. But what I really tried to impart was this general principle of least astonishment: we don't just do it for some abstract quality's sake, but because we want our design to be as intuitive to users as possible. Python offers us an exceptionally rich toolbox to play with, but we shouldn't use magic methods casually: if we add a dunder-add, we shouldn't think about it as some cool extra feature that users should be grateful for. We might feel off the hook, but when other people use it, they might expect it to work from the right-side, or in-place, because that's just how addition is, and if we don't support it we might cause them more trouble than this cool extra feature is worth, so it would be better if we didn't add it. If we ever get a Python comics, something like dunder-man, it can say that with great methods comes great responsibility.