Writing/The Unbearable Lightness of Coding
Chapter 13 · 49 min read
Objects 4: Attributes, Methods and Descriptors
Over the last few chapters, we talked a lot about objects, what they are and how they can behave. But we also glossed over what is arguably the key feature of objects: their attributes and their methods, which we just assumed to work.
So in this chapter, we'll dive into what makes this possible: how attribute resolution works, when does a function become a method, the descriptor protocol and more. I don't know why, but for some reason I Always felt that understanding this particular topic somehow sets apart junior and senior Python developers; something about its design is just really special.
***
Let's start with attributes. When we define an instantiate a class, we can use dot notation to access, set or delete its attributes:
>>> class A:
... pass
>>> a = A()
>>> a.x = 1
>>> a.x
1
>>> del a.x
In fact, when we create an object and __init__ is called, that's exactly what its code does: the object is instantiated beforehand and gets passed in as self, which we then equip with self.this = that and so on — effectively populating it with data before anyone else gets a chance to interact with it. So technically, __init__ is not really a constructor, in the sense that it doesn't construct an object, but an initializer (hence its name), preparing the object for first use. If we were to define a different method, like initialize, we'd get back an empty object:
>>> class A:
... def initialize(self, x):
... self.x = x
>>> a = A()
>>> a.x
AttributeError: 'A' object has no attribute 'x'
That is, until we call this method explicitly:
>>> a.initialize(1)
>>> a.x
1
So objects don't really have a well-defined structure or specific fields — they're more like a dictionary, mapping some names to some values.
Instance Attributes
This statement is true quite literally: objects have a special __dict__ attribute, which returns a dictionary of their attributes. In many sense, this dictionary is exactly the object's value, its state: this is what makes instances of some class different from each other:
>>> class A:
... def __init__(self, x):
... self.x = x
>>> a1 = A(1)
>>> a2 = A(2)
>>> a1.__dict__
{'x': 1}
>>> a2.__dict__
{'x': 2}
And much like we had with scopes, we can manipulate the attributes directly through this dictionary: if we change some key, the attribute changes; if we set a key, it becomes available; and if delete a key, it disappears:
>>> a1.__dict__['x'] = 3
>>> a1.x
3
>>> a1.__dict__['y'] = 4
>>> a1.y
4
>>> del a1.__dict__['y']
>>> a1.y
AttributeError: 'A' object has no attribute 'y'
This is the reason Python doesn't really have private attributes: all of the data is in __dict__, so whatever restriction mechanisms we can devise, it's always possible to bypass them:
>>> class A:
... def __init__(self):
... self._private = 1
... self.__very_private = 2
>>> a = A()
>>> a.__dict__
{'_private': 1, '_A__very_private': 2}
These values, managed in __dict__, are known as instance attributes — because they pertain to specific instances of the class; but they're not the only attributes an object can have.
Class Attributes
As we've already seen on several occasions, it's possible to define attributes on the class level, too. We do this by putting them directly in the class definition:
>>> class A:
... x = 1
This attribute is naturally available through the class object, but also through any one of its instances:
>>> A.x
1
>>> a = A()
>>> a.x
1
And that's despite not being in their __dict__:
>>> a.__dict__
{}
If we try to overwrite it through an object, it seems to work:
>>> a.x = 2
>>> a.x
2
But in fact, all we've done is add an instance attribute of the same name, which shadows that class attribute, and takes precedence in its order of resolution:
>>> a.__dict__
{'x': 2}
>>> A.x
1 # still the same
If deleted, the class attribute becomes available again:
>>> del a.x
>>> a.x
1
And it can't be deleted through an object, either:
>>> del a.x
AttributeError: 'A' object has no attribute 'x'
>>> # and yet...
>>> a.x
1
If you remember our discussion about scopes, this might look familiar. Just like in a scope, we can get a name that was either defined locally (in the object), or in an enclosing scope (in a class) — but also like in a scope, when we set or delete a name, it's an inherently local operation, hardwired to work on __dict__ without considering the class namespace.
In this case, though, we don't have special global or nonlocal statements to rewire access, because we don't need them — each class's namespace is available through its name. If we change it or delete it there:
>>> A.x = 2
>>> a.x
2
>>> del A.x
>>> a.x
AttributeError: 'A' object has no attribute 'x'
We see the changes reflected in the object as well, without even having to instantiate it; the lookup, it seems, happens on demand. The way that it works for classes is quite similar to instances, in that they have a __dict__ of their own:
>>> A.x = 1
>>> A.__dict__
mappingproxy({..., 'x': 1})
This, however, is not a standard dictionary, but a mapping proxy; which means that while we can read from it, we can't assign to or delete from it directly:
>>> A.__dict__['x']
1
>>> A.__dict__['x'] = 2
TypeError: 'mappingproxy' object does not support item assignment
>>> del A.__dict__['x']
TypeError: 'mappingproxy' object does not support item deletion
Not that this would be impossible — in Python 2, it was a plain dictionary, just like objects have. Mapping proxies were introduced as a read-only version of dict to save memory by sharing keys that recur among them; and since class dictionary don't change that often and share a lot of keys (namely, all the magic method names and other dunder-somethings), they were a prime candidate for this optimization. A side effect of this is that we can't set or delete class attribute by passing a string like 'x' to their __dict__, but this can still be done with the builtin setattr and delattr functions:
>>> setattr(A, 'x', 2)
>>> A.x
2
>>> delattr(A, 'x')
>>> a.x
AttributeError: type object 'A' has no attribute 'x'
MRO Strikes Again
Another similarity to name resolution is that lookup occurs through multiple enclosing scopes, when we define functions within functions within functions. With classes, we have:
>>> class A:
... x = 1
>>> class B(A):
... pass
>>> b = B()
>>> b.x
1
b is able to retrieve x despite the fact that it isn't in its __dict__, nor in its class's one:
>>> 'x' in b.__dict__
False
>>> 'x' in B.__dict__
False
And that's because attribute access actually traverses the inheritance tree. If we were trying to implement this lookup ourselves, we might rely on the fact that each class has a __bases__ attribute with a tuple of its base classes, doing some breadth-first search:
>>> def get_attribute(obj, key):
... if key in obj.__dict__:
... return obj.__dict__[key]
... classes = [obj.__class__]
... for cls in classes:
... if key in cls.__dict__:
... return cls.__dict__[key]
... classes.extend(cls.__bases__)
... raise AttributeError(key)
>>> get_attribute(b, 'x')
1
However, this algorithm wouldn't be accurate — because unlike scopes that are nested hierarchically, we saw that inheritance can take a more indirect route. Luckily, we also saw that this root is already precomputed in __mro__, so really, our lookup function can end up simpler, if anything:
>>> def get_attribute(obj, key):
... if key in obj.__dict__:
... return obj.__dict__[key]
... for cls in obj.__class__.__mro__:
... if key in cls.__dict__:
... return cls.__dict__[key]
... raise AttributeError(key)
>>> get_attribute(b, 'x')
1
While still accounting for complicated multiple inheritance and diamonds of death.
Dynamic Attributes
But there's one more piece to the puzzle: attributes can also be generated dynamically, using the magic methods __getattr__ (for access), __setattr__ (for assignment) and __delattr__ (for deletion); just to illustrate how they work:
>>> class A:
... def __getattr__(self, key):
... print('getting', key)
... def __setattr__(self, key, value):
... print('setting', key, 'to', value)
... def __delattr__(self, key):
... print('deleting', key)
>>> a = A()
>>> a.x
getting x
>>> a.x = 1
setting x to 1
>>> del a.x
deleting x
While our implementation doesn't do anything particularly useful, it's quite amazing that even this most basic flow of objects, dot notation itself, can be overloaded with custom logic. There's just one gotcha: while __setattr__ and __delattr__ successfully capture any attempt to set or delete an attribute, __getattr__ seems to take the back seat. If we have:
>>> class A:
... x = 'class'
... def __init__(self):
... self.x = 'instance'
... def __getattr__(self, key):
... return 'getattr'
>>> a = A()
What would you guess the attribute priority to be? See if you got it right:
>>> a.x
'instance'
>>> del a.x
>>> a.x
'class'
>>> del A.x
>>> a.x
'getattr'
In other words, __dict__ is checked first, all the classes' __dict__s are traversed after that, and only then is __getattr__ invoked — so what exactly is driving the behavior of getting an attribute?
Turns out, there's another magic method to consider: __getattribute__. This one actually trumps all the rest:
>>> class A:
... # same as before
... def __getattribute__(self, key):
... return 'getattribute'
>>> a = A()
>>> a.x
'getattribute'
The problem is, overloading it can easily render the object unusable:
>>> a.__dict__
'getattribute' # um
And that's because so much of the object's operation relies on being able to resolve its own magic attributes and methods, the overloading it is not something we'd want to do on a whim. Even if we'd try to do so responsibly, any dot notation we use inside __getattribute__ will invoke itself, leaving us with infinite recursion — so there's a pretty good reason for __getattr__'s low precedence, which lets us use "the rest of the object" while generating any additional attributes dynamically.
Implementing Attribute Access
Before we move on, let's try to implement attribute access ourselves, tricky though it may be — especially since it's about to get even trickier when we discuss descriptors. We already have some pretty good bones in get_attribute: all that's left is to figure out how to do it from inside __getattribute__, add support for __getattr__ and address a few other special cases.
For starters, we need to bypass the infinite recursion problem I mentioned; and to do that we can use super, delegating to our base class's (hopefully valid) attribute resolution. Let's just see that it works, first:
>>> class A:
... def __init__(self):
... self.x = 1
... def __getattribute__(self, key):
... x = super().__getattribute__('x')
... return x
>>> a = A()
>>> a.x
1
That's good; at the very least, we know how to not get stuck in a call loop. As an aside, this might look like a way to support private attributes in Python, denying access to e.g. any attribute starting that starts with an underscore:
>>> class A:
... def __init__(self):
... self.x = 1
... self._y = 2
... def __getattribute__(self, key):
... if key.startswith('_'):
... raise AttributeError(key)
... return super().__getattribute__(key)
>>> a = A()
>>> a.x
1
>>> a._y
AttributeError: _y
But like I said, there are no private attributes in Python, and with clever enough hacking I can invoke a superclass (like object) from outside the class, delegating to it in a similar way to fetch seemingly inaccessible attributes:
>>> object.__getattribute__(a, '_y')
2
Anyway, back to the task at hand: we have a way to access attributes from within __getattribute__ with super, and we have the core of the algorithm in get_attribute. All that's left is to account for __getattr__, so if the looking in the various __dict__s fails, we resort to calling it; and also address the few attributes we're accessing directly, so they'll never be in any __dict__ — namely, __dict__, __class__ and __getattribute__ themselves. We end up with:
>>> class A:
... def __getattribute__(self, key):
... __dict__ = super().__getattribute__('__dict__')
... if key == '__dict__':
... return __dict__
... if key in __dict__:
... return __dict__[key]
... __class__ = super().__getattribute__('__class__')
... if key == '__class__':
... return __class__
... for cls in __class__.__mro__:
... if key in cls.__dict__:
... return cls.__dict__[key]
... __getattr__ = super().__getattribute__('__getattr__')
... if key == '__getattr__':
... return __getattr__
... return __getattr__(key)
This is still not the complete picture, but it's good enough for now. As for __setattr__, it's quite straightforward:
>>> class A:
... # same as before
... def __setattr__(self, key, value):
... self.__dict__[key] = value
And __delattr__ would be straightforward to do, except for the error; if we simply try to delete self.__dict__[key], we'll get a KeyError if it doesn't exist, whereas attribute access should raise an AttributeError — so to account for that:
>>> class A:
... # same as before...
... def __delattr__(self, key):
... try:
... del self.__dict__[key]
... except KeyError:
... raise AttributeError(key)
And there we have it: a full, manual implementation of attribute access. At least, as full as we can do for now.
The Mystery of Methods
Consider the following code:
>>> class A:
... def f(self):
... print('f', self)
On the face of it, it's pretty simple: there's a callable, f, in A's __dict__:
>>> A.f
<function A.f at 0x...>
So when we access it through an instance, we expect the lookup to reach it eventually:
>>> a = A()
>>> a.f
<bound method A.f of <A object at 0x...>>
And while it does find some result — this is not the same function, but a bound method, whatever that means. So clearly, there's more to attribute access than we've discussed, at least when it comes to methods. If we keep investigating:
>>> a.f()
f <A object at 0x...>
We see the bound method words; as for the function itself:
>>> A.f()
TypeError: A.f() missing 1 required positional argument: 'self'
It doesn't, on account of missing the self argument. But how did the bound method not miss it? We can pass it in manually:
>>> A.f(a)
f <A object at 0x…>
In which case it seems to work. For a different function, however:
>>> def g(self):
... print('g', self)
>>> a.g = g
>>> a.g()
TypeError: g() missing 1 required positional argument: 'self'
This doesn't seem to work, and we have to pass self explicitly even through an object:
>>> a.g(a)
g <A object at 0x…>
Unless, that is, we define it as part of the class instead:
>>> del a.g
>>> A.g = g
>>> a.g()
g <A object at 0x…>
In which case it does work. In other words, when a function is defined inside a class and accessed through its objects, it seems to undergo a certain "specialization" — it's is bound, so to speak, to the instance through which it is accessed, and is somehow wired to pass it in implicitly as the first argument. However, if it's defined directly as part of an object, it doesn't seem to undergo the same change. So how does this work?
The Descriptor Concept
A very interesting piece of Python's attribute resolution method is about a special kind of objects, called descriptors. These objects define one of three special methods, not to be confused with the ones we discussed so far: __get__, __set__ and __delete__. The reason they're special, though, is because they customize not how the object behaves, but how it is viewed from a class. Consider, for example:
>>> class D:
... def __get__(self, instance, cls):
... print('getting through', instance, 'of class', cls.__name__)
... return 1
If we just create a D object and play around with it, there's no way for us to evoke this behavior other than calling it directly. But then:
>>> class A:
... d = D()
>>> a = A()
>>> a.d
accessed through <A object at 0x…> of class A
1
And what's more:
>>> A.d
accessed through None of class A
1
In other way, D knows — or rather, A knows, when accessing D — that it shouldn't return as-is, but invoke the behavior of being gotten, if you will; and to determine how to do it, it is passed the class and instance through which this is happening (or None, if it's happening directly through the class). That very same object, which we can fish out of A.__dict__ to avoid its transmutation, can be placed elsewhere:
>>> class B:
... d = A.__dict__['d']
>>> b = B()
>>> b.d
accessed through <B object at 0x...> of class B
1
And it's perfectly content to behave differently through that object. So if we had to support this in our __getattribute__, we'd need to augment our run through the MRO:
>>> class A:
... def __getattribute__(self, key):
... # same as before
... for cls in __class__.__mro__:
... if key in cls.__dict__:
... attr = cls.__dict__[key]
... if hasattr(attr, '__get__'):
... return attr.__get__(self, __class__)
... return attr
... # same as before
And also somehow account for attribute access through the class — but that we don't know how do just yet.
Implementing Methods
How does all of this relate to functions and methods? Well, it turns out that if we look into all the attributes of a function, which we can do with the builtin dir function:
>>> def f(x):
... print(x)
>>> dir(f)
[…, '__get__', …]
We can see __get__ hiding among them. This means that any function is, in fact, a descriptor — or at least, has the potential to become one when accessed through a class. To prove this, we can evoke its descriptor behavior explicitly:
>>> class A:
... pass
>>> a = A()
>>> af = f.__get__(a, A)
>>> af
<bound method f of <A object at 0x…>>
>>> af()
<A object at 0x...>
Lo and behold, we've created the same bound method in vitro — and calling it worked just as well, passing in the object to which we bound it as the first argument, x. This would've been the same as adding it to the class's namespace, because then its descriptor potential would have been discovered and realized automatically:
>>> A.f = f
>>> a.f()
<A object at 0x...>
>>> af == a.f
True
And I do mean the same — the bound methods even test equal.
But to really drive this point home, I'd like us to implement the method logic ourselves. Like the famous physicist Richard Feynman had said: "what I cannot create, I do not understand" — so in the same vein, let's pretend functions don't have this innate descriptor potential, and wrap them up in a class decorator that does:
>>> class Method:
... def __init__(self, function):
... self.function = function
... def __get__(self, instance, cls):
... if instance is None:
... return self.function
... return BoundMethod(self.function, instance)
This is something we can stick on top of a function definition, and it will get replaced with an object that will do our bidding when accessed through a class. That bit about instance being None is a pretty standard idiom, meaning that if our object is accessed through the class directly, we don't exhibit any special behavior, and usually just return ourselves (or in this case, the function we're pretending to be:
>>> class A:
... @Method
... def f(self):
... print(self)
>>> A.f
<function A.f at 0x...>
When the function is accessed through an object, however, we return a bound method in its stead — something that requires two ingredients: the function to call, and the method to bind it to, which will be passed in as its first argument. Its code is actually quite simple:
>>> class BoundMethod:
... def __init__(self, function, instance):
... self.function = function
... self.instance = instance
... def __call__(self, *args, **kwargs):
... return self.function(self.instance, *args, **kwargs)
That is, mimic the function's callable behavior, passing any invocation to it with near perfect-forwarding — but sneaking in the aforementioned instance as well. And it works:
>>> a = A()
>>> a.f()
<A object at 0x...>
Just like a regular function would — since, well, it's already a method in potentia.
The Descriptor Protocol
We mentioned, however, that the descriptor protocol is broader than just __get__: it also has __set__, which is used when a value is assigned to it through a class, and __delete__, which is used when it's being likewise deleted:
>>> class D:
... def __get__(self, instance, cls):
... print('getting through', instance, 'of class', cls.__name__)
... return 1
... def __set__(self, instance, value):
... print('setting through', instance, 'to', value)
... def __delete__(self, instance):
... print('deleting through', instance)
>>> class A:
... d = D()
>>> a = A()
>>> a.d
getting through <A object at 0x...> of class A
>>> a.d = 1
setting through <A object at 0x...> to 1
>>> del a.d
deleting through <A object at 0x...>
And notice how __set__ and __delete__ work only through objects; that is, A.d = 1 or del A.d would actually go ahead and replace the descriptor object in the class's namespace, or remove it entirely.
This means we have to augment our reimplementation of __setattr__ and __delattr__ to account for descriptors as well:
>>> class A:
... def __setattr__(self, key, value):
... for cls in self.__class__.__mro__:
... if key in cls.__dict__:
... attr = cls.__dict__[key]
... if hasattr(attr, '__set__'):
... attr.__set__(self, value)
... return
... self.__dict__[key] = value
... def __delattr__(self, key):
... for cls in self.__class__.__mro__:
... if key in cls.__dict__:
... attr = cls.__dict__[key]
... if hasattr(attr, '__delete__'):
... attr.__delete__(self)
... return
... try:
... del self.__dict__[key]
... except KeyError:
... raise AttributeError(key)
Descriptor State
So far, our descriptor implementation didn't really do anything when it was set or deleted; in real life, however, it should probably store and reset the assigned value, which presumably has some significance to its operation.
Doing that is easier said than done: we can't simply keep it in some attribute like self.value, since the descriptor object is defined once, as part of the class, and shared among its objects — which is why its magic methods explicitly receive the active one as an argument. We could, in principle, store a dictionary mapping object IDs to their corresponding values — but there already is a dedicated dictionary per object that we can use, namely its __dict__. So, the common way to tackle it is something like that:
>>> class D:
... def __init__(self, default):
... self.default = default
... def __get__(self, instance, cls):
... if instance is None:
... return self
... if 'd' not in instance.__dict__:
... instance.__dict__['d'] = self.default
... return instance.__dict__['d']
... def __set__(self, instance, value):
... instance.__dict__['d'] = value
... def __delete__(self, instance):
... instance.__dict__.pop('d', None)
So we start with some default value, which we put in each instance's __dict__ if we hadn't before — and when it's assigned or deleted, we apply the change there. that last bit, with pop(key, None), is called a soft delete: normally, del dict[key] would raise a KeyError if the key is missing from the dictionary, but here we are popping it out with None as a default value, so missing keys are silently ignored and multiple subsequent deletes are allowed. This method actually returns the key's value (or the default one), but we discard it, since we're only interested in the side-effect of it being removed. The reason we tolerate this behavior, by the way, is because this value can't really be deleted: the descriptor itself remains part of the class, so any future access will simply repopulate __dict__ with the default value; it's not really deletion we're implementing here, if we're honest, but a way to reset the descriptor to its initial state. Here it is in action:
>>> class A:
... d = D(1)
>>> a = A()
>>> a.d
1
>>> a.d = 2
>>> a.d
2
>>> del a.d
>>> a.d
1
Descriptor Names
Our previous implementation has a serious issue, though: it just assumes the descriptor will be called d, and uses the hardcoded string 'd' to store its state. If the descriptor is created under any other name, confusion will ensue:
>>> class B:
... e = D(1)
>>> b = B()
>>> b.e
1
>>> b.__dict__
{'d': 1}
>>> b.d
1
Luckily, Python provides a hook to coordinate this. Normally, an object has no way of knowing what name it's being assigned to (as in a = A()), or whether it's being used anonymously with no name at all (like print(A()). But since descriptors are stored as class attributes with a consistent name, it's a bit of a special case — so their __set_name__ method is called as soon as the class in which they are defined is evaluated, and receives that class and the name given to them within its context. Since it's not customary to reuse descriptor objects between classes, this is something they can actually store as part of their state — and assume that any access, assignment or deletion through objects will happen after this step, since we can't really create objects before we create their class. In conclusion, we can add:
>>> class D:
... # same as before
... def __set_name__(self, cls, name):
... self.name = name
And if we change __get__, __set__ and __delete__ to use self.name instead of 'd', everything should work just fine:
>>> class A:
... d = D(1)
>>> A.d.name # access through a class returns the descriptor object itself
'd'
>>> a = A()
>>> a.d
1
>>> a.__dict__
{'d': 1}
>>> class B:
... e = D(1)
>>> B.e.name
'e'
>>> b = B()
>>> b.e
1
>>> b.__dict__
{'e': 1}
Note, however, then when descriptors are added to a class after its definition, __set_name__ is not called — so if you're doing this for some reason, don't forget to account for that:
>>> class C:
... pass
>>> C.d = D(1)
>>> C.d.name
AttributeError: 'D' object has no attribute 'name'
Data and Non-Data Descriptors
As you go about your business, happily writing descriptors, you might notice something strange. Even if we remove __set__ and __delete__ from our previous implementation of D, it would still work:
>>> class D:
... # without __set__ and __delete__
>>> class A:
... d = D(1)
>>> a = A()
>>> a.d
1
>>> a.d = 2
>>> a.d
2
>>> del a.d
>>> a.d
1
The only clue that anything is different reveals itself if we try to delete it several times in a row:
>>> del a.d
AttributeError: 'A' object has no attribute 'd'
What happens is that, with only __get__ left, it gets called alright, which is why we get 1; but when we assign 2 to a.d, it goes straight to __dict__, effectively shadowing the descriptor — so when we access it, __dict__ takes precedence and returns 2. Not until we delete it, which also targets __dict__, does the descriptor become accessible again; but further deletions don't work, since neither it nor __dict__ can handle them at this point.
This called a non-data descriptor: one that only implements __get__, implying it doesn't keep any data per-object; so if data does become available as part of some object — namely, in its __dict__ — it is prioritized higher than the descriptor's opinion on what it should be.
If the descriptor implements __set__ and/or __delete__, then it becomes a data descriptor; and since it's now opinionated enough to care about the object's data as well, __dict__ is sidelined, and all the work is delegated to the descriptor.
The rationale is simple: descriptors that don't care about the object's state should be overwritable, while those that do should not. If we really want a descriptor that both takes priority, and is read-only, we can simply implement its __set__ and __delete__ to raise an exception if anyone tries to touch it.
Integrating this into our __getattribute__ is getting a bit ungainly, but closer still to the truth: first, we have to out if we're dealing with a data descriptor, then check __dict__, and finally check for non-data descriptors again:
>>> class A:
... def __getattribute__(self, key):
... is_desc = None
... __class__ = super().__getattribute__('__class__')
... if key == '__class__':
... return __class__
... for cls in __class__.__mro__:
... if key in cls.__dict__:
... attr = cls.__dict__[key]
... is_desc = hasattr(attr, '__get__')
... if is_desc and (hasattr(attr, '__set__') or
... hasattr(attr, '__delete__'):
... return attr.__get__(self, __class__)
... break
... __dict__ = super().__getattribute__('__dict__')
... if key == '__dict__':
... return __dict__
... if key in __dict__:
... return __dict__[key]
... if is_desc is not None:
... return attr.__get__(self, __class__) if is_desc else attr
... __getattr__ = super().__getattribute__('__getattr__')
... if key == '__getattr__':
... return __getattr__
... return __getattr__(key)
It looks like a lot, but hopefully it's pretty straightforward at this point. The only noteworthy trick in the code above is using is_desc for ternary boolean logic: it can be None, or False, or True. None indicates no matching class attribute was found at all, while the other two means it was found, and tells whether its a descriptor or not — either way, attr must be defined. We could, in principle, put attr = None and test for that, but None could be a valid value of some class attribute, so we'd have to resort to some special undefined object or auxiliary flags.
Properties
This is a pretty elaborate protocol; if it's only purpose would be to make methods possible, I'd argue that it's a bit of an overkill. But in fact, it unlocks a lot of really cool use cases — so let's go over them, and let's start with properties.
Some motivation, first. Suppose we have a class a class representing a two-dimensional Point:
>>> class Point:
... def __init__(self, x, y):
... self.x = x
... self.y = y
... def __repr__(self):
... return f'{self.__class__.__name__}({self.x!r}, {self.y!r})'
>>> p = Point(0, 1)
>>> p
Point(0, 1)
Today, our implementation uses cartesian coordinates, x and y; but tomorrow we decide to switch its implementation to an equally valid one, known as polar coordinates: r, for radius, and t, for theta, the angle at which we need to "shoot" from the origin over distance r in order to reach our desired point. This representation makes a bunch of mathematical operations much easier, so it makes sense; the problem is, it breaks our existing interface, and any code we might have written so far along with it:
>>> class Point:
... def __init__(self, r, t):
... self.r = r
... self.t = t
>>> p = Point(1, math.pi/2) # same point
>>> p.x
AttributeError: 'Point' object has no attribute 'x'
That's why in object-oriented programming languages like Java and C++, it's common wisdom never to expose any public attributes; doing so invariably makes them part of the object's interface, and prevents us from changing them in the future. The solution is to keep all attributes private, and provide dedicated methods to interact with them: accessors to get them, and mutators to modify them. In Python, it might look like this:
>>> class Point:
... def __init__(self, x, y):
... self._x = x
... self._y = y
... def get_x(self):
... return self._x
... def set_x(self, x):
... self._x = x
... def get_y(self):
... return self._y
... def set_y(self, y):
... self._y = y
It sure adds a lot of boilerplate code; but on the bright side, even if our implementation changes, we can edit these methods to adjust for this change, making the necessary conversions — like translating polar coordinates to cartesian ones:
>>> class Point:
... def __init__(self, r, t):
... self._r = r
... self._t = t
... def get_x(self):
... return self._r * math.cos(self._t)
... def set_x(self, x):
... y = self.get_y()
... self._r = math.hypot(x, y)
... self._t = math.atan(y / x)
... # similar for y
This way, code that uses our Point will feel no difference:
>>> p = Point(1, math.pi/2)
>>> p.get_x()
0
This is only possible because methods are dynamic; and even if their initial code is completely trivial, it's there as a placeholder — so that if need be, we get a chance to adapt. Attributes, which hold static values, can't be intercepted and accommodates like that — or can they?
Turns out, sometimes they can; in which case, they are called properties. The idea originated in the Delphi programming language, and was popularized by C#; but in Python, we don't need any special features to support this — the descriptor protocol we developed so far should suffice. We start with a couple public attributes, x and y, like before — and if at any point we need to switch them for r and t, we can still support x and y by defining them as methods and sticking a property decorator on top:
>>> class Point:
... def __init__(self, r, t):
... self.r = r
... self.t = t
... @property
... def x(self):
... return self.r * math.cos(self.t)
... # similar for y
This decoration, in turn, makes the method emulate an attribute:
>>> p = Point(1, math.pi/2)
>>> p.x
0.0
No invocation, no parentheses — and yet, the result is computed dynamically. Still, this is only half the story: we've successfully replaced accessotrs, but what about mutators? The syntax here is a little strange, but bear with me, we'll implement it soon enough:
>>> class Point:
... # same as before
... @property
... def x(self):
... return self.r * math.cos(self.t)
... @x.setter
... def x(self, x):
... self.r = math.hypot(x, self.y)
... self.t = math.atan(self.y / x)
... # similar for y
>>> p = Point(1, math.pi/2)
>>> p.x = 2
>>> p.x
2 # assignment worked
>>> p.r, p.t
(2.236, 0.464) # changed as a result
Implementing Accessors
It looks funky the first time you see it — but we're well prepared to understand what exactly is going on, and even write its code ourselves. property is a class decorator, replacing our method with a descriptor object — which, upon access, invokes the method it's wrapped around and returns the result. The code is exceedingly simple:
>>> class Property:
... def __init__(self, function):
... self.function = function
... def __get__(self, instance, cls):
... if instance is None:
... return self
... return self.function(instance)
And while it's not the full pictures, it gives us an alternative to accessors already. We can stick @Property on top of a method and compute x and y from r and t like in the example above; but just to entertain a different scenario, we can also have:
>>> class User:
... def __init__(self, first_name, last_name):
... self.first_name = first_name
... self.last_name = last_name
... @Property
... def full_name(self):
... return f'{self.first_name} {self.last_name}'
That is, have a derived property, based on some other attributes. We could store it as a regular attribute, of course — but then if either the first or last name are changed, it needs to be changed as well. Here, being derived dynamically, it updates on its own:
>>> user = User('jane', 'doe')
>>> user.full_name
jane doe
>>> user.first_name = 'alice'
>>> user.full_name
'alice doe'
Of course, if we set full name to something else, Python wouldn't know to update the name parts; and since the Property we implemented is a non-data descriptor, it would end up in __dict__ and fall out of sync with the rest of our object. The builtin property decorator does something more clever, which we can illustrate with a simpler example:
>>> class A:
... @property
... def p(self):
... return 1
>>> a = A()
>>> a.p
1
>>> a.p = 2
AttributeError: property 'p' of 'A' object has no setter
So property it's a data descriptor, which takes precedence and denies our assignment, making p a read-only property by default.
Implementing Mutators
But it's more than that: since the original method is replaced with a property object, that object comes with methods of its own, which let us alleviate this restriction:
>>> A.p
<property object at 0x...>
>>> A.p.setter
<function property.setter>
OK, let's give it a go:
>>> def set_p(self, p):
... print('setting p to', p)
>>> A.p.setter(set_p)
<property object at 0x000001E6649E5A80>
>>> a.p = 2
AttributeError: property 'p' of 'A' object has no setter
Strangely enough, though, the method returns a property object — and worse, doesn't seem to take effect. The reason is that it's actually a decorator as well, so we're expected to replace the original p with its upgraded result:
>>> A.p = A.p.setter(set_p)
>>> a.p = 2
setting p to 2
So now it works. This design decision is not so peculiar once we realize it is meant to be used inside a class definition:
>>> class A:
... @property
... def p(self):
... return 1
... @p.setter
... def set_p(self, p):
... print('setting p to', p)
>>> a = A()
>>> a.p
1
>>> a.p = 2
AttributeError: property 'p' of 'A' object has no setter
Except, not it doesn't work; the reason this time is that under those circumstances, we have to keep the name of the property exactly the same. What we've done is effectively replace the method p with a property, and then replace the method set_p with its enhanced version, leaving us with:
>>> a.set_p = 2
setting p to 2
Which is not what we had in mind. So, one last try:
>>> class A:
... @property
... def p(self):
... return 1
... @p.setter
... def p(self, value):
... print('setting p to', value)
>>> a = A()
>>> a.p
1
>>> a.p = 2
setting p to 2
Which is the same as the code we've seen for x, but hopefully now we understand why. The same could be applied to full_name:
>>> class User:
... # same as before
... @full_name.setter
... def full_name(self, full_name):
... self.first_name, self.last_name = full_name.split(' ', 1)
>>> user = User('jane', 'doe')
>>> user.full_name = 'alice doe'
>>> user.first_name
'alice'
The property object also provides a deleter decorator, which similarly registers a method (again with the same name) to be invoked when the property is being deleted. If we were to add all this functionality to our own implementation, we'd end up with:
>>> class Property:
... def __init__(self, get, set=None, delete=None):
... self._get = get
... self._set = set
... self._delete = delete
... def __set_name__(self, cls, name):
... self.name = name
... def __get__(self, instance, cls):
... if instance is None:
... return self
... return self._get(instance)
... def __set__(self, instance, value):
... if not self._set:
... raise AttributeError(f'cannot set attribute {self.name}')
... self._set(instance, value)
... def __delete__(self, instance):
... if not self._delete:
... raise AttributeError(f'cannot delete attribute {self.name}')
... self._delete(instance)
... def setter(self, set):
... return self.__class__(self._get, set, self._delete)
... def deleter(self, delete):
... return self.__class__(self._get, self._set, delete)
And notice how setter and deleter are registration decorators, which we haven't seen before: they don't so much decorate the function as capture it for later use, returning enhanced Property object instead. At this point, we can do:
>>> class A:
... @Property
... def p(self):
... return 1
... @p.setter
... def p(self, value):
... print('setting p to', value)
... @p.deleter
... def p(self):
... print('deleting p')
>>> a = A()
>>> a.p
1
>>> a.p = 2
setting p to 2
>>> del a.p
deleting p
And it really wasn't all the much code for a pretty sophisticated functionality.
Static Methods
Another feature from other object-oriented languages we might want to emulate in Python is that of static methods: ones that are not specific per object, but rather belong in the class scope. Going back to our Point class, suppose each point can compute its distance from another:
>>> class Point:
... # same as before
... def distance(self, other):
... return ((self.x - other.x)**2 + (self.y - other.y)**2)**0.5
This works well for pairs of points — but if we want to sort a list based on each point's distance from the origin, we'd have to do something like this:
>>> points = [Point(3, 0), Point(1, 0), Point(0, 2)]
>>> origin = Point(0, 0)
>>> points.sort(key=lambda p: p.distance(origin))
>>> points
[Point(1, 0), Point(0, 2), Point(3, 0)]
If this functionality is common enough, we might want to add it to the Point class:
>>> class Point:
... # same as before
... def sort(self, points):
... origin = Point(0, 0)
... points.sort(key=lambda p: p.distance(origin))
... return points
But it's not exactly a method, is it? It doesn't really act on the object whose behavior it supposed to represent, but is rather a helper function that operates on something else. It's not particularly good design; and even when it is related to our object, as in:
>>> points[0].sort(points)
[Point(1, 0), Point(0, 2), Point(3, 0)]
This will crash if the list is empty. We could, in principle, put it in a separate function — and that would be OK, except keeping it within the class's scope can be tidier still. What if we remove the self from sort, and use it as a simple nested class function?
>>> class Point:
... # same as before
... def sort(points):
... # same as before
>>> Point.sort(points)
[Point(1, 0), Point(0, 2), Point(3, 0)]
This seems to work; but now, if we call it through a method, we get an error:
>>> points[0].sort(points)
TypeError: Point.sort() takes 1 positional arguments but 2 were given
Which is hardly surprising, since it becomes a bound method, attempts to sneak in self as the first argument, and ends up with an extra argument on hand. We could decide, in principle, to never call it through a point object — but it stays part of its interface, which is pretty inelegant.
Another type of static methods we might run into is factory methods, which let us construct new objects of some class in a particular way. Normally, we'd use the constructor for that — but consider what happens if we want to support instantiating points from either cartesian or polar coordinates, regardless of the implementation we're using (suppose it's the former). If we were to squeeze both cases into a single __init__, we'd end up with:
>>> class Point:
... def __init__(self, x=None, y=None, r=None, t=None):
... cartesian = x is not None and y is not None
... polar = r is not None and t is not None
... if cartesian and not polar:
... self.x = x
... self.y = y
... elif polar and not cartesian:
... self.x = r * math.cos(t)
... self.y = r * math.sin(t)
... elif polar and cartesian:
... raise ValueError('provide x and y or r and t, but not both')
... else:
... raise ValueError('provide either x and y, or r and t')
Which works, but is rather horrible. Instead, we can do:
>>> class Point:
... def __init__(self, x, y):
... self.x = x
... self.y = y
... def from_cartesian(x, y):
... return Point(x, y)
... def from_polar(r, t):
... return Point(r * math.cos(t), r * math.sin(t))
And as long as we call these methods through the class, they work:
>>> p1 = Point.from_cartesian(0, 1)
>>> p1
Point(0, 1)
>>> p2 = Point.from_polar(1, math.pi/2)
>>> p2
Point(0, 1)
However, should we call it through an object for whatever reason, it will crash like before. So what do we do?
Implementing Static Methods
Basically, we're looking to "disable" the descriptor potential of a function, so it doesn't automatically bind to the object through which it is called — and Python lets us do just that with the builtin staticmethod decorator:
>>> class Point:
... # same as before
... @staticmethod
... def sort(points):
... # same as before
... @staticmethod
... def from_cartesian(x, y):
... # same as before
... @staticmethod
... def from_polar(r, t):
... # same as before
>>> p1 = Point.from_cartesian(1, 0) # works through a class
>>> p2 = p1.from_polar(2, math.pi/2) # works through an object
>>> Point.sort([p2, p1]) # works through a class
[Point(1, 0), Point(0, 2)]
>>> p1.sort([p2, p1]) # works through an object
[Point(1, 0), Point(0, 2)]
Implementing it ourselves is almost trivial, since we don't actually need to do anything — rather, make sure that nothing gets done when accessing a function through a class:
>>> class StaticMethod:
... def __init__(self, f):
... self.function = function
... def __get__(self, instance, cls):
... return self.function
However, this is still not ideal; not our implementation, which works fine, but the whole idea of a static method. If you look back at Point's code, we've been hardcoding its class name in our functions — which makes sense, since they're so tightly related to it that we've decided to put them inside it. At the same time, if we ever subclass Point to add, say, a color to it — then trying to create a ColorPoint through any of the factory methods would result in the wrong type:
>>> class ColorPoint(Point):
... def __init__(self, x, y, color):
... super().__init__(x, y)
... self.color = color
>>> ColorPoint(0, 1, color='red')
ColorPoint(0, 1)
>>> ColorPoint.from_polar(1, math.pi/2)
Point(0, 1)
Now, you might say I've cheating a bit, since I can't pass a color to the factory methods regardless — if we to inherit them in a way that really works, we'd have to reimplement them anyway, if only to adjust their signatures. But that's not necessarily true — good factory method implementation can account for subclasses, accepting extra keywords that they simply pass along, like so:
>>> class Point:
... # same as before
... @staticmethod
... def from_polar(r, t, **kwargs):
... return Point(r * math.cos(t), r * math.sin(t), **kwargs)
Now the point is a bit more evident:
>>> class ColorPoint(Point):
... def __init__(self, x, y, color):
... super().__init__(x, y)
... self.color = color
>>> ColorPoint(0, 1, color='red')
ColorPoint(0, 1)
>>> ColorPoint.from_polar(1, math.pi/2, color='red')
TypeError: Point.__init__() got an unexpected keyword argument 'color'
Class Methods
If only there was a way for us to pass in the dynamic class — maybe the same way as we pass in the dynamic object in methods, as an implicit first argument. And indeed, Python provides us with such functionality through the builtin classmethod decorator, whose first (and implicit) argument is indeed the class (conventionally named cls):
>>> class Point:
... # same as before
... @classmethod
... def from_polar(cls, r, t, **kwargs):
... return cls(r * math.cos(t), r * math.sin(t), **kwargs)
>>> ColorPoint.from_polar(1, math.pi/2, color='red')
ColorPoint(0, 1)
Not only that, but class methods work just as well as static ones through an object; so really, it's them that we needed all along. Other languages simply didn't have the concept of a class object, one that can be passed in as an argument — so there was no choice but to hardcode it inside. But in Python, it's quite easily doable: we need an implementation that's halfway between Method and StaticMethod, in that it does bind the function — but to the class rather than its instance:
>>> class ClassMethod:
... def __init__(self, function):
... self.function = function
... def __get__(self, instance, cls):
... return BoundClassMethod(self.function, cls)
>>> class BoundClassMethod:
... def __init__(self, function, cls):
... self.function = function
... self.cls = cls
... def __call__(self, *args, **kwargs):
... return self.function(self.cls, *args, **kwargs)
And if I may be blunt, this is always better than a static method; in fact, we could do away with the builtin staticmethod entirely, and I can't think of a single use case that wouldn't be more appropriately done with classmethod.
To be fair, this is particularly evident in factory methods; they are designed to instantiate a new object, so it's clear how they would benefit from a dynamic class being provided. What about our sort method, though? It, too, hardcodes the Point class needlessly, if only to create an origin point — so really:
>>> class Point:
... # same as before
... @classmethod
... def sort(cls, points):
... origin = cls(0, 0)
... points.sort(key=lambda p: p.distance(origin))
... return points
Would be better. But, you might argue, this is almost a coincidence! What if distance would receive x and y, not a point object, so all we'd need is (0, 0) and no cls. In that case, I'd say that class methods are still better, because a way to implement it that is nicer still would be to store such an origin as a class attribute, available via cls:
>>> class Point:
... origin = 0, 0
... # same as before
... @classmethod
... def sort(cls, points):
... points.sort(key=lambda p: p.distance(cls.origin))
... return points
You see, my logic is as follows: either a function is somehow related to a class, and we want to keep it there — in which case, it will only benefit from access to the class in which it is kept; or, it truly has no need for this extra argument — which raises the question, why is it part of that class to begin with? If it neither instantiates the class, nor accesses its attributes, nor calls other (class) methods, is there really a reason not to define it separately of it?
As an aside, it's a nice idea to keep the origin as a separate attribute — it might be useful in other cases, too. But if we want it to be a point like before, we're in a pickle: instantiating it as part of the class definition is impossible, since the class itself is still in the process of being defined; and while assigning it afterwards is doable, it's feels quite hacky. Now, there's no builtin decorator for a class property, but we could easily make one ourselves:
>>> class ClassProperty:
... def __init__(self, function):
... self.function = function
... def __get__(self, instance, cls):
... return self.function(cls)
>>> class Point:
... @ClassProperty
... def origin(cls):
... return cls(0, 0)
In which case, accessing it would invoke the descriptor code and give back a result, indistinguishable from a static value, just like we had before:
>>> Point.origin
Point(0, 0)
However, it would also create a new object on every access — which isn't very efficient, and can lead to odd situations like this:
>>> Point.origin is Point.origin
False
Cached Properties
Here's a different idea: we change the descriptor code to replace itself with the value on the first access, when its computed:
>>> class CachedClassProperty:
... def __init__(self, function):
... self.function = function
... def __set_name__(self, cls, name):
... self.name = name
... def __get__(self, instance, cls):
... value = self.function(cls)
... setattr(cls, self.name, value)
... return value
And now, we really do have a fixed origin point, created in the moment we touch it first:
>>> class Point:
... @CachedClassProperty
... def origin(cls):
... return cls(0, 0)
>>> Point.__dict__
{..., 'origin': <CachedClassProperty object at 0x...>}
>>> Point.origin
Point(0, 0)
>>> Point.__dict__
{..., 'origin': Point(0, 0)}
This is actually a great idea for regular properties as well. In some cases, like our derived full_name, we'd like the property to stay nimble — both because it's relatively quick to concatenate a couple of strings, and because we'd like updates to propagate, so if the first or last name changes, it is recomputated accordingly. However, in some cases a heavy calculation might be involved, or an otherwise slow operation like reading a file — in which case we can store the result in the object's __dict__, and use it hence without having to re-run the property's code:
>>> class CachedProperty:
... def __init__(self, function):
... self.function = function
... def __set_name__(self, cls, name):
... self.name = name
... def __get__(self, instance, cls):
... if instance is None:
... return self
... value = self.function(instance)
... instance.__dict__[self.name] = value
... return value
At which point, we can do:
>>> class File:
... def __init__(self, path):
... self.path = path
... @CachedProperty
... def data(self):
... return open(self.path).read()
>>> open('hello.txt', 'w').write('hello world')
>>> f = File('hello.txt')
>>> f.data # reads file
'hello world'
>>> f.data # instant
'hello world'
Oh course, you might reasonable argue that a file's data can also change, and you'd be right; there's a tradeoff here. If it's a small file that changes frequently, we might want to stick to a standard property; but if it's large or doesn't change often, caching the result might be worth it. And in any case, we can always reset it — invalidating the cache, so to speak — by simply deleting the attribute from the __dict__, which will the property code to run again on its next access:
>>> open('hello.txt', 'w').write('something else')
>>> f.data # still the same
'hello world'
>>> del f.data
>>> f.data
'something else'
In fact, this functionality is useful enough that in Python 3.8, it was added to the standard library: we no longer need to implement it ourselves (or install a 3rd party library); simply from functools import cached_property and decorate ahead.
Objects 4: Conclusion
So yeah, methods and attributes. This chapter was a bit longer than usual, but we covered a lot of ground: from instance to class and superclass attributes, ordered according to the MRO algorithm, to how all functions have the potential to become a method and just do it on demand. We've implemented a pretty well-rounded dunder-getattribute, accounting for recursion, for dynamic attributes via dunder-getattr, and for the entire descriptor protocol, including data and non-data descriptors. We saw how it works not just for methods, but also properties, which let us have public and derived attributes; and also for class methods, which let us operate on class objects, static methods that remain simple functions, and cached properties that store and reuse their results. In most other languages, accessing an attribute or calling a method is just a matter of calculating an offset or going through a virtual table, so I just love how much thought went into the design of this process here. It's not a simple abstraction, but it's an extremely clever one: imitating scopes, inverting control, having all sorts of convenient hooks; and it lets us implement basic concepts like methods and properties using the same tools that govern all other behaviors, and which we also get to use, as equals. I guess what feels so fundamental to me about understanding this architecture is all of us use attributes and methods all the time, but unless we know how it works, we probably wouldn't have guessed. So understanding it gives us this profound appreciation for the intricacy of our tools, and hopefully some motivation and insight that can help us achieve better mastery.