Writing/The Unbearable Lightness of Coding
Chapter 11 · 43 min read
Objects 2: Inheritance
One important aspect of object-oriented programming that we haven't covered yet is inheritance. I'll be honest: I don't like it, and with a few minor exceptions, I don't recommend using it — but it's still an important part of the language, so we'll do well to understand how it works (and how it doesn't). We'll cover the syntax, see its superficial appeal, and discuss how it's used in software design; but we'll also address the various problems it causes, why it doesn't live up to Python's potential, and what alternative solutions can be applied instead.
***
The idea behind inheritance is quite simple: a class can be extended by another class, which then inherits all its definitions, and can add to them or override them as it sees fit. The inherited class is called the superclass or base class, and the inheriting one is called the subclass or derived class. All we have to do to establish this connection is to mention the superclass next to the subclass's name when we define it:
>>> class A: # superclass
... def inc(self, x):
... return x + 1
>>> class B(A): # subclass
... def add(self, x, y):
... return x + y
And suddenly, objects of class B have both the functionality from B and from A:
>>> b = B()
>>> b.add(1, 2)
3
>>> b.inc(1)
2
The same is not true in the opposite direction: objects of class A have the functionality from A only, and have no idea that they were further extended in B:
>>> a = A()
>>> a.inc(1)
2
>>> a.add(1, 2)
AttributeError: 'A' object has no attribute 'add'
This relationship of an object having particular functionality is often described as "is-a": b is-a B and is-a(n) A, while a is-a A (and isn't-a B). In Python, you can test for it with the builtin isinstance (and issubclass) function:
>>> isinstance(a, A)
True
>>> isinstance(a, B)
False
>>> isinstance(b, B)
True
>>> isinstance(b, A)
True
# and that's because:
>>> issubclass(B, A)
True
# but not:
>>> issubclass(A, B)
False
Extending Classes
Putting our laconic example aside, the reason this might be seem useful is that it lets us define some basic functionality in the superclass, and then hone or augment it in more specialized subclasses, all of which get to share the same base attributes and methods without having to repeat them; in other words, it lets us define a hierarchy of functionality. Suppose, for example, that we're writing a very realistic computer game, in which we'd like to simulate dogs and cats. If you have a dog, you might know that some of its most prominent features is barking and pooping — so we might end up with:
>>> class Dog:
... def bark(self):
... print('woof')
... def poop(self):
... print('💩')
And if you have a cat, you might know that some of its most prominent features is meowing and pooping — so to that, we might add:
>>> class Cat:
... def meow(self):
... print('meow')
... def poop(self):
... print('💩')
I don't know about you, but I smell a redundancy, which can be ameliorated if we hoist the duplicate code into its own class, like Animal, and make Dog and Cat inherit from it:
>>> class Animal:
... def poop(self):
... print('💩')
>>> class Dog(Animal):
... def bark(self):
... print('woof')
>>> class Cat(Animal):
... def meow(self):
... print('meow')
And it makes sense, doesn't it? Both dogs and cats are-a(n) animal — so by organizing them in such a taxonomy, we are able to tease out the generic, shared functionality into a superclass, and have each subclass focus on its own particularities in a clean, localized manner.
Augmenting Classes
Sometimes, though, we don't want to simply add new methods to a subclass, but change the way a superclass does something. If we need to add eating functionality, we might put it in the Animal class, since it's pretty generic:
>>> class Animal:
... # same as before
... def eat(self, food):
... print('slurp', food)
But while an abstract animal might eat anything, actual cats and dogs are more picky. Let's say that both of them like meat, but only a dog would eat a carrot, and only a cat would eat catnip. If we were to throw an exception for inapproriate food, we'd have to implement a slightly different version of eat for each class:
>>> class Dog(Animal):
... # same as before
... def eat(self, food):
... if food not in ('meat', 'carrot'):
... raise ValueError('dog refused to eat ' + food)
... print('slurp', food)
>>> class Cat(Animal):
... # same as before
... def eat(self, food):
... if food not in ('meat', 'catnip'):
... raise ValueError('cat refused to eat ' + food)
... print('slurp', food)
Which means we'd override the superclass's eat method entirely, and end up duplicating its code in each subclass. Printing "slurp" is not too much work — but anything more meaningful would be hard to maintain in multiple copies.
Luckily, there's a way to tap into the superclass's functionality from inside a subclass, even while overriding it: the builtin super function. It returns a proxy to the superclass, so with its help we can add our particular checks and still delegate the generic work "upwards", to Animal:
>>> class Dog(Animal):
... # same as before
... def eat(self, food):
... if food not in ['meat', 'carrot']:
... raise ValueError('dog refused to eat ' + food)
... super().eat(food)
>>> class Cat(Animal):
... # same as before
... def eat(self, food):
... if food not in ['meat', 'catnip']:
... raise ValueError('cat refused to eat ' + food)
... super().eat(food)
It's a bit of an odd create, super. In other languages it's more of a keyword than a function call, because it only works from within methods, and is somehow able to figure out from its context what class it's being called from, and who's its parent. In Python 2, we had to provide it with clues explicitly (e.g. calling super(Dog, self)) so it can orient itself accordingly; only in Python 3, using some clever trickery, they managed to emulate the same contextual, keyword-like behavior with a clean super(). But that aside, its gist should be pretty clear — "give me the superclass, I want to call its method".
Filling In Classes
Still, Cat and Dog share some eerily similar code; it feels like we could avoid this duplication entirely, if only we could generalize the base eat method to account for food preferences, and use the subclasses to specify what those are.
One way to do this is to realize that Animal is actually an abstract class: we're not going to initialize an object of this type, so it doesn't actually need an eat method at all. If all we care about is consolidating repetitive code, a private helper method will do:
>>> class Animal(Animal):
... # same as before
... def _eat(self, food, preferences):
... if food not in preferences:
... raise ValueError('animal refused to eat ' + food)
... print('slurp', food)
In which case, our concrete classes can fill in the gaps:
>>> class Dog(Animal):
... # same as before
... def eat(self, food):
... self._eat(food, ('meat', 'carrot'))
>>> class Cat(Animal):
... # same as before
... def eat(self, food):
... self._eat(food, ('meat', 'catnip'))
Alternatively, we can do it the other way around: put the eat method in Animal, and make it rely on a _get_food_preferences() helper, to be overridden later with more particular code:
>>> class Animal:
... # same as before
... def eat(self, food):
... if food not in self._get_food_preferences():
... raise ValueError('animal refused to eat ' + food)
... print('slurp', food)
... def _get_food_preferences(self):
... raise NotImplementedError()
Note that now our class really is abstract; if we were to instantiate an animal and attempt to feed it, we would get an error explicitly telling us that _get_food_preferences is not implemented — Python even has a dedicated exception for that. Now we can do:
>>> class Dog(Animal):
... # same as before
... def _get_food_preferences(self):
... return 'meat', 'carrot'
>>> class Cat(Animal):
... # same as before
... def _get_food_preferences(self):
... return 'meat', 'catnip'
Which is much cleaner, if you ask me. In fact, since both cats and dogs like meat, we can remove even that duplication, putting any common foods in Animal and extending that list with super:
>>> class Animal:
... # same as before
... def _get_food_preferences(self):
... return ['meat']
>>> class Dog(Animal):
... # same as before
... def _get_food_preferences(self):
... return super()._get_food_preferences() + ['carrot']
>>> class Cat(Animal):
... # same as before
... def _get_food_preferences(self):
... return super()._get_food_preferences() + ['catnip']
Although here, Animal loses its abstractedness; we suddenly can, in principle, instantiate a generic animal that for some reason only eats meat, whatever that means. Still, it's a useful enough practice to warrant its own name: it's called templating, because the superclass effectively provides a template that the subclasses then fill in, whether it means implementing something unimplemented, or overriding some default implementation with a more specific one.
Subclass Constructors
Another common augmentation is overriding constructors. Both cats and dogs can have a name, right? — so it makes sense to capture this common logic in Animal:
>>> class Animal:
... def __init__(self, name):
... self.name = name
... # same as before
However, we might need to initialize either cats or dogs or both with additional information — so how do we do that? Dogs, for example, might get a list of commands they're familiar with, like "sit" or "fetch", while cats famously don't care what you want them to do. On the other hand, cats might get a purring threshold, indicating how easy or hard it is to get them into this delightful state that dogs unfortunately can't achieve:
>>> class Dog(Animal):
... def __init__(self, name, commands):
... self.name = name
... self.commands = commands
... # same as before
>>> class Cat(Animal):
... def __init__(self, name, purring_threshold):
... self.name = name
... self.purring_threshold = purring_threshold
... # same as before
Once again, we end up with some unfortunately code duplication. We could have a helper method like before (self._set_name(name) might feel a bit superficial, but like I said, this is a silly example); however, the common practice here is to use super, delegating common construction to whatever ancestors our inheritance tree might have:
>>> class Dog(Animal):
... def __init__(self, name, commands):
... super().__init__(name)
... self.commands = commands
... # same as before
>>> class Cat(Animal):
... def __init__(self, name, purring_threshold):
... super().__init__(name)
... self.purring_threshold = purring_threshold
... # same as before
Calling __init__ so explicitly is a bit jarring, but that's what we have. Arguably, a subclass should always do this, because it can't really know what exactly is happening in its superclasses; even if we decide that not all animals have a name, there might be some parameterless shared functionality up there, like assigning a unique ID to each:
>>> class Animal:
... id_counter = itertools.count()
... def __init__(self):
... self.id = next(self.id_counter)
... # same as before
In which case we'd want to do this:
>>> class Dog(Animal):
... def __init__(self, name, commands):
... super().__init__()
... self.name = name
... self.commands = commands
... # same as before
>>> class Cat(Animal):
... def __init__(self, name, purring_threshold):
... super().__init__()
... self.name = name
... self.purring_threshold = purring_threshold
... # same as before
And note that this is completely harmless even if Animal has no __init__ at all, because all classes implicitly inherit from object; so worst case scenario, its constructor (which doesn't do anything) will be invoked in vain. The alternative, if we ever do decide to add a constructor to Animal, is to go around updating all our subclasses accordingly, so we might as well do this in advance.
In fact, we could use this constructor for our benefit, even if we end up dropping this name idea altogether (what if we need to support ants someday? Will we really name each one?). The _get_food_preferences helper, after all, is not really a function — it's more of a list, isn't it, that Animal could be initialized with. So we can do:
>>> class Animal:
... def __init__(self, food_preferences):
... self.food_preferences = food_preferences
... # same as before
... def eat(self, food):
... if food not in self.food_preferences:
... raise ValueError('animal refused to eat ' + food)
... print('slurp', food)
In which case each subclass (including ants) can set it accordingly:
>>> class Dog(Animal):
... def __init__(self, commands):
... super().__init__(('meat', 'carrot'))
... self.commands = commands
... # same as before
>>> class Cat(Animal):
... def __init__(self, purring_threshold):
... super().__init__(('meat', 'catnip'))
... self.purring_threshold = purring_threshold
This idea is called a private constructor, since are Animal class is still abstract and shouldn't be instantiated directly; the parameters of its __init__ are used to calibrate the common functionality it provides to its subclasses, and them deferring to it with super().__init__ is a convenient hook to do so.
Name Mangling
This presents a different problem, though. Like I said, each subclass is not actually expected to know the implementation details of its entire inheritance tree: it might know that the common eat functionality is provided, and that it can be customized by passing the food preferences to its parent's constructor; but how exactly that parent stores or manages this information — whether it's food_preferences or _foodprefs — is really Animal's business.
In such a case, a subclass might accidentally override an important attribute of its superclass, thus compromising its own function. When we want a dog to sit or fetch, for example, we might need to train it with some snacks — in which case it might respond more or less enthusiastically to different treats, which we could store in some weighted dictionary like {'chicken': 0.4, 'duck': 0.6, 'bull_penis': 1.0}. What happens, though, if we absentmindedly decide to keep it in an attribute we call food_preferences, overriding Animal's attribute with the same name? Consider an even simpler case:
>>> class A:
... def __init__(self):
... self._x = 1
... def f(self):
... print('f', self._x)
>>> class B(A):
... def __init__(self):
... super().__init__()
... self._x = 2
... def g(self):
... print('g', self._x)
Ideally, we'd like a B object to print 1 in f and 2 in g, but this is not what happens:
>>> b = B()
>>> b.f()
f 2
>>> b.g()
g 2
And that's despite the fact that we stored it in a private attribute, _x, just like we're supposed to do with such implementation details.
To mitigate this, Python supports name mangling: if we just add another underscore to the name, it suddenly works:
>>> class A:
... def __init__(self):
... self.__x = 1
... def f(self):
... print('f', self.__x)
>>> class B(A):
... def __init__(self):
... super().__init__()
... self.__x = 2
... def g(self):
... print('g', self.__x)
>>> b = B()
>>> b.f()
f 1
>>> b.g()
g 2
And I know what you're thinking: so Python does have private attributes, after all! I mean, if we were to check out what __x is in b:
>>> b.__x
AttributeError: 'B' object has no attribute '__x'
It sure seems like it's disappeared, and is only made available to the methods within a particular class. This is not what happens, however; like this feature's name might imply, all it does is mangle the names of attributes starting with two underscores, prepending them with a unique enough prefix of _<class-name>:
>>> b._A__x
1 # That's A's __x
>>> b._B__x
2 # That's B's __x
Similarly, if Animal would store the food preferences in __food_preferences, it would end up as _Animal__food_preferences, while a similar attribute in Dog would end up as _Dog__food_preferences, thus avoiding a name clash. Python is nice enough to substitute this ungainly clutter for us, so we just add __ and don't worry about uniqueness — but it's important to realize that there's no actual privacy involved.
Multiple Inheritance
So far, our inheritance relationships could be organized in a tree: a superclass might have had multiple subclasses, but each subclass had a single superclass. This is the case in most languages that support inheritance, and for good reason — a problem known, rather dramatically, as the deadly diamond of death.
Suppose A is extended by B and by C, and then D inherits from both B and C. What would that mean? If there's a method in A that both B and C override, but D doesn't, then which version of it does it get? It's a conundrum alright — and if you plot these relationships visually, it looks a diamond (well, a rhombus), hence the name.
For that reason, multiple inheritance is generally frowned upon. C++ allows it, but refuses to compile code with such diamond diagrams, while other languages, like Java and C#, outright forbid it. The reasoning is always the same: like goto, it's a "harmful" feature, allowing developers to shoot themselves in the foot. It shouldn't come as a surprise, then, that with Python's opposite and empowering attitude it was decided to support it — and even solve the diamond hurdle at that, because really, it's not such a deadly problem of death after all. Here's the resolution:
>>> class A:
... def f(self):
... print('A')
>>> class B(A):
... def f(self):
... print('B')
>>> class C(A):
... def f(self):
... print('C')
>>> class D(C, B):
... pass
>>> d = D()
>>> d.f()
C
But why C, you might ask — why not B? It makes sense that it's not A, since D inherits from two of its subclasses, and thus has two more specific versions to choose from, but how do we decide which is more "valid"?
The answer is simple: C comes first. We can just as easily have:
>>> class D(B, C):
... pass
>>> d = D()
>>> d.f()
B
In which case B comes first, so its version takes precedence.
Problem solved; the only thing left to explain, I guess, is why would we even need multiple inheritance. And just like with infinite loops or goto, the answer is simple: we do not. Anything that's possible with it is possible without it; but in some cases, it might make sense.
Now, suppose that our game would support mythical creatures as well, so after adding Horse, Bird and Lion, we'd also want to support Pegasus and Griffin. Well, we can conceive of a pegasus as a combination of a horse and a bird; but since both are animals, we have a diamond, and wouldn't be allowed to merge their shared functionality in such a way under other circumstances. A griffin is even more complex: it has the body of a lion and the head and wings of an eagle — so if Lion inherits from Feline (it should share some common ancestor with Cat, after all), and Eagle is a subclass of Bird, we end up with an inheritance diagram that's a full blown hexagon. Python, however, handles it just fine; all we have to choose is whether a griffin is more lion than eagle or more eagle than lion: both Griffin(Eagle, Lion) and Griffin(Lion, Eagle) work, but with a slightly different resolution order.
Method Resolution Order
To be fair, Python's resolution order is a little bit more complex. In most cases you can think of it as a breadth-first search up the inheritance tree: check all the direct superclasses in the order they appear; than go up one level (in the same order) and check all the superclasses' superclasses in the order they appear, and so on. In practice, though, it's a bit more nuanced, and has some surprises in store.
The algorithm Python uses to flatten out all these inheritance shapes and doodles is called the method resolution order, or MRO, and you can always find it in the __mro__ attribute of a given class:
>>> class D(C, B):
... pass
>>> D.__mro__
(<class 'D'>, <class 'C'>, <class 'B'>, <class 'A'>, <class 'object'>)
>>> class D(B, C):
... pass
>>> D.__mro__
(<class 'D'>, <class 'B'>, <class 'C'>, <class 'A'>, <class 'object'>)
And notice how, like we've already mentioned, all classes implicitly inherit from object; that's where they get their default behaviors like __repr__ from.
In some sense, that's all we need to know, really: Python figures out some order for us, and we can always check what exactly it is, and act accordingly; how exactly it does so is beyond our scope of interest.
What's interesting, though, is how super follows this order to the dot. If you're coming from other languages, you might expect it to always go up; even its name sort of implies that we're delegating to the superclass functionality. However, given Python's elaborate support for multiple inheritance, a more correct way to think about it would be as something that returns the next class in the MRO. Consider this, for example:
>>> class A:
... def f(self):
... print('A')
>>> class B(A):
... def f(self):
... print('B')
... super().f()
>>> class C(A):
... def f(self):
... print('C')
... super().f()
>>> class D(C, B):
... def f(self):
... print('D')
... super().f()
What will happen when we call D's f()? It prints D, sure, and goes to C next — but what would C's super().f() do? If we were to create a C object:
>>> c = C()
>>> c.f()
C
A
It goes straight to A, like we might expect. But in D's flow, it actually goes sideways:
>>> d = D()
>>> d.f()
D
C
B
A
That is, jumping from C to B, and only then to their common ancestor, A. Trying to wrap your head around it from super's perspective can be confusing — but if we take a look at its MRO again:
>>> D.__mro__
(<class 'D'>, <class 'C'>, <class 'B'>, <class 'A'>, <class 'object'>)
Then we see exactly the same order: D, then C, then B, then A. This can get even more confusing if we remove C's connection to A completely:
>>> class C: # no A
... def f(self):
... print('C')
... super().f() # ???
>>> class D(C, B):
... def f(self):
... print('D')
... super().f()
Now, super's perspective makes no sense at all: C doesn't even have a superclass to go to! And yet:
>>> d = D()
>>> d.f()
D
C
B
A
It's somehow included in the sequence just right, and knows to jump on to B despite the fact that they're not actually related. The reason is as simple as it's anticlimactic — that's just the MRO Python comes up with:
>>> D.__mro__
(<class 'D'>, <class 'C'>, <class 'B'>, <class 'A'>, <class 'object'>)
If this isn't weird enough, consider what happens if we switch D's inheritance order:
>>> class D(B, C):
... def f(self):
... print('D')
... super().f()
>>> d = D()
>>> d.f()
D
B
A
This time, C is skipped entirely. Why? Well, you could read the MRO algorithm's implementation and understand its intricacies — or, you could just check out its result:
>>> D.__mro__
(<class 'D'>, <class 'B'>, <class 'A'>, <class 'C'>, <class 'object'>)
As it happens, Python decides that since C is secondary to B and not related to A, D must is more of an A than it is a C, so C comes last; and since we didn't put a super().f() in A, the chain stops there — you can go ahead and add it, for all the sense it makes, and see that the chain will continue.
So like I said, it's not really a BFS (otherwise, C would have come before A); but it's pretty close, and we can always check what exactly it is easily enough, and in most reasonable cases it will end up just like we expect. That's not to say MRO is infallible: we can trip it up if we juxtapose reversed diamond diagrams together:
>>> class A:
... pass
>>> class B:
... pass
>>> class C(A, B):
... pass
>>> class D(B, A):
... pass
>>> class E(C, D):
... pass
TypeError: Cannot create a consistent method resolution order (MRO) for bases A, B
But then, I would argue that this is an unreasonable case.
Mixins
No doubt, multiple inheritance can be confusing — which is why it's not a good idea to use it casually, when we're not sure how exactly the inheritance relations will interweave. There is, however, a perfectly safe and sensical case where we can leverage its power, and that's when the various classes we are inheriting from don't share any common ancestry whatsoever.
In this case, we're effectively mixing in different behaviors, and blending them all in a single class that presumably makes use of this diverse combination. One class might provide some advanced analysis methods, while another provides functionality to save files in a particular format, and yet another has a set of handy functions to log the execution to our screen. In this case, having something along the lines of:
>>> class Processor(Analyzer, Saver, Logger):
... def process(self, data):
... self.log('analyzing', data) # from Logger
... result = self.analyze(data) # from Analyzer
... self.log('saving', result)
... self.save(result) # from Saver
Is completely valid. Some libraries even expose their interfaces this way, as a class to be "mixed in" and utilized by a subclass — in which case we would call it a mixin class.
Inheritance Squabbles
Inheritance is an powerful feature, and it unlocks interesting horizons of possibilities — but like I said in the beginning, I don't like it. You might have developed a similar intuition throughout this chapter, although I did my honest best to do it justice and present clever ways in which it can be used.
Still, something about it just isn't… right. It might seem neat, at first; in the example above, we've implemented a complex processor that analyzes and saves data, logging its execution, in just a few lines of code. The alternative, doing so with separate objects:
>>> class Processor:
... def __init__(self, analyzer, saver, logger):
... self.analyzer = analyzer
... self.saver = saver
... self.logger = logger
... def process(self, data):
... self.logger.log('analyzing', data)
... result = self.analyzer.analyze(data)
... self.logger.log('saving', result)
... self.saver.save(result)
Feels needlessly verbose and clunky. And if we go back to our computer game, then:
>>> class Pegasus(Horse, Bird):
... pass
Packs a pretty powerful punch — two lines and we have a whole new entity.
But consider how such a codebase will look after a while. Unless our taxonomy is extremely sensical and well organized, we will end up with haphazard classes of varying degrees of abstraction scattered all over the place, bound to each other by impalpable ties of latent side effects in implicit order.
Forget about spaghetti code — this code is basically cellophane noodles. I mean, that Pegasus class is sure fun to write, but how would reading it feel? It holds no clue as to what it is or what it does, so we have to look for the Horse and Bird classes, which probably won't give us the full picture either. And as we jump between files and climb up the inheritance tree into increasingly abstract and general-purpose classes, we'll have to retain all this information in our head, and reconcile all its interdependencies on the fly.
But that's not all. Suppose our game also includes some robots:
>>> class Robot:
... def drive(self):
... print('vroom')
>>> class CleaningRobot(Robot):
... def clean(self):
... print('🧻')
>>> class MurderRobot(Robot):
... def kill(self):
... print('☠️')
And then our company gets sold to Boston Dynamics, who naturally want us to implement a MurderDogRobot. With single inheritance, it's not entirely clear how to combine the code above with our previous tree:
>>> class Animal:
... def poop(self):
... print('💩')
>>> class Dog(Animal):
... def bark(self):
... print('woof')
Should we inherit from MurderRobot and Robot, or from Dog and Animal? We need the kill method from the first, the drive method from the second, the bark method from the third, and nothing from the fourth; if we introduce a common ancestor and attempt to reorganize the code to minimize duplication, the best I could come up with was this:
>>> class Object:
... def bark(self):
... print('woof')
>>> class Animal(Object):
... def poop(self):
... print('💩')
>>> class Dog(Animal):
... pass
>>> class Robot(Object):
... def drive(self):
... print('vroom')
>>> class MurderRobot(Robot):
... def kill(self):
... print('☠️')
>>> class MurderDogRobot(MurderRobot):
... pass
Which is of course absurd, since all our animals (and all our robots, for that matter) end up barking. With multiple inheritance, things are somewhat easier:
>>> class MurderDogRobot(MurderRobot, Dog):
... pass
Except our robot ends up pooping, which is not ideal either. It might be a silly example, but don't write it off on that account: we'll see in one from real-life in a moment, but I'd like to outline the common theme in all these inheritance-related issues first.
First, inheritance encourages us to think in terms of classes, subclasses and subsubclasses — so every task starts looking like a potential (sub*)class, leading to a phenomenon known as class explosion. Then, inheritance creates the impression that code can always be shared, so any duplication can be hoisted and abstracted if necessary — leading to code fragmentation, where functionality is spread out across files instead of compartmentalized in cohesive modules. And lastly, to leverage its advantages, it forces us to commit to a taxonomy that usually becomes increasingly difficult to extend and justify, as things naturally evolve in unforeseen ways — a form of premature optimization on a level of software design.
All these issues stem from the same faulty reasoning — namely, shortsightedness. If you're just writing some short, one-time script then by all means, write code however you like to get the job done; but in developing a long-term project we have to account for change. Using a language feature like inheritance, which might feel like an appealing shortcut at first glance, often reveals itself as a fallacy upon deeper reflection: is a pegasus really a horse and a bird? It doesn't squeak, lay eggs, or does any other bird-like things, except flying — so it's more of a flying horse, really; and if we'd like to reuse our flying functionality, then maybe it's more related to wings than to birds, and we're better off implementing it as a separate, independent component that can then be equipped on one animal or another. And once we start thinking in terms of different systems and how they combine and interact, it's a very different attitude to objects than attempting to articulate what they are, philosophizing whether this is-a that, or trying to predict some organizing principle that binds all possible futures together in a nice knot.
Real-Life Example
So far, I've been trying to avoid unnecessary complexity by relying on creative examples, but I think it's time for a story from the real world. To parse files of various formats, the following base class was designed:
>>> class Reader:
... def __init__(self, path):
... self.path = path
... def __repr__(self):
... return 'Reader(%r)' % self.path
... def read(self):
... self._fp = open(self.path, 'rb')
... while not self._eof():
... yield self._next_chunk()
... def _eof(self):
... return self._fp.tell() == self.size - 1
... def _next_chunk(self):
... raise NotImplementedError()
The idea is as follows: the reader takes care of opening the path (and does so in binary mode, since it's more generic — you can still read text files this way, while text mode doesn't let you read binary). Then it reads the file lazily, to support large files, loading one chunk after another, until it detects the end of file, or EOF (when the file cursor is as far as its size allows).
The initial formats it was meant for were, unsurprisingly, large binary files: exactly the kind that we parse piecewise, by reading a few bytes to determine the length of the next segment, then reading that many bytes to load it — and so on, until we run out of file. This way, all one had to do to support a new format was implement _next_chunk using an already open file in self._fp, and one would get a full-blown, lazy and standardized parser, with a nice representation to boot.
It worked well enough — until a JSONReader was needed, which was supposed to read either a list or a dictionary and lazily yield its items. To determine that, it received the expected document type (either 'list' or 'dict'):
>>> class JSONReader(Reader):
... def __init__(self, path, type):
... super().__init__(path)
... self.type = type
And already we had an issue, because our nice representation string turned out wrong:
>>> jr = JSONReader('data.json', 'list')
>>> jr
Reader('data.json')
It doesn't specify the document type we're reading, and it doesn't even get the class name right; but alright, it's a minor detail. We can override the __repr__ ourselves:
>>> class JSONReader:
... # same as before
... def __repr__(self):
... return 'JSONReader(%r, %r)' % (self.path, self.type)
The next problem was more substantial: parsing JSON files is tricky, and they weren't so big to begin with, so using Python's builtin json module made more sense than to implement it from scratch. However, the way it works is by reading the entire document at once — so to fit our iterative chunk-by-chunk logic, we had to cheat a little:
>>> class JSONReader(Reader):
... def __init__(self, path, type):
... # same as before
... self._iter = None
... def _next_chunk(self):
... if self._iter is None:
... self._iter = self._load_iter()
... try:
... return next(self._iter)
... except StopIteration:
... pass
... def _load_iter(self):
... document = json.load(self._fp)
... if self.type == 'dict':
... return iter(document.items())
... else:
... return iter(document)
In other words, load the file as an iterator the first time _next_chunk is called, and then keep calling next on it until it raises StopIteration, effectively implementing a for-loop manually.
Now there were two problems: json.load(self._fp) didn't work, because back then it didn't support files open in binary mode — so we had to change it to:
>>> class JSONReader(Reader):
... # same as before
... def _load_iter(self):
... document = json.loads(self._fp.read().decode())
... # same as before
Effectively negating the value of opening the file in advance, since we ended up reading and decoding all of it at once anyway. And once this happened, the file cursor jumped all the way to the end, so _eof returned True and stopped the loop despite the fact that we had more than one item to yield. OK, so:
>>> class JSONReader(Reader):
... def __init__(self, path, type):
... # same as before
... self._ended = False
... def _next_chunk(self):
... if self._iter is None:
... self._iter = self._load_iter()
... try:
... return next(self._iter)
... except StopIteration:
... self._ended = True
... # same as before
... def _eof(self):
... return self._ended
In other words, we had to override _eof to use a special indicator, which we'd set ourselves when our iteration was over — effectively negating the value of the parent's clever while-loop as well. Not only that, but the current implementation actually yields an extra None at the end, because StopIteration is raised and _ended is set only after the last, futile _next_chunk is already called — so we had to either implement some lookahead in JSONReader, or go back to Reader and change its code to account for that:
>>> class Reader:
... # same as before
... def read(self):
... self._fp = open(self.path, 'rb')
... while not self._eof():
... chunk = self._next_chunk()
... if not chunk:
... break
... yield chunk
... # same as before
All of this headache, where the following code would have sufficed:
>>> def read_json_file(path, type):
... data = json.load(open(path))
... yield from data.items() if type == 'dict' else data
Unfortunately, the Reader was embedded into a larger framework, and provided additional functionality that I've omitted — so we were stuck with making it work. Anyway, think what happened here: tackling the problem with inheritance initially looked like a smart idea, capable of abstracting a lot of the work away. If you have some experience in software design, you might have had some intuition that it's a bad abstraction — but did you expect the code to end up looking as bad as it did? Instead of just solving the problem in two lines, we ended up reimplementing __init__, __repr__ and _eof to get a clumsy version of _next_chunk to work, then going back to the base class and amending its read method. Basically, 100% of the functions mentioned needed to be changed — and we got 0% of the benefits: no easy hook, no lazy loading, not even a measly representation string.
The Truth Behind Inheritence
At this point, you might ask yourself why inheritance is even a thing. Lots of language features can save time; what sets apart the great ones from the rest is just how sustainable they are — how they pan out in the long run, how their good influence permeates. In this sense, as we've seen, inheritance has a major weakness: it requires us to come up with an idea of how things are going to work very early on — and we're probably going to make mistakes, since predicting the future is notoriously hard. The problem is that once we've committed to a certain way of doing things, its gets increasingly difficult to disentangle ourselves from these self-imposed constraints; so how come this practice is so widely taught, and nary a word is said in most textbooks about its dangers?
The reason, it turns out, has nothing to do with Python — which is why I don't think inheritance fits well within its overall abstraction. You see, languages like Java or C++ are statically typed: values must declare whether they're a number, a string or an object in order for the code to compile. This design decision made sense at the time, and still has some benefits — but it also significantly complicates that most important interface in software design, the function. As we've seen, generalizing some code and customizing its execution through parameters is an absolutely essential practice to address complexity; but if these parameters have to be of a particular, narrow type, it severely limits our ability to do this well. In our computer game, for example, we had cats and dogs who could eat some food and then poop it out. In a statically typed language, that would mean defining two functions:
void feedDog(Dog& dog) {
dog.eat('meat');
sleep(1);
dog.poop();
}
void feedCat(Cat& cat) {
cat.eat('meat');
sleep(1);
cat.poop();
}
However, this is a prime opportunity for generalization: both behaviors have nothing to do with their dogness or catness, merely their animalness. In Python, it's not a problem:
>>> def feed(animal):
... animal.eat('meat')
... time.sleep(1)
... animal.poop()
>>> feed(Dog())
slurp meat
💩
>>> feed(Cat())
slurp meet
💩
But in C++, there's no way for a function to receive either a cat or a dog, either that type or another. So to counter this behavior, a shared superclass could be used:
void feed(Animal& animal) {
animal.eat('meat');
sleep(1);
animal.poop();
}
Since a cat is-an animal and a dog is-an animal, this function would accept both; and since both eat and poop are defined accordingly inside the Animal class, the compiler would successfully ascertain that calling these methods makes sense. In other words, as long as the declared type in the function's signature is a superclass of the actual type of the argument passed in, the compiler will accept code that adheres to the superclass's interface, common to all its subclasses. In Python, there's no restriction of this sort — we can write, for example:
>>> def door_rings(animal):
... animal.bark()
>>> def open_fridge(animal):
... animal.meow()
And both functions will accept both types of object — they'll just crash when we attempt to make a cat bark, or a dog meow. In C++, such code would simply not compile, because neither bark nor meow are part of an Animal object. To address this, though, we could declare a virtual method with a predictable signature:
class Animal {
public:
virtual void speak();
};
And notice it has no implementation, making Animal an abstract class that can't be directly instantiated, since it's effectively missing some pieces to work. A subclass, though, may fill in the gaps:
class Dog : public Animal {
public:
void speak() {
printf("woof\n");
}
};
class Cat : public Animal {
public:
void speak() {
printf("meow\n");
}
}
And a function that accepts an animal and makes it speak would make sense: the compiler will be appeased, since we've consolidated both behaviors into a similar interface, declared in Animal — and yet each object will behave differently, according to its own implementation. This is known as polymorphism: writing code that expects something that can come in many (poly-) forms (morph), and resolving the particularities of each form dynamically, while still being able to validate its general shape in advance.
So in statically typed languages, inheritance is absolutely crucial: is is the only way to write polymorphic code — which is to say, generic, reusable code — in a type safe way. Doing the same in C, for that matter, would be impossible; we can't make one struct subclass another, so we'd have to compose them in clever ways, cast them into different types and rely on low-level considerations of how structures are laid out in memory, effectively duping the compiler and bypassing the language type safety mechanisms entirely.
That's what inheritance really is for: all the discussion about consolidating or templating code is really fluff that was conceived later, when people tried to figure out the best ways to make do with what they have. Python, on the other hand, is a dynamically typed language: its objects have types only insofar that they have different behaviors, but the language doesn't discriminate them or enforce this in any way. This is often referred to as duck typing: if something walks like a duck and quacks like a duck, it might as well be a duck, so we can treat it as one; and if for some reason it really isn't, we'll get a runtime error and deal with it then.
ABC
That's not to say that a similar type safety can't be emulated in Python. The standard abc module lets us defined an abstract class, in which we declare some abstract (or virtual) methods:
>>> import abc
>>> class Animal(abc.ABC):
... @abc.abstractmethod
... def speak(self):
... pass
And then any class that attempts to subclass it but doesn't implement the necessary details will fail to instantiate:
>>> class Dog(Animal):
... pass
>>> dog = Dog()
TypeError: Can't instantiate abstract class Dog with abstract method speak
This way, any method that expects to receive an animal can rest assured that any such object will have a speak method, conforming to Animal's interface. This doesn't mean we can't pass in a method of a different type — but then, with type annotations:
>>> def excite(animal: Animal):
... animal.speak()
We can use a 3rd party library like mypy to make sure this doesn't happen.
Composition Over Inheirance
In my personal opinion, though, we shouldn't do it. Unless we're writing some super-abstract, ultra-generic framework, and we want to guarantee that inexperienced programmers who don't read documentation and aren't well-versed in debugging don't shoot themselves in the foot — this just imposes unnecessary restrictions, and goes completely against Python's philosophy that we are all consenting adults.
I think that the main reason it even exists, let alone as part of the standard library, is that people coming from other languages often times feel uncomfortable over this paradigm shift, and look for ways to make their code feel more familiar and "professional".
This is also the reason, I think, that inheritance is still a big topic, even in Python; but really, there's a much better alternative to solving nearly all the problems we tackled along the chapter: composition. Its gist is trivial: rather than organizing our components in a rigid taxonomy, let's simply compose them in various ways — Python is, after all, extremely flexible and dynamic, so this is excessively easy to do. Instead of placing complicated flying code in Bird, and then having to make Pegasus a subclass of it to access this functionality, we can encapsulate it in a separate Wings object, and simply have:
>>> class Bird:
... def __init__(self):
... self.wings = Wings()
... def fly(self):
... self.wings.fly()
>>> class Pegasus:
... def __init__(self):
... self.wings = Wings()
... def fly(self):
... self.wings.fly()
Yeah, it's a bit more verbose — but it's straightforward code, clearly indicating where our functionality resides; and when we go there, all of it is localized in a single, cohesive unit. Even if it, in turn, relies on further composition, the interfaces between the different parts of the system can be clearly defined and delineated, instead of bleeding into each other like they do with inheritance. We don't automatically all the superclass's implicit functionality — we have to explicitly call the composite to get it to do its thing; and we aren't tempted to go "up" so as to consolidate some generic functionality in a place where it doesn't really fit — this modular, systemic approach prompts us to consider where it actually belongs, and use that.
In the case of the MurderDogRobot, for example, we might similarly realize that driving is not inherent to robots, but rather to wheels — and equip it with a Wheels object if necessary (besides, don't dog robots have legs?); and killing is a function of some Weapon, of which there can be several, so it's probably a good idea not to bake it into the implementation, but pass it in somehow and have it use it. As for barking, it feels a bit strange to define a Barker class, doesn't it? But this might lead us to decide it's better to transpose this functionality: rather than scattering it among the different Animal subclasses, where it's a bit of an eclectic consideration, we can consolidate all of it in a class that answers the question "what sounds do different animals make", solve this problem there, and have the various species delegate this concern to this component. This, in turn, can lead to better tests (double-checking each animal makes the correct sound without hardcoding its string there):
>>> def test_dog_speak(dog, animal_sounds):
... assert dog.speak() == animal_sounds.dog_sound()
Easily localize it depending on language:
>>> animal_sounds.localize('ES')
>>> dog.speak()
'guau'
And other not-necessarily-predictable considerations, which we've not isolated in a dedicated and cohesive module.
As for our reader example, it might make us realize that the functionality of reading a file and parsing its content is actually two different tasks — so while the reader can address the first one, it might very well delegate the second to another object:
>>> class Reader:
... def __init__(self, path, parser):
... self.path = path
... self.parser = parser
... def read(self):
... self._fp = open(self.path, 'rb')
... yield from self.parser.parse(self._fp)
In which case, writing a JSONParser would be as easy as:
>>> class JSONParser:
... def __init__(self, type):
... self.type = type
... def parse(self, fp):
... data = json.loads(fp.read().decode())
... return iter(data.items()) if self.type == 'dict' else iter(data)
It's still not perfect — like, we have to read the file and decode it instead of simply opening it in text mode and loading it with json.load — but I never said composition is a silver bullet; different components will always have to adjust to each other, and as long as it's relatively straightforward, it's OK. And if you feel like I cheated here, changing the while-loop just to get my point across, consider how the code would look otherwise:
>>> class Reader:
... # same as before
... def read(self):
... self._fp = open(self.path, 'rb')
... while not self.parser.eof(self._fp):
... yield self.parser.next_chunk(self._fp)
It's possible to go down this road, sure, but it's also more obviously wrong: since parser is a separate component, in charge of yielding out chunks of data, why do we have to iterate over it ourselves, micromanaging it with guiding questions rather than just letting it do its thing? The responsibilities are no longer as diffuse as they were with inheritance, because we weren't implementing a template for its algorithm — so without these worries hanging over our heads, we might be more inclined to opt for the best interface, and leave it to the parser to figure out how to conform to it.
# TODO: always better, send me counterexamples if you find them.
A Few Exceptions
There are a few exceptions, though, where inheritance is actually needed, that I'll be remiss not to mention. When we're implementing an exception class — a topic that we'll cover later — we can't raise any old object:
>>> class CustomError:
... pass
>>> raise CustomError()
TypeError: exceptions must derive from BaseException
So we have to subclass some existing exception class, usually Exception:
>>> class CustomError(Exception):
... pass
>>> raise CustomError()
CustomError
But honestly, this is largely a cosmetic change. We can also subclass native types, which is pretty cool:
>>> class ForgivingDict(dict):
... def __getitem__(self, key):
... return self.get(key, key)
>>> d = ForgivingDict({'x': 1})
>>> d['x'] # key exists
1
>>> d['y'] # key doesn't exist, but still works
'y'
Which can be useful in some circumstances, and saves us from reimplementing the entire functionality of a dictionary. This can be useful in various metaprogramming scenarios, where Python expects to receive a dictionary and we'd like to modify the way it works — but we'll talk about it in due time, like when we get to metaclasses. Generally, in the realm of metaprogramming, relying on inheritance to connect classes in some clever ways can be pretty handy — but these are not your usual consideration, and not usually what comes to mind when the topic is discussed.
Last but not least, inheritance might work better with IDE code completion and other IntelliSense features; if you're adamant about your type annotations, you probably have a penchant for statically typed languages, and as long as your taxonomies are not overly elaborate, inheritance might end up looking cleaner than composition, where many more things are passed around as arguments and require re-annotation as a result. We'll revisit this consideration when we talk about type hinting, but if nothing else, I hope that I've convinced you to be more careful about how you approach this feature, rather than taking for granted that it's the right way to do things.