Design Patterns: the Gang of Four, and Whether We Still Need Them

A Pattern Language

In the introduction, I complained that most of what's written about software design is a tedious deliberation about the same old design patterns, from some thirty years ago. It's time to make good on this complaint — and, as always, to be fair about it: to actually go over the old patterns, and see what they were worth, and whether they still are.

The story starts in 1977, when the architect Christopher Alexander and his colleagues published A Pattern Language: Towns, Buildings, Construction — a catalog of 253 patterns. Each describes a problem that recurs in the built environment, proposes a solution, and discusses the consequences; from entire regions down to window seats, and since the patterns reference each other, it becomes a language that can be used to design a town or a house.

In 1994, Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides, now and forever known as the Gang of Four, borrowed this format for Design Patterns: Elements of Reusable Object-Oriented Software. In it, they described 23 patterns for object-oriented code — each with an intent, a solution, and its consequences, and divided into three families (creational, structural and behavioral) — with examples in C++ and Smalltalk. It became the classical way to teach software design; and to this day, if you go to a job interview, you'll probably be asked about singletons and factories.

The format itself is not bad: David Beazley's Python Cookbook, also a catalog of problems and their solutions, is absolutely wonderful. The problem I have with it is that the patterns are taught prescriptively, as if they are the law, and you should follow them to the letter — if this is your problem, then this is its solution. I find this tedious, confusing, and paternalistic; it teaches the solution, not the method of arriving at one — and if there's one thing that's certain, it's that reality changes, and solutions have to change with it.

I once had a student who objected to this notion, saying that design patterns are valuable, because all the lovely infrastructure we have today was built with them. And it's true; but it's also true that horses were valuable, too — we rode them everywhere, until we invented something better, and mostly stopped. The point is not whether something was valuable in the past, but whether it's still valuable now.

A better way to study them, which I borrow from Brandon Rhodes (his talk on Python design patterns, and his website, python-patterns.guide, are highly recommended), is to first solve the problem in the obvious way, in Python — and then impose constraints on the solution, like "what if functions couldn't be passed as arguments?" or "what if everything had to be statically typed?", and see how it can be solved anyway. Pile enough constraints on top of each other, and eventually you'll arrive at the original pattern — at which point it should become pretty evident just how much of it was a clever workaround for the languages of the time.

So this chapter is not a rant, where I go over one design pattern after another and shoot them down. Some of them were useful, and ceased to be; some of them are still useful, at least in some cases and with some modifications; some of them describe a concept that's always relevant, even if its implementation is not; and one of them, the template method, is incredibly prevalent — and, in my opinion, incredibly terrible.

We'll go over five classical patterns — strategy, abstract factory, singleton, factory method and template method — as well as revisit inheritance and composition; we'll go on a tour of all the patterns that Python's features made invisible; and then we'll have a few "patterns" of my own — although they're less about solutions, and more about platforms: drivers, middleware and aspects with plugins.

Which is why this chapter is so far into the book. To weigh a pattern honestly, we'd need all the tools we developed over the course of the book — first-class functions, decorators, generators, descriptors, metaclasses and modules — so it's better to have them handy, rather than re-teach them. As such, I'll keep referring back to the chapters where we first encountered them.

Strategy

Every pattern in the book comes with an intent, written in a weird, legalese kind of language. For strategy, it is "define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it". It's a pretty good idea, and a pretty straightforward one at that; it just sounds much more complicated than it actually is.

So, let's say we have a function to sort some items, but we don't want to hardcode how exactly they should be compared. Instead, we receive a comparator object, whose compare method returns a positive number if the first item is larger, a negative one if the second is, and 0 if they're equal. We can define a base class to lay out this interface, and implement various strategies as its subclasses:

>>> class Comparator:
...     def compare(self, s1, s2):
...         raise NotImplementedError()
>>> class Alphabetically(Comparator):
...     def compare(self, s1, s2):
...         if s1 > s2:
...             return +1
...         if s1 < s2:
...             return -1
...         return 0
>>> class ByLength(Comparator):
...     def compare(self, s1, s2):
...         if len(s1) > len(s2):
...             return +1
...         if len(s1) < len(s2):
...             return -1
...         return 0
>>> def sort(items, comparator):
...     results = []
...     for item in items:
...         for i, anchor in enumerate(results):
...             if comparator.compare(item, anchor) < 0:
...                 results.insert(i, item)
...                 break
...         else:
...             results.append(item)
...     return results
>>> sort(['alice', 'charlie', 'bob'], Alphabetically())
['alice', 'bob', 'charlie']
>>> sort(['alice', 'charlie', 'bob'], ByLength())
['bob', 'alice', 'charlie']

Our sort is a deliberately primitive implementation of an insertion sort. It goes over every item in the list, and looks for the first item in the results that should come after it, so it can be inserted before it; or, if it doesn't find one, the else clause (remember those, from the Loops chapter?) kicks in, and it gets appended to the end. The point, though, is that we get the same names, alphabetized or by length, depending on the strategy object we pass in.

In Python, however, half of this is ceremony. If you recall our functions chapter, we mentioned that functions are themselves objects; so we don't need a Comparator class, and we don't need to subclass it to implement various strategies, and we don't need to give them all a compare method — we can just pass in the function, and for a one-off, a lambda will do:

>>> def sort(items, compare):
...     results = []
...     for item in items:
...         for i, anchor in enumerate(results):
...             if compare(item, anchor) < 0:
...                 results.insert(i, item)
...                 break
...         else:
...             results.append(item)
...     return results
>>> def by_length(s1, s2):
...     return len(s1) - len(s2)
>>> sort(['alice', 'charlie', 'bob'], by_length)
['bob', 'alice', 'charlie']
>>> sort(['alice', 'charlie', 'bob'], lambda s1, s2: len(s2) - len(s1))
['charlie', 'alice', 'bob']

As an aside, by_length is so simple that I just subtracted the lengths, since negative, zero or positive was all that the contract required; and the lambda simply flips them to sort in descending order. But it turns out that Python's own sorted function (and the list.sort method) solved this problem in an even more elegant way: instead of a function that compares two items, it accepts a function that reduces each item to a key, by which they are then sorted. The max function, and a few other ones, accept a similar argument, so it's worth familiarizing ourselves with this idiom:

>>> names = ['alice', 'Charlie', 'bob']
>>> sorted(names)
['Charlie', 'alice', 'bob']
>>> sorted(names, key=str.lower)
['alice', 'bob', 'Charlie']
>>> sorted(names, key=len)
['bob', 'alice', 'Charlie']
>>> max(names, key=len)
'Charlie'
>>> by_length = lambda s1, s2: len(s1) - len(s2)
>>> sorted(names, cmp=by_length)
TypeError: sort() got an unexpected keyword argument 'cmp'
>>> import functools
>>> sorted(names, key=functools.cmp_to_key(by_length))
['bob', 'alice', 'Charlie']

Note that uppercase letters come before lowercase ones, since that's how their code points compare; so we can use key=str.lower for a case-insensitive sort, or key=len to sort by length. In Python 2, they even had a cmp argument, but they removed it in Python 3.0 — hence the error message; and if you do have such an old comparison function lying around, functools.cmp_to_key will wrap it for you. But really, the whole point is that this design is actually better: the keys are computed once per item, rather than once per comparison.

Someone asked me this in class, actually: "but what if we already have a whole infrastructure of comparator objects? Wouldn't we want to use a sort that accepts them?" — of course we would, or we could just wrap them with cmp_to_key. But the question we should be asking ourselves is, why did we develop this framework to begin with, when it's possible to achieve with a single argument?

So in conclusion, the strategy pattern is, very simply, about passing the algorithm in as an argument. It sounds an awful lot like dependency injection, doesn't it (we'll say more about it in the next chapter)? And in a high-level language like Python, it's such a natural thing for a dependency to be a strategy, that it doesn't even warrant a name; whoever writes a comparator hierarchy in Python must be so preoccupied with the idea of "design patterns", that they completely miss the point — even though they're implementing one that is generally considered good and sophisticated. But we've seen it already, throughout the book, under other names: from higher-order functions, to the callbacks of the async chapter, to the event handlers of the GUI one.

Abstract Factory

The intent behind this pattern is to "provide an interface for creating families of related or dependent objects without specifying their concrete classes." The textbook example, which we actually encountered in the previous chapter, is that of a graphical user interface with windows, buttons and text fields, which can have different themes, like white or black.

In procedural programming, we'd have to pass the theme name everywhere, and branch on it whenever we needed to draw a window: if it's white, draw a white window; and if it's black, draw a black window. Then, we'd have to do the same for a button; and so on. If we need to add a red theme, we'd have to touch all of this code and add it there, too.

In object-oriented programming, we can encapsulate this decision in a theme object, which knows how to draw white widgets or black ones. Better yet, we can delegate the actual drawing to the widget, so the theme can simply create them, and they can take care of their own rendering. This is our abstract factory: an object that groups the creation of a family of related objects — in this case, a white window and a white button. Since universities still teach Java, here it is in Java, and running it:

$ cat Themes.java
public class Themes {
    static void draw(Theme theme) {
        Window window = theme.createWindow();
        Button button = theme.createButton();
        window.draw();
        button.draw();
    }
    public static void main(String[] args) {
        draw(args[0].equals("white") ? new WhiteTheme() : new BlackTheme());
    }
}

abstract class Window { abstract void draw(); }
abstract class Button { abstract void draw(); }
abstract class Theme {
    abstract Window createWindow();
    abstract Button createButton();
}

class WhiteWindow extends Window { void draw() { System.out.println("white window"); } }
class WhiteButton extends Button { void draw() { System.out.println("white button"); } }
class WhiteTheme extends Theme {
    Window createWindow() { return new WhiteWindow(); }
    Button createButton() { return new WhiteButton(); }
}

class BlackWindow extends Window { void draw() { System.out.println("black window"); } }
class BlackButton extends Button { void draw() { System.out.println("black button"); } }
class BlackTheme extends Theme {
    Window createWindow() { return new BlackWindow(); }
    Button createButton() { return new BlackButton(); }
}
$ java Themes.java white
white window
white button
$ java Themes.java black
black window
black button

Let's go over it, because it's a bit mind-bending. We start with two abstract classes, Window and Button, and an abstract Theme, which creates them. Then, for every color, we create a concrete window, a concrete button, and a concrete theme, which knows to create those. This way, draw doesn't have to care or know which theme it got; so if we want to add a red theme, all we need to do is define a red window, a red button and a red theme that binds them — nothing else needs to change. This is actually a pretty big deal, and it did take some overhead to achieve it: we've ended up with nine different classes just to say that white things go together. Incidentally, Java 11 and up can run a single source file directly, as long as its first class has a main method.

If we draw it with UML, we'll have so many arrows that we'll get lost; and usually, it's presented in its generalized form, which uses even more incomprehensible names like AbstractProductA, ConcreteProductB1 and ConcreteFactory2, so it's even worse. If we port this design pattern over to Python and implement it faithfully, we won't gain much, because its inheritance hierarchy will buy us nothing — Python doesn't care about types, remember? And while we can bring those types back with the abc module, as we've seen in the Inheritance chapter (a subclass that forgets to implement an abstract method will simply be unable to instantiate), I never understood why people do this in Python. It takes all the overhead that everyone hated about older languages, and had no choice about — and migrates it over to a language that was specifically and explicitly designed to free us from this burden. It's like a Rube Goldberg machine: yeah, it wipes your mouth with a napkin in the end, but do you honestly need all that contraption to do it?

So let's see what the Pythonic version would look like. First of all, we have duck typing, so we can get rid of all the abstract classes. And since our classes are objects, we can also get rid of all the factory methods, which don't do anything other than invoke a constructor — we can simply store the class itself, right? This gives us:

>>> class WhiteWindow:
...     def draw(self):
...         print('white window')
>>> class WhiteButton:
...     def draw(self):
...         print('white button')
>>> class BlackWindow:
...     def draw(self):
...         print('black window')
>>> class BlackButton:
...     def draw(self):
...         print('black button')
>>> class WhiteTheme:
...     Window = WhiteWindow
...     Button = WhiteButton
>>> class BlackTheme:
...     Window = BlackWindow
...     Button = BlackButton
>>> def draw(theme):
...     window = theme.Window()
...     button = theme.Button()
...     window.draw()
...     button.draw()
>>> draw(BlackTheme)
black window
black button
>>> draw({'Window': WhiteWindow, 'Button': WhiteButton})
AttributeError: 'dict' object has no attribute 'Window'
>>> from types import SimpleNamespace
>>> draw(SimpleNamespace(Window=WhiteWindow, Button=WhiteButton))
white window
white button

To wit, WhiteTheme and BlackTheme are classes with two class attributes — and draw simply instantiates whatever theme.Window and theme.Button happen to be. If we'd have a dictionary with the same two classes, it wouldn't work, because draw accesses the theme's window and button with dot notation, not square brackets; so a dictionary is not quacking the right way. But a SimpleNamespace with the same two classes would work just fine. In other words, in Python, an abstract factory is really nothing more than a dictionary of classes — or a namespace thereof.

I once had a student who told me about a problem he was asked during a job interview: he solved it in a very Pythonic way, but the interviewer wasn't satisfied, and kept pestering him until he came up with an object-oriented solution involving abstract classes. It's important to understand that just because someone is in a position of interviewing you, it doesn't mean that they are automatically a better programmer than you: the whole world is a consensus of children that weren't told that they've grown up. If you want to write Python in a Pythonic way, by all means do so — and if you want to write C++ in an object-oriented way, so be it (at least you don't have much choice there). But trying to write C++ in Python gains you nothing (in C++, you'd at least get the benefit of speed, or a compiler that checks your types).

A student once asked me a very good question: why would a white window be a different class from a black one, instead of just a single window class with a color attribute? And that's a good question, really. In fact, this is known as composition, and we'll get to it in a moment; but many people solve problems with class hierarchies, simply because it's the de facto standard they were taught — so let's keep this question in mind for now.

The verdict: we've grouped constructors in a factory object, and the "abstract" part is actually unnecessary when dealing with ducks, because the factory is really a dictionary of classes. It used to be useful, and then it stopped being useful — my problem is not with the pattern itself, but with teaching it (or anything, really) as eternally and undoubtedly useful.

Singleton

The intent of the singleton pattern is to restrict the instantiation of a class to one single object; so for example, we might have one database object, which lets us keep one open connection rather than opening a new one on every access; or one filesystem object, since having eight of those, each with their own internal state, describing the same filesystem is bound to get out of sync; or the cache of imported modules, which we've already encountered in the Modules chapter.

This is actually quite a controversial pattern, and is often considered to be an anti-pattern (something that you should never do). This is because it's effectively a way of introducing global state into your system — and code that relies on a single global variable, where every function goes and gets its inputs from there, and possibly writes its outputs back to it, is rarely a pleasure to work with. But it's not really the pattern's fault: the problem is not that there's one of something, but how the rest of the code gets to it. If every function goes to some one place to get its input, then it's an issue of global state — but if the instance is passed in as a dependency, then its uniqueness is really orthogonal to how it's communicated. In this case, testing is actually easier, since we can pass in a mock object to emulate different behaviors; we'll talk more about this in the next chapter.

Anyway, the classic implementation, as in C++ or Java, looks like this:

>>> class Singleton:
...     instance = None
...     @classmethod
...     def get_instance(cls):
...         if cls.instance is None:
...             cls.instance = cls()
...         return cls.instance
>>> instance1 = Singleton.get_instance()
>>> instance2 = Singleton.get_instance()
>>> instance1 is instance2
True
>>> instance3 = Singleton()
>>> instance1 is instance3
False

Two calls to get_instance will return the same object; but nothing stops anyone from calling the class directly to create another object. (Note that this implementation is also not thread-safe; that is, if two threads call get_instance at the same time, they might both see that the instance doesn't exist yet, and create it — a lock is required to prevent this). And while we could throw an error in __init__ if an instance already exists — it's not going to work for long, since object.__new__ actually creates the object (as we saw in the lifecycle chapter) without calling __init__ at all.

A better way, then, is to override __new__, the implicit class method that creates an object before its __init__ method is called to initialize it — so that the interface stays natural, and all calls to the class are automatically funneled through it, returning the same object as expected:

>>> class Database:
...     _instance = None
...     def __new__(cls, url):
...         if cls._instance is None:
...             cls._instance = super().__new__(cls)
...         return cls._instance
...     def __init__(self, url):
...         self.url = url
>>> db1 = Database('sqlite:///users.db')
>>> db2 = Database('sqlite:///orders.db')
>>> db1 is db2
True
>>> db1.url
'sqlite:///orders.db'
>>> db3 = object.__new__(Database)
>>> db1 is db3
False

Now, however, users of this code will be surprised twice: first, because they expect their object to be constructed with __init__, which actually happens on every call — so the second "construction", which is actually a "reconstruction", silently re-initialized the one database object we have with a different URL; and second, because the line that creates objects is always reachable from outside, so they can create a new object by calling object.__new__ on our class — and if they set _instance back to None, they'd simply reset our singleton.

To prevent this, we can think outside the box: modules are singletons, too. Whenever we import something, Python keeps a single copy of it in sys.modules, so the module itself can be used as the instance:

>>> open('singleton.py', 'w').write('x = 1\n')
6
>>> import singleton as instance1
>>> import singleton as instance2
>>> instance1 is instance2
True
>>> instance2.x = 2
>>> instance1.x
2
>>> import sys
>>> del sys.modules['singleton']
>>> import singleton as instance3
>>> instance1 is instance3
False
>>> instance3.x
1

And indeed, both imports yield the same object; if we set some state through one, it is accessible through the other as well. But as we've seen in the Modules chapter, this approach can be defeated, too: if we delete the module from sys.modules, the next import will actually reload and reinitialize it — so we'll get a new instance with x set to 1 again. The truth is, any protection we can devise will have a way to circumvent it — Python is just too dynamic a language, and if we know it well enough, we can patch anything.

That's why Python has a rather anticlimactic solution to this problem, which is in fact quite in line with its overall philosophy: we are all consenting adults, remember? What's the point of all this protection, then? If we want to have a singleton, we can just name its class with a leading underscore, indicating that it's "private, use at your own risk — it might change in the next version"; then we can create a single instance of it at the module level, and tell people to use this:

>>> class _Database:
...     def __init__(self, url):
...         self.url = url
>>> database = _Database('sqlite:///users.db')
>>> another = type(database)('sqlite:///orders.db')
>>> another is database
False
>>> class Borg:
...     _shared_state = {}
...     def __init__(self):
...         self.__dict__ = self._shared_state
>>> drone1 = Borg()
>>> drone2 = Borg()
>>> drone1 is drone2
False
>>> drone1.x = 1
>>> drone2.x
1
>>> drone1.__dict__ is drone2.__dict__
True

If anyone wants to create a new instance, they can do so with type(database)(...) — but then, nobody is going to accidentally call a name starting with an underscore and expect it to work, right? So, if we want to create a single object — then let's create a single object, not kill a mosquito with a cannon.

Incidentally, there's a variation of the singleton pattern that I really like, because it attacks the problem from the other side: it's called the Borg (and it was actually invented by Alex Martelli, back in the early 2000s). It suggests that instead of having one object, we have many objects that share their state — that is, they all have the same __dict__, so although they are not literally the same object, as is would tell you, their state is literally the same dictionary, so they behave exactly the same. It's a funny name, because in Star Trek, the Borg are a race of aliens with a collective consciousness — so it's really this shared dictionary that plays the role of their "queen", and the rest of the objects are the drones.

Anyway, I think that this is a much more humanistic approach. It's a question of trade-offs: on one hand, we want to enable people, and on the other hand — protect them from themselves. But there's something old-fashioned and paternalistic about the second approach, in which you assume that you understand other people, their values and what's good for them, better than they do; if your servers are down and you really need an ugly hack, Python is there for you — but then it's your responsibility to clean up after yourself. A hammer that would run away when you try to cut bread with it would be pretty absurd, wouldn't it? So the idea of "let's have just one of something" is always relevant; it's the implementation that's not.

Factory Method

This one is actually exciting, because it's useful. Not for the reason its authors intended, but we'll get there. The idea behind it is that a lot of times, we'd like to create objects without specifying the exact class of the object we'd like to create. For example, suppose we have a point class, which can be either in cartesian or in polar coordinates. Both implementations have the same interface, but the one is implemented in terms of x and y, the other in terms of r and theta, so they need to be created differently. A factory method lets us do exactly that: it returns either one object or another, and the code using it doesn't need to worry which.

That's what sets it apart from the abstract factory: in an abstract factory, we had an object that was used to create a family of related objects; here, we have a single method that's used to create a single object. In Python, this isn't very interesting, since classes are objects — so we can as easily put one point class or another behind the Point name and invoke it to create one or the other.

What is interesting is that sometimes, the way we want to create an object varies. A cartesian point can be constructed from x and y, while a polar point can be constructed from r and theta. But in Python, we only have one constructor, and we can't overload it like we could in C++ or Java. So, we give our constructors explicit names: we've actually done exactly that in the chapter on attributes, methods and descriptors. Point.from_cartesian and Point.from_polar, which we first implemented as static methods, then as class methods, so ColorPoint.from_polar would return a ColorPoint, thereby letting the subclass determine the exact class (which, come to think of it, was the original point, ha ha).

The benefits are pretty cool: we get several constructors, and can give them informative names; passing in 1 and 2 and expecting them to be x and y or r and theta is pretty vague, but from_polar is pretty unambiguous. And as long as people use these named constructors, we can change the implementation of the class without fear of breaking anyone's code; we can even change the internal representation, for example to store r and theta instead.

This is actually used all over the standard library:

>>> dict.fromkeys(['alice', 'bob'], 0)
{'alice': 0, 'bob': 0}
>>> int.from_bytes(b'\x01\x00', 'big')
256
>>> bytes.fromhex('cafe')
b'\xca\xfe'
>>> import datetime
>>> datetime.date.fromisoformat('2019-12-11')
datetime.date(2019, 12, 11)
>>> datetime.datetime.fromtimestamp(0, tz=datetime.timezone.utc)
datetime.datetime(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)

dict.fromkeys, int.from_bytes, bytes.fromhex, date.fromisoformat and datetime.fromtimestamp (given an explicit time zone, so the result doesn't depend on the machine) are all class methods, named after what they construct from. It's not exactly the factory method described in the book (Brandon Rhodes frames it this way, and I like his framing better), but it is useful, and that's a good enough reason to learn something; or to learn from history, if you will, so that we don't repeat it. It's just that when we look at history, we tend to see a mixture of good and bad; but when we see an established method, we tend to assume it's good until proven otherwise beyond any doubt, like with bloodletting, smoking, or alchemy.

Template Method

Next, let's consider the template method. Its intent is to define the high-level logic of some algorithm in a base class, and leave exact specifics to subclasses — it's a pretty standard pattern, so you might have seen it before (and maybe even used it yourself). To give an example, a worker might live by eating, working and sleeping; but the working part is a bit of a blank, because it depends on the worker's profession. The base class can implement the general skeleton, and leave this blank for the subclasses to fill:

>>> class Worker:
...     def live(self, days=1):
...         for day in range(days):
...             print('eating...')
...             self._work()
...             print('sleeping...')
...     def _work(self):
...         raise NotImplementedError()
>>> class Doctor(Worker):
...     def _work(self):
...         print('examining...')
>>> class Actor(Worker):
...     def _work(self):
...         print('performing...')
>>> class Philosopher(Worker):
...     def _work(self):
...         print('making coffee...')
>>> Philosopher().live()
eating...
making coffee...
sleeping...
>>> Worker().live()
eating...
NotImplementedError

Let's break it down. The live method is our template: it defines the high-level logic, which it then delegates to the _work method, which is the blank. This blank is unimplemented in the base class, and simply raises an error; the underscore in its name is an indicator that it's not intended for users, only for subclasses to override. In the philosopher's case, it's implemented to work by making coffee (because that's what philosophers' job prospects are like). On my slide, live was actually a while True loop, which is more realistic, but somewhat harder to demonstrate — so here, I've condensed it to live for a given number of days, but you get the idea. The interesting point is that our base class is actually a broken object, because it simply can't work: it can only eat, and then throw an error.

More generally, we can describe this pattern as an AbstractClass with some method that calls _template_method_1 and _template_method_2, which are then implemented in a ConcreteClass. We've actually seen this before: when we were overriding run to subclass threading.Thread (in the Concurrent Programming chapter), we were using a template method; and I even mentioned its classic bug, of forgetting to call super().__init__(). We also encountered a similar idea when extending Jinja templates: a base template with a placeholder, to be filled in by any derived template (in the Web Programming chapter). Hell, even the Thread class's docstring has a line about it: "There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass."

So this pattern is all over the place. It's worth noting, though, that the standard library actually provides us with another option: we can pass in a callable, as a strategy, instead of subclassing an entire class and overriding a method. For example, json can't serialize dates: we could either subclass JSONEncoder and override its default hook (which is a template method), or pass a function as the default argument (which is a strategy):

>>> import datetime, json
>>> event = {'name': 'lesson 7', 'date': datetime.date(2019, 12, 11)}
>>> json.dumps(event)
TypeError: Object of type date is not JSON serializable
>>> class DateEncoder(json.JSONEncoder):
...     def default(self, obj):
...         if isinstance(obj, datetime.date):
...             return obj.isoformat()
...         return super().default(obj)
>>> json.dumps(event, cls=DateEncoder)
'{"name": "lesson 7", "date": "2019-12-11"}'
>>> json.dumps(event, default=datetime.date.isoformat)
'{"name": "lesson 7", "date": "2019-12-11"}'

Let's break it down: the JSONEncoder subclass checks the type, and if it's a date, returns its ISO representation — otherwise, it defers to its base class's implementation, which raises a TypeError. The second version does the same, except instead of a whole new class, it just passes in date.isoformat — which, if you remember our chapter on methods, is not actually a bound method, but a plain callable that receives the date as an argument. Same output, one line instead of six — and the same goes for loading, where we can use a similar object_hook argument.

So my claim is this: wherever you can use a template method, you can actually use composition instead — and you should. It might seem like a common, useful pattern; it might even feel smart — but in my opinion, it's absolutely terrible. To really drive this point home, though, we need to revisit inheritance itself.

Inheritance, Revisited

So yeah. To recap from the Inheritance chapter, when programming in statically typed languages, we saw that containers are designed to hold a specific type of element, and functions have a certain signature — they accept an argument of a particular type. This severely limited code reuse, so we invented polymorphism, and eventually, inheritance became the way to define an interface. So when I'm working on a Java project, I thank God for it: and if I'm working on a Python project, which has duck typing, I don't need it at all.

But the fact that it's so essential, and so common, made it very tempting to abuse. People started piggybacking on its side effects to solve other problems — and worse, they started thinking that this is actually the right way to solve them. "Look how clever I am," they thought, "not only can I solve this problem with polymorphism, but I can also use it to solve other problems, like a template method!" And if you think about it, it really does look smart — except, it really isn't.

I mean, a lot of houses in Tel Aviv have triangular roofs; and I get it, they're great for places with snow. But when was the last time it snowed in Tel Aviv? In the meantime, it heats up the house and creates an attic space you can't use, because it's a triangle, so you can't actually put anything in it. But people still build it, anyway, because it's… well, it's the thing you do, isn't it? A house has a roof, and a roof is triangular; that's how it looks, that's what it's supposed to be. But it's completely nonsensical, if you think about it; there are much better solutions for rain, or for sun. And much the same happens with inheritance.

In the Inheritance chapter, we saw that the murder dog robot fell right in the middle of the animal tree and the robot tree, and ended up pooping thanks to multiple inheritance. And then, we saw a real reader hierarchy, which was actually a template method — a base class that opened files in binary mode, decided when to stop, and generally forced the JSON reader to do all sorts of hacks, until it finally gave up and yielded an extra None at the end. The point is, it did a pretty bad job of predicting what its subclass would want, and in general, we're pretty bad at predicting the future.

So let's add a filter to that reader, and see what happens. Its natural place would be in the base class, right? After all, it's the same for all formats. But at the same time, it's called from the subclass's _next_chunk, so if you look back at that code, it ends up in read calling _eof, which is in the child, then _next_chunk, which is also in the child, and calls filter, which is in the parent, and back to read: that's seven jumps back and forth, between code that might very well be in different files or even subsystems. In the Inheritance chapter, I called it cellophane noodles, but in class, I usually call it spaghettini code: it's like spaghetti, only thinner.

Now, you might have heard the term spaghetti code in the context of goto statements. Dijkstra's famous letter, "Go To Statement Considered Harmful", is from 1968; and for a long time, it was a matter of principle to avoid goto — and famously, Linus Torvalds defended it in the Linux kernel. But so many programmers who wouldn't touch it with a ten foot pole have no problem writing elaborate class hierarchies, where one jump is the least of your worries. Goto considered harmful? Seriously?

Mattias Petter Johansson (or MPJ, of Fun Fun Function fame — definitely worth a watch, and preferably at 2x speed) puts it really well: "Inheritance encourages you to build a taxonomy of objects very early on," he says, "and you are probably going to make mistakes while doing that, because humans cannot predict the future. And once you've built yourself this taxonomy, it's really hard to get out of it."

And here's a case in point: in an email exchange with a student, we were talking about scikit-learn, a really great library whose functionality I have nothing against. However, its estimators inherit from BaseEstimator, which provides… well, nothing, really. There's an isinstance check, which you could emulate by doing hasattr(o, 'fit'), and there's get_params/set_params, which is basically a class around a dictionary — but then, of arguments, not of state. TransformerMixin then provides fit_transform, which is completely trivial, and could be implemented as a function accepting fit. And finally, RobustScaler made use of self._validate_data, which is a stateless helper function hidden in the base class — why wouldn't it be a separate function to begin with?

>>> import sklearn
>>> sklearn.__version__
'1.9.1'
>>> from sklearn.preprocessing import RobustScaler
>>> [cls.__name__ for cls in RobustScaler.__mro__]
['RobustScaler', 'OneToOneFeatureMixin', 'TransformerMixin', '_SetOutputMixin', 'BaseEstimator', 'ReprHTMLMixin', '_HTMLDocumentationLinkMixin', '_MetadataRequester', 'object']
>>> from sklearn.base import BaseEstimator, TransformerMixin
>>> [name for name in vars(TransformerMixin) if not name.startswith('__')]
['fit_transform', '_sklearn_auto_wrap_output_keys']
>>> hasattr(BaseEstimator, '_validate_data')
False
>>> from sklearn.utils.validation import validate_data

So let's break it down: in version 1.9.1, a scaler inherits from eight different classes (if you count object), most of which are mixins and private helpers. TransformerMixin still provides fit_transform, as well as a private attribute; but _validate_data is no longer there — it has graduated to validate_data, a public, standalone function in sklearn.utils.validation, and exactly the sort of refactoring suggested by my argument.

To be fair, inheritance is not so horrible — it's just mostly unnecessary. Subclassing something once can be useful, and there are even situations where it's absolutely required, like exceptions. But the problem is when you end up with a whole family — over time, these relationships diverge so much that it becomes impossible to see them. And there's an analogy hiding here, but this is a professional book, so I'll leave it unsaid.

Composition, Revisited

So what's the alternative? Composition. In this paradigm, we contain our dependencies — that is, store them as components, ideally passed in from outside, and delegate work to them. Not "delegate to my parent", but "delegate to the object I was given". When we did exactly this in the Inheritance chapter, and the reader took a parser, the whole tangle of the template method disappeared.

The price we pay is a bit more code: every delegated method becomes a bit of boilerplate, like def foo(self): return self.component.foo(). In statically-typed languages, it's even more cumbersome, which I believe is the reason it never really caught on there; inheritance was so much simpler, so we stuck with it, and then we shook off the shackles of static types but kept the solution that was designed for them. In a dynamic language like Python (or in a language like Javascript, which MPJ uses to demonstrate this point), composition is the way to go.

Consider our workers again: there's no such thing as a worker who is a doctor — there's a worker who has a profession, and medicine is an object that knows how to be performed. When you tell the worker to work, it tells its profession to perform:

>>> class Doctor:
...     def work(self):
...         print('examining...')
...     def diagnose(self, symptom):
...         return 'flu' if symptom == 'fever' else 'nothing'
>>> class Philosopher:
...     def work(self):
...         print('making coffee...')
>>> class Worker:
...     def __init__(self, name, profession):
...         self.name = name
...         self.profession = profession
...     def live(self):
...         print(f'{self.name} is eating...')
...         self.profession.work()
...         print(f'{self.name} is sleeping...')
...     def __getattr__(self, name):
...         return getattr(self.profession, name)
>>> house = Worker('House', Doctor())
>>> house.live()
House is eating...
examining...
House is sleeping...
>>> house.diagnose('fever')
'flu'
>>> house.profession = Philosopher()
>>> house.live()
House is eating...
making coffee...
House is sleeping...
>>> house.diagnose('fever')
AttributeError: 'Philosopher' object has no attribute 'diagnose'
>>> import copy
>>> copy.copy(house)
RecursionError: maximum recursion depth exceeded

Let's go over it again, just in case: live explicitly delegates the work, and __getattr__ — which we discussed in the attributes chapter, and saw how it's called only when normal lookup fails — forwards everything else to the profession, so the doctor's diagnose method actually works on the worker. And there's something that composition can do, which inheritance can't: have a career change, simply by switching professions — after which, the worker no longer knows how to diagnose. But this example also highlights a problem: copy.copy creates the copy without invoking __init__, and then looks for an attribute on it — which inevitably fails, prompting __getattr__ to look for self.profession, which doesn't exist on the copy yet, so __getattr__ is called again, and again, and again, until a RecursionError is raised. Pickling presents a similar problem.

But there's a way to solve this: in fact, it's a whole component of its own. If the boilerplate is getting annoying, we can write something that generates it for us — a class decorator, if you will, which we saw in the Decorators chapter. This decorator, then, can be applied to a class, and add a forwarding property for every name given to it:

>>> def delegate(component, *names):
...     def decorator(cls):
...         for name in names:
...             def forward(self, name=name):
...                 return getattr(getattr(self, component), name)
...             setattr(cls, name, property(forward))
...         return cls
...     return decorator
>>> @delegate('profession', 'work', 'diagnose')
... class Worker:
...     def __init__(self, name, profession):
...         self.name = name
...         self.profession = profession
>>> house = Worker('House', Doctor())
>>> house.diagnose('fever')
'flu'
>>> import copy
>>> copy.copy(house).diagnose('cough')
'nothing'

And this should work. Let's see what happens: for every name, delegate creates a property, whose getter goes and fetches the component, then the attribute from it. The default argument, name=name, is a trick we saw in the Assignment chapter to address the late binding of the loop variable, and generally make sure every property has its own copy of the name. This way, only the listed names are forwarded, and if any of them are missing, it's a normal AttributeError, and the copying works as expected.

But you might be wondering, how do we make sure a component has the methods that it's supposed to? The short answer is, we don't — or, we use abc, or a Protocol for the type checker (from the Type Hinting chapter), if it's really important to us. Personally, I write tests — so if I compose the real objects together, the moment one of them doesn't have a method that the other one is trying to call, the test will crash with an AttributeError anyway. In other words, I don't protect people from themselves — I just trust them to pass in a duck, and let the duck typing take care of the rest.

So yeah, in a dynamic language, composition is almost always better than inheritance. It results in a much sharper and simpler interface between components, and any shared logic can be extracted into a new component, rather than agonizing over whether it should be the parent or the child. It also encourages separation of concerns — but it does incur a bit more boilerplate, especially in constructing and composing all the different components together. But that almost sounds like a good reason to develop a new component, doesn't it?

Invisible Patterns

So, five out of twenty three; what about the rest? In a talk from 1996 about design patterns in dynamic languages, Peter Norvig found that sixteen out of those twenty three were essentially invisible, or simpler, in languages like Lisp or Dylan. Python, too, belongs to the same family, and most of these patterns are features that we've already covered.

The iterator pattern, for example, is a way to traverse a collection without exposing its underlying structure. It's what we have in the iteration protocol, with __iter__ and __next__, which is used by every for loop — and which we can write ourselves using generators (as we've seen in the Generators chapter). The decorator pattern is a way to dynamically wrap an object to augment its behavior, while still keeping its original interface intact. In Python, it's exactly what decorators do — which, as we've seen in the Decorators chapter, can be used for much more. The proxy and adapter patterns are both about delegation, which we've already seen as well, with __getattr__ in the previous section. The singleton and flyweight patterns are about storing a single copy of something: and in fact, the Python interpreter already does that for None, booleans, small integers and interned strings (see the chapter on IDs, types and values).

The command pattern is a way to encapsulate a request as an object, so that it can be queued or undone. In Python, that's simply a function — or a bound method, if it needs a state — which is already an object, so we can store it and call it later:

>>> document = []
>>> undo_stack = []
>>> def append(text):
...     document.append(text)
...     undo_stack.append(document.pop)
>>> append('hello')
>>> append('world')
>>> document
['hello', 'world']
>>> undo_stack.pop()()
'world'
>>> document
['hello']

document.pop, which we store (and don't call) is a command: a bound method that remembers its list (from the methods chapter). undo_stack.pop()() pops the last command off the stack, and actually calls it, so it pops the document. And if the command requires some arguments, we can use functools.partial to "freeze" them in.

Similarly, the observer pattern is a way to subscribe to some event, to get notified when it happens. It's exactly what we did in the GUI chapter, with event handlers — a list of callbacks, if you will. The state pattern is a way to change behavior when some internal state changes — exactly what we did in the async chapter, when our protocol's state was stored as a bound method, self.next. The prototype pattern is a way to create new objects by copying an existing one — and that's what the copy module is for. The chain of responsibility pattern is what we saw in the web chapter, when we implemented a middleware.

There are a few more, that aren't in the Gang of Four's book, but are pretty standard in other languages: the execute-around pattern is a way to set something up and tear it down around a block — like a context manager (from the lifecycle chapter); Java's getters and setters are property; and if you're familiar with registries of subclasses, that's what metaclasses and __init_subclass__ (from the Metaclasses chapter) are for.

There is, however, one pattern that's still useful, and widely used in Python, so let's talk about it a bit more. The visitor pattern is a way to run some operation over a structure with many different types of nodes — without having to modify the nodes' classes to account for it. It's used in the Python standard library, in ast.NodeVisitor: it dispatches every node to a method that's named visit_ plus this node's class name — so it can be found by name, and not by some complicated double dispatch as in the original C++ implementation. Similarly, functools.singledispatch lets us dispatch a function based on the type of its first argument:

>>> import ast, collections
>>> class NameCounter(ast.NodeVisitor):
...     def __init__(self):
...         self.names = collections.Counter()
...     def visit_Name(self, node):
...         self.names[node.id] += 1
>>> counter = NameCounter()
>>> counter.visit(ast.parse('x = y + x * f(z)'))
>>> counter.names
Counter({'x': 2, 'y': 1, 'f': 1, 'z': 1})
>>> from functools import singledispatch
>>> @singledispatch
... def describe(node):
...     return type(node).__name__
>>> @describe.register
... def _(node: ast.Name):
...     return f'the name {node.id}'
>>> @describe.register
... def _(node: ast.Constant):
...     return f'the constant {node.value!r}'
>>> [describe(node) for node in ast.walk(ast.parse('x + 1'))]
['Module', 'Expr', 'BinOp', 'the name x', 'Add', 'the constant 1', 'Load']

Let's go over it: the visitor traverses the tree, calling visit_Name on every name, which counts the x node twice (and f once, since the function called is a name, too). When a node can't be described further, describe falls back to its class name; and the overloads, registered by the type annotation of their first argument, take care of names and constants, while ast.walk simply traverses the nodes in breadth-first order. As a side note, NodeVisitor is intended to be used by subclassing — a template method — but that's OK (if we're subclassing, we probably do it just once, and from a stable and well-documented base class).

So what is it about these patterns that makes them valuable? It's not the code, really — it's the concept: "there's only one of these", or "this algorithm is a parameter", or "construct it by name", or "visit every node". It's a useful idea, and as such, it's a useful name to have for it — and that was Alexander's point all along. Having a pattern language allows us to discuss our design in a shared vocabulary, and a code review goes much faster if you can say "that's just a strategy" and everyone understands what you mean. In Python, though, the implementations are mostly a line, or a feature.

Drivers and Middleware

For the rest of the chapter, we'll talk about some patterns that are actually mine; or rather, my take on ideas that are quite common in good code. The reason I like them is because they're more platform than solution: they don't say "here's a problem, solve it like this", but "build infrastructure like this, and entire classes of problems will become solvable". So take them with a grain of salt; they're entirely opinionated, and quite situational. But I find them interesting, because I think they address a particular challenge in software design: namely, how do we design systems to be future-proof, so they can evolve and adapt to new requirements? Or even, how do we design systems to be so future-proof that we can't even predict the requirements, but we know that whatever they are, we'll want to account for them?

Let's start with drivers. It's a modified, simpler version of the Bridge pattern, whose intent is to decouple an abstraction from its implementation, so that the two can vary independently. A file system can be local or remote over SSH; a database can be relational or not; a reader can read different formats; a server can speak different protocols. In each case, the interface is the same, but the implementation is different — so the way I like to configure it is with a URL.

As we've seen in the Web Development chapter, the anatomy of a URL, scheme://username:password@host:port/path?query#fragment, is pretty versatile; and it turns out that it's useful for much more than the web. file:///data.txt, ssh://dan:1234@host/file.txt, sqlite:///data.db (that's three slashes, because the host part is empty), postgresql://dan:1234@localhost/db, tcp://0.0.0.0:8000 — all of these are valid configurations for what we're going to do; and it's not even dependency injection, because I don't actually pass the dependency — I pass a hint as to how to construct it:

>>> import urllib.parse
>>> class Database:
...     def __init__(self, url):
...         self.driver = find_driver(url)
...     def create_user(self, name):
...         return self.driver.create_user(name)
...     def get_user(self, user_id):
...         user = self.driver.get_user(user_id)
...         if user is None:
...             raise LookupError(f'no user with id {user_id}')
...         return user
>>> drivers = {}
>>> def find_driver(url):
...     scheme = urllib.parse.urlsplit(url).scheme
...     if scheme not in drivers:
...         raise ValueError(f'invalid url: {url}')
...     return drivers[scheme](url)
>>> class MemoryDriver:
...     def __init__(self, url):
...         self.users = []
...     def create_user(self, name):
...         self.users.append(name)
...         return len(self.users)
...     def get_user(self, user_id):
...         if 1 <= user_id <= len(self.users):
...             return self.users[user_id - 1]
>>> drivers['memory'] = MemoryDriver

Let's break it down: a Database holds a driver (composition) which is found by find_driver, which simply looks up the URL's scheme in a dictionary of driver classes. Some methods, like create_user, are plain pass-throughs — but others, like get_user, are a bit more interesting, in that they turn the driver's return value of None into a LookupError, so no driver ever has to worry about this particular edge case:

>>> import sqlite3
>>> class SQLiteDriver:
...     def __init__(self, url):
...         path = urllib.parse.urlsplit(url).path[1:]
...         self.connection = sqlite3.connect(path)
...         self.connection.execute('CREATE TABLE IF NOT EXISTS users (name TEXT)')
...     def create_user(self, name):
...         cursor = self.connection.execute('INSERT INTO users VALUES (?)', (name,))
...         return cursor.lastrowid
...     def get_user(self, user_id):
...         row = self.connection.execute('SELECT name FROM users WHERE rowid = ?', (user_id,)).fetchone()
...         return row and row[0]
>>> drivers['sqlite'] = SQLiteDriver
>>> for url in ['memory://', 'sqlite:///:memory:']:
...     db = Database(url)
...     user_id = db.create_user('alice')
...     print(url, user_id, db.get_user(user_id))
memory:// 1 alice
sqlite:///:memory: 1 alice
>>> db.get_user(2)
LookupError: no user with id 2
>>> Database('mongodb://localhost/test')
ValueError: invalid url: mongodb://localhost/test

Here, we have a SQLite driver, which extracts its path from the URL (the :memory: is how you tell SQLite you want an in-memory database). It then creates a table, and uses parameterized queries. The same loop works for both URLs, and the interface takes care of edge cases once — a missing user, an unknown scheme.

This is nice, because it decouples the interface from the implementation: if we want to add a new backend, we just write another driver. What's more, the interface takes care of edge cases and any other shared logic, so the drivers can make certain simplifying assumptions — like an update_user method that only calls the driver if something actually changed, so every driver can assume the update is real. And since there's one database class, and multiple drivers, it makes sense to solve problems once on the database side; this way, each driver is highly cohesive, focusing on a single technology, and testing it is a breeze — you just patch find_driver. It's the modern-day template method, but without the tangle.

Let's move on to middleware. In Python terminology (borrowed, just like drivers), this is a system of hooks: a way to register callables that will run before or after certain events, with complete man-in-the-middle powers: they can change the call, its return value, or even hijack it entirely. We've already seen how it works for the web, in the Web Programming chapter, and Django calls it a framework of hooks. Similarly, we can implement it for databases, to run before or after certain operations like saving, updating, deleting or even selecting; or for parsers, to run before or after specific parsers:

>>> class Database:
...     def __init__(self):
...         self.objects = []
...         self._on_create = []
...         self._on_select = []
...     def on_create(self, *types):
...         def decorator(f):
...             self._on_create.append((types, f))
...             return f
...         return decorator
...     def on_select(self, **query):
...         def decorator(f):
...             self._on_select.append((query, f))
...             return f
...         return decorator
...     def create_object(self, **kwargs):
...         self.objects.append(kwargs)
...         for types, f in self._on_create:
...             if kwargs.get('type') in types:
...                 f(kwargs)
...         return kwargs
...     def select_objects(self, **query):
...         for hook_query, f in self._on_select:
...             if hook_query == query:
...                 result = f(self, **query)
...                 if result is not None:
...                     return result
...         return [obj for obj in self.objects if query.items() <= obj.items()]

Let's break it down: both on_create and on_select are decorator factories (see Decorators chapter). They simply register the function, associating it with some types or query, and then return the function as-is. create_object invokes the matching create hooks on the newly created object; and select_objects invokes the matching select hooks, and if any of them returns something other than None, that's the result — otherwise, it falls back to the regular query:

>>> import random
>>> db = Database()
>>> @db.on_create('chair', 'sofa')
... def make_sittable(obj):
...     obj['sittable'] = True
>>> db.create_object(type='chair')
{'type': 'chair', 'sittable': True}
>>> db.create_object(type='lamp')
{'type': 'lamp'}
>>> @db.on_select(type='random')
... def select_random(db, **query):
...     types = sorted({obj['type'] for obj in db.objects})
...     return db.select_objects(type=random.choice(types))
>>> db.select_objects(type='random')
[{'type': 'lamp'}]
>>> db.select_objects(type='random')
[{'type': 'chair', 'sittable': True}]

In this case, we make all chairs sittable without having to pass it in explicitly (the lamp definitely is not), and the select hook hijacks control entirely, inventing a random type (seeded for a reproducible run) that the database didn't even have.

Verdict: the modern-day chain of responsibility; since we can't predict what users will want, we give them access to our system's internals. Incidentally, that's what Python does, too — if you want to implement __getitem__, or even descriptors or __getattribute__ if you really want to go down the rabbit hole, in 90% of cases you don't have to, but when you do, you can. Similarly, git has plumbing and porcelain commands: the porcelain ones are the sink you usually use, but if it gets clogged, you open the cabinet underneath and use the plumbing.

Aspects and Plugins

Aspect-oriented programming might sound intimidating, or maybe old-fashioned to some; the term was coined by Gregor Kiczales and his team at Xerox PARC in the late 1990s, which later gave us AspectJ. But actually, it's something you've seen already, and it's super powerful: it lets us separate what's known as cross-cutting concerns by adding advice through pointcuts.

Cross-cutting concerns are the parts of our responsibilities that are not directly related to the core logic, but still need to live in it somehow — like authentication, history or logging. Advice is the additional behavior that we'd like to add to it — something like "authenticate, or redirect to the login page" — and a pointcut is a way of specifying which part of the code this should happen to — for example, "all functions whose name starts with secure_", without actually changing the code itself. In fact, the Decorators chapter used decorators for exactly those concerns; now, I'd like to generalize this even further.

My generalization is simple: let users define functions, classes, and other things — and then find a way to incorporate them into your system. It's the difference between an application and a framework, or between prescriptive and descriptive approaches: pytest, for example, runs any function starting with test, and injects dependencies through arguments whose names match their fixtures. Django takes your classes and turns them into tables, and your functions into a webserver; all you have to do is define what you'd like to see happen, and it goes on to make it happen for you. It does all the "boring" work, so you can focus on the interesting parts and get them done.

To illustrate, consider a firewall — a program that monitors every packet and decides whether to allow it or not. The most obvious way to implement this would be to accept rules as strings: "block port 80", or "block packets that contain foo". Then, someone says, "well, I actually wanted foo and bar", or "I actually wanted foo or bar"; and pretty soon, you end up designing, parsing and evaluating a whole programming language of strings. So why not accept code instead?

>>> import inspect, itertools
>>> class Firewall:
...     def __init__(self):
...         self.filters = []
...     def block(self, rule):
...         keyword, _, word = rule.partition(':')
...         if keyword != 'contains':
...             raise ValueError(f'invalid rule: {rule}')
...         self.filters.append(lambda packet: word in packet)
...     def filter(self, f):
...         self.filters.append(f().should_drop if inspect.isclass(f) else f)
...         return f
...     def on_packet(self, packet):
...         for should_drop in self.filters:
...             if should_drop(packet):
...                 return 'drop'
...         return 'allow'
>>> firewall = Firewall()
>>> firewall.block('contains:foo')
>>> @firewall.filter
... def should_drop(packet):
...     return 'bar' in packet or 'baz' in packet
>>> @firewall.filter
... class EverySecondPacket:
...     def __init__(self):
...         self.counter = itertools.count()
...     def should_drop(self, packet):
...         return next(self.counter) % 2 == 1
>>> [firewall.on_packet(packet) for packet in ['foo', 'bar', 'hello', 'world', 'and', 'goodbye']]
['drop', 'drop', 'allow', 'drop', 'allow', 'drop']

Let's go over it: block deals with the only simple string rule we allow, while filter registers a function, or, if it receives a class, instantiates it and registers its bound should_drop method, whose self stores the state (namely, an itertools.count). Finally, on_packet drops a packet if any filter indicates that it should; and the class actually drops every second packet that gets that far (world and goodbye; foo and bar have been dropped already). This is known as progressive disclosure: to achieve a simple goal, like blocking a port, you provide a string; but to run some logic, you can also provide a function; and if you want a state, you can provide a class. Even better, it's not a solution — it's a platform: in security, new attacks keep coming, so a firewall that blocks yesterday's attacks is not nearly as valuable as a firewall that lets you add tomorrow's defenses to it.

There are two ways this idea can be refined further. First, a yes-or-no return value is pretty limiting, so we can pass every filter a context object with methods like drop (or, allow to some subnet), and check it afterwards. This is known as dependency injection, or even demand — like in pytest's fixtures. Second, to enable various shared utilities, like calculating a standard checksum, plugins can simply import utils, assuming we provide such a module in sys.modules in advance.

Now, the plugins usually live in separate files, so the framework would have to go and fetch them; in Python, this means importing them. Consider the following code, then:

>>> import importlib, pathlib, sys, types
>>> def load_rules(firewall, root):
...     root = pathlib.Path(root).absolute()
...     sys.path.insert(0, str(root.parent))
...     for path in sorted(root.iterdir()):
...         if path.name.startswith('_') or path.suffix != '.py':
...             continue
...         module = importlib.import_module(f'{root.name}.{path.stem}')
...         for name, value in vars(module).items():
...             if name.startswith('rule_') and callable(value):
...                 firewall.filter(value)
>>> sorted(path.name for path in pathlib.Path('rules').iterdir())
['_draft.py', 'checksum.py', 'no_qux.py']
>>> print(pathlib.Path('rules/checksum.py').read_text())
import utils

def rule_checksum(packet):
    return utils.checksum(packet) == 42

>>> utils = types.ModuleType('utils')
>>> utils.checksum = lambda packet: sum(packet.encode()) % 256
>>> sys.modules['utils'] = utils
>>> firewall = Firewall()
>>> load_rules(firewall, 'rules')
>>> [firewall.on_packet(packet) for packet in ['qux', 'aah', 'hello']]
['drop', 'drop', 'allow']

Let's go over it again: load_rules adds the directory's parent to sys.path, imports every .py file in it as rules.<name> (no __init__.py needed, thanks to namespace packages, which we discussed in the Modules chapter), and skips any files starting with an underscore — an easy way to temporarily disable a rule. The pointcut, then, is a naming convention: every callable whose name starts with rule_. So the plugins don't import the firewall, or use its decorators — they're just code. In checksum.py, we import utils, which exists only in sys.modules; and we see that qux and aah are dropped (aah's byte sum is 298, which is 42 modulo 256), and hello is allowed.

Interestingly, plugins are developed separately from the framework, which leads to plugin seniority: when many plugins use the same utility, it makes sense to promote it to the platform. This, in turn, can even help with Conway's law, from Melvin Conway in 1968: "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations". This is exactly why big tech companies like Google or Facebook have entire teams whose sole purpose is to build tools for the rest of the company.

Design Kōans

Before you jump up and exclaim "hold on, you're just teaching other design patterns! You like yours, and dislike the Gang of Four's!", let me tell you — it's kind of true. It's just that most interesting insights about life, and software development in particular, are some sort of generalization — and any generalization can be called a pattern. That's why I wasn't sure what to call them — maybe "design kōans", since their purpose was to make you think rather than tell you what to do.

So, what did they all have in common? Drivers, middleware, aspects and plugins: they were all about enabling the user. To extend the implementation, to augment the functionality, to add to the whole system — there's some sort of empathy to it. Love thy neighbor as thyself (ואהבת לרעך כמוך); and since you're going to be the first neighbor, the first user of your own platform, trying to solve the first problem with it — you'll be the first to feel how convenient it is, or isn't.

And they were all about composing components, not hacking constraints. We didn't try to predict an entire taxonomy beforehand, or solve an entire problem in advance — we solved many small things separately, and then put them together. It's not a particular style or paradigm, really; it's basic problem solving — divide and conquer, understand the problem, break it into small parts, then compose those parts. As a result, the solutions are much more adjustable and adaptable — ready for a future we can't predict.

So the verdicts come in three flavors. Some of those patterns were useful, but aren't anymore; like the abstract factory, or the comparator classes in the strategy pattern — that's a horse we rode to the point where we could build cars. Some of them are still useful, with adjustments; like the factory method. Some encapsulate concepts that are always relevant, like the singleton — although their implementation is another matter; and the template method is both common and harmful.

In fact, I don't mind design patterns; I just think that teaching the solution, rather than the method of arriving at it, is wrong. Reality changes — that's what reality does — and if all we have is the solution, we keep building triangular roofs, because that's what a house is supposed to look like. But if we understand why the roof is triangular, we can say "hey, that was in Europe — we don't live there anymore." And if we're using outdated solutions, it's bad even if they still sort of work.

Patterns, at the end of the day, are vocabulary. Like Alexander's pattern language, they give us names to talk about designs — and that's worth a lot. In the terms of the introduction, they're part of the craft — they're necessary, but not the art. A vocabulary, however, is not an understanding: every pattern in this chapter, old or new, was ultimately about dealing with complexity — deciding where it lives, who sees it, and who pays for it. The kōans worked, because they put the complexity in the right place; and the old patterns didn't work, where they didn't, because they buried it in ceremony.

Which brings us to the deeper question — and the next chapter: what is complexity, where does it come from, and how do we manage it? Not as a catalog of solutions, but as a way of thinking; and, like I promised in the introduction, with some help from other disciplines.