The Plugin Design Pattern
People sometime say there are no design patterns in Python; I don't agree. There are design patterns in *every* programming language - in fact, the term was coined by [Christopher Alexander](https://en.wikipedia.org/wiki/Christopher_Alexander), who was an Architect, so there are probably design patterns in every creative field. What I think these people mean, is that the "classical" design patterns, those described by [The Gang of Four](https://en.wikipedia.org/wiki/Design_Patterns), don't really apply to Python - and that's OK, because they were invented for much more "old-fashioned" languages, like C++ and Java. (Yeah, I know Java is from 1995 and Python is from 1991, so technically Python is older. But Python 2.7 is from 2010, and anyway, Python's *mentality* is much more modern.) ![pause](/media/images/pause.png) Let's talk about a design pattern I came to think of as the Plugin Design Pattern. Like the Gang of Four, I'd like to start with a story: Say you're working on a web-based system that your operations division uses to drive your product. They need the flexibility to dynamically add functionality to that system, so you sit down with them and design the following abstraction: Each ``function`` has a ``name``, a ``type``, and a ``value``; operations can add such functions on their own, using an admin interface, and this dynamically extends the API. So for example, after adding the function ``foo`` with type ``text/plain`` and the value ``Hello, world!``, sending a ``GET`` request to ``/foo`` with the ``Accept: text/plain`` header will return ``Hello, world!``. Neat, right? ![self-extending API](/media/images/the-plugin-design-pattern-1.png) But over time you run into the problem of Evolving Abstractions: the world changes, so your product changes, and your system needs to change as well. But more often than not, these changes are little and gradual mutations, and this slow, organic evolution makes your system look like a Platypus. ![Platypus](/media/images/platypus.png) Looking like a Platypus is not desirable. Suddenly, several types need to be supported, so you change ``type`` to be a list of comma-delimited strings. Then, some ``value``s needs to have an expiration date, but you can't migrate the database right now, so you put in that column a JSON with a ``value`` key, and an optional ``expire`` key. But that key needs to be computed dynamically, so you write a small parser that evaluates whatever's between ``[[`` and ``]]``, and provides special functions such as ``now()``, which returns the current timestamp. ![evolving abstractions](/media/images/the-plugin-design-pattern-2.png) And so, after some time, you end up maintaining a full-fledged [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) you've never intended to create, and everybody are quite miserable about it. ![pause](/media/images/pause.png) Here's an idea: let's replace all the textboxes and parameters (well, maybe except the name) with one big textbox, and write actual Python code in it. ![simple plugin](/media/images/the-plugin-design-pattern-3.png) Operations can now add whatever they want, *literally*; and do so in a language that's well-documented and quite popular. Oh, and also general purpose, so the evolving abstractions are contained: check out that second example from before. It doesn't look as bad: ![complex plugin](/media/images/the-plugin-design-pattern-4.png) There are several issues with this, though: 1. **Security** Some people might say that dynamically evaluating user input is insecure - and they would be right, except this isn't really user input, but code written by the operations division; superuser input, if you'd like, and those who get access to the system's admin interface can hopefully be trusted. "OK," you might say, "but you're indirectly giving them control over the entire server: running Python code, they can delete files, which wasn't possible when the system was more statically parametrized." And you would be correct, of course: but operations usually has access to the server anyway, so protecting it from them may be quite superfluous; and let's face it, nobody "accidentally" writes code that thrashes an entire system (yeah, yeah. Famous last words). Moreover, running the server in a virtual machine or a container, which is something you should do anyway, makes this much more... well, containable. 2. **Proficiency** "Python? What, like the snake?" "No, like Monty Python, that British comedy show." "Oh, I don't know. I don't really dig British humor." In other words: can you really expect your users to be able to write valid Python code? But again, these are not end users we're talking about, but the operations division, and they are usually more than capable of writing small snippets of Python code. If they're not, or if they don't dig British humor, you have a much more acute problem in your organization. 3. **Ease of Use** Now that's an actual issue. If 99% of the cases were simple, with only one type and one value, and just 1% of them required something more - maybe refactoring the entire system to make this 1% possible, but making the other 99% much harder to maintain along the way - wasn't a very smart move. I mean, compare writing ``text/plain`` and ``Hello, world!`` to this: ```python type = 'text/plain' def handle(request): return 'Hello, world!' ``` It's not *that* bad, but it's more work, and while your DSL might have had custom, tailor-made functions, reimplementing these in Python might require quite some code. Except, it really doesn't have to: common enough cases can be encapsulated in some package with utility functions and classes, making the code as simple as invoking or instantiating them with the right parameters. Take the previous two examples: ```python import utils type = 'text/plain' handle = utils.Value('Hello, world!') ``` ```python import utils type = 'text/plain', 'text/xml', 'text/html' handle = utils.Expirable('Hello, world!', duration=60) ``` Not that bad, isn't it? It's somewhat more work than before, but it's ``O(1)`` more effort; that is, it doesn't scale with the case's complexity, as long as that case is common enough to have its utilities. And uncommon cases, well, they aren't that common, and there you can do whatever needs to be done yourself - that was kinda the whole point. ![pause](/media/images/pause.png) Of course, this solution isn't always applicable; but that's the reason they're called design pattern*s*, and not *the* design pattern. Letting (super)users write Python code, plugging it into the system, and maybe even providing a package with various utilities for common cases - is an interesting template solution that can be applied to a whole class of problems. As a side note, it also really boosted our product: operations started using Python more, which resulted in more automation; eventually they even started writing their own utilities, which made it possible for them to experiment and pilot ideas much quicker. This in turn lead to a much better dialogue between them and the R&D, because it focused us on features the were actually *needed*, and/or that weren't easy to implement using the existing framework. Also, we could tell upper management we were *agile*, so everybody could smile and nod and murmur in appreciation and pat each other on the back. That was fun.