Writing/The Unbearable Lightness of Coding
Chapter 23 · 21 min read
Workflow: How to Write a Python Project
Until now, we've been talking about various Python features in separation — so in this chapter, I'd like to bring some of them together into the practical (and somewhat contentious) question of how a Python project should look.
Obviously, there is the code part: the actual functions and classes that do the work the project has set out to achieve. But there are many other considerations we haven't addressed yet: how this code is tested and documented, how it manages dependencies and distribution, whether it makes use of linting, coverage or continuous integration, and so on.
As always, I'll try to give a levelheaded, impartial overview — and presumably fail, since my personal experience of what works better and what works worse will invariably color how my presentation. Before we begin, then, I'd like to acknowledge that there is no "right" or universally agreed upon way to write a Python project: different people came up with different structures, methodologies and auxiliary libraries to streamline their development process, and you should use the one that makes the most sense in your particular circumstances. Even if it means zipping the souce files together and sending them over email, there's no shame in that — if anything, the fact that Python can work this way is an amazing advantage.
The Package
However our process turns out to be, the first thing we need to do is put our code somewhere. We might be talking about a single file, but usually it's a package: a directory with __init__.py and a bunch of submodules. In this case, I'd like to repeat my advice from before: this is an opportunity to expose our interface properly, hoisting up whatever functionality might be relevant, so users don't have to go digging around our internal structure. If we have:
p/
__init__.py
a.py # defines class A
b.py # defines class B
Then we might want our __init__.py to be:
from .a import A
from .b import B
So that using the package is as simple as:
>>> import p
>>> p.A
<class 'p.A'>
>>> p.B
<class 'p.B'>
This is also a good time to consider our usage patterns in general. If we end up hoisting all of our classes and functions into the top namespace, our package might become a bit overwhelming — if you've ever used Photoshop, you probably know that moment when you launch the program and are greeted by a complicated screen with dozens of buttons. I find that a better approach is progressive disclosure: like in friendlier image editing software, showing just the most basic or commonly used buttons up front, and hiding the more advanced functionality in one submenu or another that can be expanded on demand. In the immortal words of Larry Page, the creator of the PERL programming language: "easy things should be easy, and hard things should be possible." In the full quote, he was actually making a stab at certain other languages: "Java is sort of the opposite of that; it makes everything equally difficult." — referring to common scenarios like printing a file, which in Java would look like this:
BufferedReader br = new BufferedReader(new FileReader("file.txt"));
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
That is, wrapping the path not in one but two dedicated classes, before iterating over its lines. Compare this to Python's print(open('file.txt').read()), and you can really appreciate what simplicity means — while more advanced features, like buffering, are still supported; they just don't show up where they're not needed.
Taking inspiration from this, we should aim to make our interface as concise and gradual as we can; whoever uses our code will cherish not having to import and compose several classes just to get going. So if B is secondary, or a utility, we might put in the a utils/ subpackage, exposing it as p.utils.B — or at least specify in our documentation that the entry point to our code is really justy A, from which we B objects are derived via something like get_b() — it depends on our implementation, of course; in this case, we can leave B in the top namespace just in case someone needs to do something like isinstance(b, B). Note that this is about more than just imports: we're talking about the possible routes in which our abstractions unfold.
A more mundane question might be where do we put this p/; I like to keep it at the root of the project, while some people prefer to put it in an src/ directory. In any case, this brings us to our next consideration: given some code, how do we start working with it?
Dependency Management
If our code is pure Python and only uses the standard library, anyone can just download it, drop it in their import path and go ahead. More often than not, however, our code requires some requirements, or depends on dependencies, or otherwise needs some 3rd party libraries — all of this means the same thing. While we develop our package, we might install them manually as we decide to use them; for our users, however, they should all be installable at once, as part of our broader installation.
The simplest paradigm in this regard, and my personal favorite, is to just use pip. When we need to install something, we might do pip install numpy and pip install click; and when we're ready to share our work, a quick pip freeze > requirements.txt will dump all the currently installed packages (along with their exact versions and subdependencies) in a file, from whence others will be able to synchronize their environment by running pip install -r requirements.txt.
A more elaborate approach would be to define a setup.py file and lay out our installation process there. A minimal file of this sort might look like this:
from setuptools import setup, find_packages
setup(
name = 'example',
version = '0.1.0',
packages = find_packages(),
install_requires = [
'click',
'numpy',
],
)
A simple call, in which we specify our project's name, version, code (conveniently collected for us by find_packages(), since it's a bit of a hassle to list it properly in a format that setuptools would understand), and the requirements. Note that this specification will actually install the latest versions of numpy and click; this is not necessarily what we want (or at least, not something we can guarantee will work), so we should probably add the appropriate version constraints:
from setuptools import setup, find_packages
setup(
# same as before
install_requires = [
'numpy==1.21.0',
'click>=8.0.0,<9.0.0',
],
)
Which will make sure to install the dependencies we expect, even if they undergo a complete overhaul later on. At this point, anyone who acquires our package can run python setup.py install to install it (and all its dependencies) in their standard site-packages/ directory. Or, if they want to use it in "development mode", leaving the code where it is to tweak if necessary (while still installing all its dependencies and making it universally importable), pip install . (notice the dot) will do the job.
This takes more work than a simple requirements.txt file; but as wel shall see, setup.py is useful in other situation as well, so we'll probably end up having it anyway. To combine the best of two worlds, however, we can work with a requirements.txt file like before, while also deriving setup.py's configuration from it, to not repeat ourselves:
import pathlib
from setuptools import setup, find_packages
requirements_path = pathlib.Path(__file__).parent / 'requirements.txt'
setup(
# same as before
install_requires = requirements_path.read_text().splitlines(),
)
Here, we use __file__ to orient ourselves, locating a requirements.txt file right next to setup.py, and then simply reading it into a list of strings — luckily, pip freeze dumps it a format that setuptools finds accetable.
Virtual Environments
There is a problem with our naive approach, however: project1 might depend on version 1.21 of numpy, while project2 requires version 2.0. We can't have both installed at the same time — and generally, it's a bit lousy to dump all the dependencies of all the projects we've ever worked with in the same Python environment, let alone the main one that's shared by the entire system.
So instead, Python supports virtual environments: local copies of Python's directory structure (with their own executable, pip, and site-packages/), which can be generated as simply as:
$ python -m venv .env
Assuming you want it all in the .env/ directory, which is how I usually call it — you're free to choose any other name. Once ready, we can use its particular binaries to install dependencies and run code:
$ .env/bin/pip install -r requirements.txt
$ .env/bin/python
>>> import p # dependencies are available, so import should succeed
However, seeing as it's a bit cumbersome, we would normally activate this environment first, configuring the standard pip and python commands to point to .env:
$ . .env/bin/activate
(.env) $ # virtual environment is now active, so:
(.env) $ pip install -r requirements # goes into .env/.../site-packages
(.env) $ python
>>> import p # it works!
That weird dot at the beginning is a way to source a script, executing it in the current shell on Linux and Mac; on Windows, we would just run .env\Scripts\Activate. This will augment our prompt to let us know the virtual environment is active, until we're done for the day and want to go back to our standard environment:
(.env) $ deactivate # special command introduced upon activation
$ # prompt disappeared - virtual environment is no longer active
I keep each of my virtual environments in the root directory of the project — and since I call all of them .env, I like to pass in the --prompt=<project> flag when creating them, to configure the prefix that indicates which virtual environment is active to something more informative than (.env):
$ python -m venv .env --prompt=example
$ . .env/bin/activate
(example) $ # ready to work on our example project
There are other approaches, too. If we're doing data science, for example, we might be better off installing Anaconda: a Python distribution specifically optimized for scientific programming. It comes with more than 300 libraries (like numpy, scipy and pandas) already preinstalled, as well as a variety of tools for notebook management, visualizations, Excel integration and more.
This can be great to start off quickly, but we still might run into a situation where we want to update a library in one project without affecting the others; so it also includes the conda tool to help with that:
$ conda create --name example
$ conda env list
# conda environments:
#
base * /home/user/anaconda
example /home/user/anaconda/envs/example
$ conda activate example
(example) $ conda update numpy # goes into the example venv; or, we can:
(example) $ conda activate # goes back to the base environment, and:
$ conda install --name example click # also goes into the example venv
As you can see, conda takes a different approach: the virtual environments are stored in one centralized location and managed for us, and it's generally somewhat more polished than Python's builtin pip and venv. For example, if we install a package and then decide we don't need it after all, there's no easy way to clean that up with pip: running pip uninstall might remove the package itself, but not its subdependencies (so it's usually easier to delete the virtual environment and reinstall everything from scratch than go about finding and uninstalling them manually). conda, on the other hand, keeps a history of all the changes we make to the environment, which we can see with conda list --revisions, and revert as simply as conda install --revision=<num>.
Package Management Tools
The reason I like venv is because it's the cleanest, simplest way to solve the problem; the reason I like conda is that it targets a particular audience, and addresses its particular needs with grace. There are, however, many other package management tools — the popular ones at the moment being pipenv, poetry and hatch, as far as I can tell — whose aim is much broader: they attempt to streamline the development process for any and all applications and libraries.
On one hand, they do come with some nifty features. pipenv, for example, also manages virtual environments for us, but does so automatically; all we do is run:
$ pipenv install numpy click
And it creates a (centrally managed) virtual environment for our project. When we want to run our code in this environment, then, we do:
$ pipenv run python script.py # or e.g. python -m p
It also creates a lock file, listing not just our dependencies and their versions, but also their hashes — special strings that guarantee that the exact content of the dependency is the same. This is a security measure, intended to make sure that even if hackers manage to plant malicious code in some version of numpy we happen to use, people who install our library won't blindly download and run it — the fact that the dependency's code changed would be enough to invalidate it, even though its version was correct. In other words, the assumption that once a version is published, its code never changes again is a bit naive — sure, that's how things are supposed to work, but in practice such suppositions are exactly what hackers take advantage of, so it's better to confirm it.
It also creates and manages a Pipfile for us — kind of like setup.py, but in a simpler format known as TOML, which might look like this:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
numpy = "*"
click = "*"
So yeah, Pipenv might look nice, but the reason I don't like it (besides being very slow) is that all these tools work great — until they don't. For example, should we move or rename our project, it will get confused (despite us keeping the correct Pipefile and Pipefile.lock); we need to run special commands to account for that, namely pipenv --rm before we make the change and pipenv install after. Or, if we want a script to call our Python code, we have to change our working directory to the exact location where our environment was created, and run it from there — there's no easy way to tell what its specific virtual environment is as we could do in case of .env/bin/python.
So we arrive at even more elaborate tools, like poetry. We use it like so:
$ poetry new example
$ cd example/
$ poetry add numpy click
It also generates a lock file (in poetry.lock), and keeps our project settings in pyproject.toml, yet another alternative to setup.py (and also in TOML):
[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["User <user@example.com>"]
[tool.poetry.dependencies]
numpy = "^2.0.1"
click = "^8.1.7"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
It's even more complicated, but it sets everything up for having even more features: you can add scripts and conditional dependencies, build and distribute everything with a single command, and so on. Once again, it's great as long as it works, and terrible when it doesn't — a phenomena I'd like to address for a moment, since it has interesting implications on software design.
Imagination and Alienation: Half an Argument
Besides the audacity of calling a tool for something as bland as package management poetry (but then, maybe it's ironic?), the intentions of the library are clearly noble: to take a complicated process many people struggle with, and make it simpler. Using it, however, lets people not care about the underlying mechanism: to give an extreme example, I've met C++ programmers who were so used to pressing F5 in their IDE in order to compile code that they didn't know what a compiler is, let alone how to invoke it manually.
This dependence has two drawbacks. The obvious one is that such tools effectively become life support systems — when they break (and believe me, they do), the clueless programmer will be at a loss, whereas someone who understands the essentials (as I believe a programmer's job title should imply) will know what to do, or at least how to investigate the problem further.
The second drawback is more subtle: my claim here is that such tools lead to a failure of the imagination. Having to deal with reasonable hardship can be a good thing, and given the right attitude (which can be cultivated through education), it's usually not even as hard as it sounds. If you're having trouble with something for the third time already — write a script; and over time, those scripts will accumulate into a system uniquely tailored to your particular needs, hand in hand with your growing understanding of the domain in question, and ability to articulate ideas about this domain.
Here, the counterargument is that this is wasteful, because not only are we reinventing the wheel, but disregarding the accumulated wisdom of all those people that have perfected it — and in some sense, this is true. But seeing as our profession, according to this metaphor, is all about building such wheels, I don't think it's unreasonable to expect us to know how to do so, or to spend time studying various methods of wheel building and integrating their insights into our own style. That is, after all, how innovation happens; the naive assumption that it requires extreme specialization, so we better leave it to the experts, usually yields minute, evolutionary improvements, not substantial, revolutionary paradigm shifts.
And hey, I'm not arguing that we should get back to programming with ones and zeroes (although understanding how this is done is not a bad idea); nor that we become intimately familiar with every nook and cranny of every technology we come in contact with. But the delegation of something as fundamental as deployment (and, as we shall see in a bit, coding style) is something that strikes me as a particularly bad idea. Some tools increase our agency, and some tools rob us of it; some tools make us love our work, while others let us not care about it. And not surprisingly, those tools that we make for ourselves we tend to like, and be proud of, and learn from, and care about — even if we initially conceived them to care about something less.
Another argument in favor of tools like package managers is standardization: the fact that as people come and go, learning the peculiarities of custom scripts is more difficult than being familiar with one single framework. This is also somewhat true — assuming, of course, that people really are familiar with that single framework, and not just its most basic commands (because that level of familiarity can be achieved quite quickly even for the most unusual system, provided it has minimal documentation — which, as we'll see, it should). More to the point, though, is that there is something profoundly alienating in this way of thinking: it aims to remove people from the equation, making them fungible and irrelevant for the sake of productivity. Such language might sound dramatic, I know — so let me take a break and resume this line of thinking a bit later, after we cover a few more points, and hopefully this attitude will make a bit more sense.
Hatch
In the meantime, I'd like to soften my stance a little, because the truth is, Python itself was developed as such a passion project; and as is often the case, it didn't (and arguably, couldn't) get everything right. Package management is exactly one of those topics that weren't accounted for at the beginning, and grew quite messy with time as a result — compare it to Ruby's gems or Rust's crates, and it should be clear that there's room for improvement. The interesting question is how such an improvement might look.
And while personally I don't use any package manager just yet, I see the Python community converging towards a more organized and thought through approach: that pyproject.toml we've seen before, for example, is supposed to become the new standard as setuptools are phased out, so it's worth familiarizing ourselves with it, too. Besides, more often than not, new standards start out as a myriad of different, experimental libraries, until enough traction is gained to consolidate it in a comprehensive enough way. So I really don't want to speak ill of the developers or libraries such as pipenv or poetry: they're wonderful people doing wonderful work — and giving it away for free, to top it off. Any criticism I make is aimed at attitudes, not individuals; and to articulate the difference I'd like to introduce one final package manager, called hatch.
Like I said, I don't use it myself — but I agree with its philosophy much more. Besides providing a thinner layer over the bare essentials, so anyone can dip below it on demand, hatch seems to be quite self aware about the impossibility of streamlining different projects with different needs; rather than striving to solve it all in some grand, unifying way, it gives central stage to plugins: the ability of users to extend and customize it to their needs (which, incidentally, requires easy access to the bare essentials, so I dare say that the intention behind the overall design is coherent). Where others provide tools and libraries to solve our problems, here the philosophy is more that of a framework, merely helping us solve our problems for ourselves.
Conveniently enough, it uses the new pyproject.toml standard, which presents a great opportunity: for the rest of the chapter, as we discuss various topics related to workflow, I'll describe how to do things both in the older, low-level, setuptools way, and in the newer, tidier approach — as embodied by hatch. Hopefully, it will make me seem less like a crazy primitivist, while still adhering to my (arguablly crazy and primitive) principles; so we get to solve two bugs with one print, as the saying goes.
To bring us up to speed on the commands we've already covered, here's how we would do them in hatch: hatch new example creates s project, hatch env create sets up a virtual environment for it, and hatch shell opens a shell with that environment active (similar to venv's activate script). To add dependencies, we edit pyproject.toml and put them there — hatch simply makes sure they're synchronized to the environment (installing them if necessary) on its next invocation. There's also hatch run, to execute a particular script; and to find out where our virtual environment is located, we can do:
$ hatch run python -c 'import sys; print(sys.executable)'
/home/user/.local/share/hatch/env/virtual/.../example/bin/python
Which, to be fair, was possible in our other package managers too — but I love that they put it front and center in their documentation, and how transparent it is as opposed to, say, some opaque command like hatch env show. It makes you think, doesn't it, so if you didn't already know this trick, it teaches you something new.
When you open the generated pyproject.toml, you might notice that it starts with:
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
Just like poetry's one ended with requires = ["poetry-core"] and build-backend = "poetry.core.masonry.api". This really should drive home how open ended Python's package management is at the moment: the new, formal standard for doing it is specifically designed to let 3rd party projects like poetry and hatch plug into it, implementing a standard without committing to this or that way of doing things. These lines, then, define this implementation (as well as its dependencies, before we can handle those of the project) — there's no endorsement on the language's part, at least for now, of the "right" way to do things.
hatch's way is quite comprehensive: along with this file, it generates a bunch of others, which we'll revisit as we discuss the particular topics to which they pertain. But it's also interesting what it doesn't generate — namely, a lock file, despite there being no apparent drawback to this security measure. Besides not having gotten to it just yet, the truth is that it isn't strictly necessary either: the requirements.txt file now supports hashes in addition to pinning the exact versions (which hatch doesn't do either) — and despite pip being unable as of yet to generate those itself, the pip-tools library provides a simple command to do just that (as well as one to synchronize to generate them from a pyproject.toml file, fixing the versions installed in the current virtual environment). Together with hatch's encouraging support for plugins, integrating this into your project, should you need it, would be relatively easy — and invariably keep you in the loop as to how this is done, seeing as you'll be the one outlining this loop to begin with.
Python Versions
Now that we're up to speed, let's talk about the next hindrance in our development process: Python has a lot of versions, and it's an ongoing project, so they keep releasing more. Installing more than one version on our system was quite messay — let alone more than one minor version, where having python (2) and python3 was still doable. That is, until pyenv came alone.
Again, a simple tool that solves a particular problem: with pyenv install --list, we get all the possible Python versions (along with distributions, such as Jython, implemented in Java, or IronPython in C#); and with pyenv install <version> we download and build them for our machine. After that, we can list all the installed versions with pyenv versions, and see the current one with pyenv version — presumably, it's going to be the default installation that came with our system. The trick is in running pyenv global <version> to change it, or pyenv local <version> to replace it in the current directory (and its subdirectories). This is done by dropping a simple .python-version file with the desired number or distribution, and replacing the python command with what's known as a shim: a thin layer that wedges itself between the command and the actual Python executable — just enough to locate the topmost .python-version file (or the global configuration, if it doesn't find one) and redirect the invocation there, thus letting us control its routing with ease. All we have to do to install the beta version of Python 3.13, then, is:
$ python --version
3.12.4
$ pyenv install --list
...
3.13.0b2
...
$ pyenv install 3.13.0b2
...
$ pyenv local 3.13.0b2
$ python --version
3.13.0b2
And if we want to see what actually happens when we call python:
$ which python
/home/user/.pyenv/shims/python # that's pyenv's shim
Interestingly, conda also supports a few versions of Python, which we can specify by passing the additional e.g. python=3.12 when creating a virtual environment — but really, it's a limited set of options intended to keep Anaconda in line with the last few versions of the language, so if data scientists want to stay up to date they can; it doesn't provide nearly the same resolution as pyenv — as far as I know, nothing does. It's extremely concise and to the point, aiming to solve one problem and solving it exceedingly well, with a design as non-intrusive and self contained as one can imagine.
Package Distribution
So we've written our code, defined our dependencies, and made sure that everything happens in a separate virtual environment with the exact Python version we wanted. At this point, we're ready to release our code to the world, and there a few ways we can do this.
First, there are source distributions: basically wrapping our code in an .tar.gz archive, albeit in a particular format that others can then install with pip. The way this is done with a setup.py is by running the python setup.py sdist command, which creates this archive in the dist/ directory. It also creates an egg-info directory, which ends up inside this .tar.gz — that's the particular format I mentioned, storing all the necessary metadata about our code.
manifest
Versioning
major, minor, bugfix, alpha beta and such
with hatch
Entrypoints
Wheels
Tests
Fixtures
Design for Testing
not TDD