Writing/The Unbearable Lightness of Coding
Chapter 16 · 40 min read
Modules: from Dynamic Evaluation to Packages
At this point, we have climbed Python's concept hierarchy almost to its top: from basic language constructs like assignment, conditions and loops, to functions, objects, classes and metaclasses. The last piece left to our puzzle is modules and packages: objects, once again, but ones that group and hold everything else together.
We've seen them already, of course — they're the mechanism by which we fetch useful code written elsewhere, whether it's from the standard library of 3rd party ones. We can use an import-statement to make it available:
>>> import datetime
>>> datetime.datetime.now()
datetime.datetime(2024, 1, 1, 12, 0, 0, 0)
Or extract specific functionality with import-from:
>>> from datetime import datetime
>>> datetime.now()
datetime.datetime(2024, 1, 1, 12, 0, 0, 0)
We can extract all their functionality, effectively merging them with our global scope, in what's known as a wildcard import:
>>> from datetime import *
>>> datetime.now()
datetime.datetime(2024, 1, 1, 12, 0, 0, 0)
Module Aliasing
That last option, though, is not the best practice: it makes it harder to tell where exactly any name comes from. I once saw code that did:
join(p1, p2)
And was profoundly confused by what this join meant — until I located the line:
from os.path import join
Which is entirely valid, of course — it doesn't even use import * — but somehow, I grew to recognize the path concatenation function by its full name, os.path.join, so seeing it as a plain join was rather disorienting.
For this reason, it's generally better to import modules as separate namespaces — especially if the components that we're going to use have vague and generic names like join. If it was called join_paths, maybe, it might be agreeable — in fact, in that case, os.path.join_paths might feel redundant. Datetimes are another example of that: writing datetime.datetime feels like we're stuttering, and if a module is particularly deeply nested, writing package.module.submodule.component is quite a mouthful.
So there's a compromise: import-statements can be followed by an as-clause, allowing to import them as a namespace while assigning it to a shorter, more concise name. We can import package.module.submodule as p, and then use p.component; and as for datetimes, I'm particularly fond of this convention:
>>> import datetime as dt
>>> dt.datetime.now()
datetime.datetime(2024, 1, 1, 12, 0, 0, 0)
Which clearly distinguished the module, dt, from the class, datetime. Similar conventions are pretty common in Python: numpy and pandas, despite not having very long names to begin with, are conventionally imported as np and pd, just to make access to them even shorter.
We can also use the as-clause in an import-from statement, like so:
>>> from os.path import join as join_paths
Python gives us all the tools to organize our namespace well — the rest is on us.
Module Files
But what are these modules that we are importing? If we write one ourselve, like:
# example.py
def hello():
print('hello world')
Then we can import it:
>>> import example
>>> example
<module 'example' from '/home/user/example.py'>
And Python will wrap our code in a module object, extract its name and expose its contents:
>>> example.hello()
hello world
In fact, to expose these contents, Python would execute the module's code, binding its name (or whatever we specified in an as-clause) to the resulting globals(). If we were to call our function inside the module:
# example.py
def hello():
print('hello world')
hello()
Then we'd see it's happening right as we import the module, before it's even made available for our use:
>>> import example
hello world
Modules as Scripts
If we're writing code we'd like to run as a script, this is not a problem:
/home/user$ python example.py
hello world
In fact, it's precisely what we want: not just to define a bunch of functions and classes, but to actually use them and make something happen.
But if we're writing a library, to be imported elsewhere, then launching anything substantial upon import can become quite an undesirable side effect. To avoid this, we'd have to separate the two use cases: never import a script, and never directly execute a library. Except there's a trick that lets us enjoy the best of both worlds: you see, each module has a special __name__ attribute:
>>> example.__name__
'example'
When it's imported, its the same as its file name (with .py stripped off); but when it's executed as a script, it changes to __main__ — since, well, the module becomes the main part of our program. If we were to just print it:
# example.py
print(__name__)
Then upon import, we'd get is obvious name:
>>> import example
example
But upon execution, we get:
/home/user$ python example.py
__main__
This lets us detect when our module is being executed, simply by comparing its __name__ to __main__ — so we can tell this scenario apart from a regular import like so:
# example.py
def hello():
print('hello world')
if __name__ == '__main__':
hello()
It's not terribly elegant, but it works: when we import the module, to be used as a library, the last bit that actually invokes our function doesn't run:
>>> import example
# no print
>>> example.hello()
hello world
And when we execute it as a script, the condition holds true, so we get:
/home/user$ python example.py
hello world
Just like we wanted. As an aside, if our script's purpose is rather elaborate, it's better not to squeeze all of it under this condition, but extract it into a function like main, using this odd clause at the bottom just to invoke it — and indeed, that's what you'll find in most Python code that is intended for direct execution.
Modules as Scopes
If you think about, this is quite similar to how class bodies are evaluated: there's no special format, no intricate instructions to follow — it's just code that runs. In fact, as we've just seen, this is true to such a degree that we have to take special measures when we don't want it to run "too much", like when we import a script as a library.
And while the script scenario is particularly common, it's by no means exceptional as a concept. If we want our module to support multiple versions of Python, for example, while also accounting for their slight differences, we might do:
import sys
if sys.version_info[:2] >= (3, 6):
class Ordered(type):
def __init__(cls, name, bases, attrs):
cls._fields = list(attrs)
elif sys.version_info[0] >= 3:
import collections
class Ordered(type):
def __prepare__(name, bases):
return collections.OrederedDict()
def __init__(cls, name, bases, attrs):
cls._fields = list(attrs)
else:
raise ImportError('ordered attributes are not supported for Python 2')
That is, we define different Ordered impleentations depending on the Python version: for 3.6+ it doesn't do anything except assigning its (ordered) dictionary keys to _fields, while for 3.0-3.6 it defines __prepare__ to replace it with OrderedDict, and for any version below it raises an error; all via the same from ordered import Oredered.
Unlike class bodies, however, we're not dealing with some special, dedicated scope: the module's code is evaluated into a standard globals() dictionary. We can see it, in fact, as the module's __dict__:
>>> example.__dict__
{..., 'hello': <function hello at 0x...>}
There's our hello function, along with a whole lot of standard module attributes. To double-check we're dealing with a global scope, we can do:
>>> example.hello.__globals__ is example.__dict__
True
Which assures us that as far as our hello function is concerned, example is its whole world. Its __code__.co_filename and __code__.co_firstlineno attributes, for example:
>>> example.hello.__code__.co_filename
'/home/user/example.py'
>>> example.hello.__code__.co_firstlineno
3
Are all wired accordingly, to the same file that's available as the module's __file__:
>>> example.__file__
'/home/user/example.py'
Dynamic Modules
So really, there's no reason we can't create module objects on they fly — they're little more than a namespace into which some code was executed. To create one, we need to use types.ModuleType and give it a name:
>>> import types
>>> m = types.ModuleType('m')
After which we can use the builtin exec function with its __dict__ as the global scope:
>>> exec('''
... def hello():
... print('hello world')
... ''', m.__dict__)
This produces a result that's almost indistinguishable from an import:
>>> m
<module 'm'>
>>> m.__name__
'm'
>>> m.__dict__
{..., 'hello': <function hello at 0x...>}
>>> m.hello()
hello world
Except, maybe, lacking __file__ and some other standard attributes — but if we need them, we can set them as well.
Module Types
The import statement is more sophisticated than that, however. For starters, not all modules are files:
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> sys.__file__
AttributeError: module 'sys' has no attribute '__file__'
Some modules, like sys, gc and time are built-in — meaning they're implemented as part of the Python interpreter, and their functionality is merely exposed via a module-like interface. Others are more predictable:
>>> import os
>>> os
<module 'os' from '.../os.py'>
But it's unclear how import manages to find them, since they're not located in our current working directory like example.py. Since os is part of the standard library, it's not surprising that wherever its installed, import-statements scan are configured to find it; but the same holds true for 3rd party libraries:
>>> import numpy
>>> numpy
<module 'numpy' from '.../site-packages/numpy/__init__.py'>
Which are located in yet another, separate directory, called site-packages; and besides, what's that weird __init__.py?
The Import Path
Before we talk about that last bit, let's get a clearer picture of how import's lookup happens. We know that it can find built-in modules — but since there's a limited number of those, such logic can be hardcoded. We also know that it checks our current working directory, since it was able to locate example.py; and that it checks some other, standard directories too, like the standard library's root, or site-packages.
If we write a module in some unexpected location, though, like /tmp/m.py, we shouldn't be surprised to see:
>>> import m
ModuleNotFoundError: No module named 'm'
Because unless Python was launched inside the /tmp directory, it has no reason to look there. The way import decides where to look is quite complicated; but on its most basic level, when we're talking about locating files, it's all managed by the sys.path list:
>>> sys.path
['',
'/home/user/.pyenv/versions/3.12.4/lib/python312.zip',
'/home/user/.pyenv/versions/3.12.4/lib/python3.12',
'/home/user/.pyenv/versions/3.12.4/lib/python3.12/lib-dynload',
'/home/user/.pyenv/versions/3.12.4/lib/python3.12/site-packages']
On my Linux (and I'm using pyenv, which we'll talk about later), this is how it looks — on your machine, depending on your operating system and how your Python is set up, it might look different. The purpose of the directories, however, should be rather similar: the empty string stands for the current working directory; then a zip file is checked, whatever that means; then the standard library's directory, where all the standard modules are located; then something called lib-dynload — and finally site-packages, where all the 3rd party libraries are installed. If we were to add /tmp to this list:
>>> sys.path.append('/tmp')
Then m becomes just as importable as everything else:
>>> import m
>>> m
<module 'm' from '/tmp/m.py'>
Simply because now, Python knows that it should look there, too; and if we remove it (or any other directory), trying to import any of its modules stops working.
The order of the directories also matters: as you can see, the current working directory comes first, which is why calling our own modules by standard names (like json) is a bad idea: import json will always find our module first, and we'll effectively lose access to all the native JSON functionality Python's equipped with. So as a rule of thumb (unless we can come up with a better name altogether), we can call our module json_.py instead, using import json_ to set it apart from the standard one.
Strange Imports
Now for the other anomalies. The import system, it turns out, is perfectly happy importing code from a compressed zip archive. If we take example.py, zip it and remove the original:
/home/user$ zip example.zip example.py
adding: example.py (deflated 8%)
/home/user$ rm example.py
Python can still find it, provided this zip is part of its lookup path:
>>> sys.path.append('example.zip')
>>> import example
>>> example.hello()
hello world
This is a pretty convenient method to share a bunch of related files in a single archive, which we can quickly and easily include and utilize, even without extracting it into some standard location.
As for dyn-lib, it holds all the Python libraries compiled into actual binaries, usually for performance reasons. If we look at the math module, for example:
>>> import math
>>> math
<module 'math' from '.../lib-dynload/math.cpython-312-x86_64-linux-gnu.so'>
We'll see that it's actually a compiled, binary .so file — a so called dynamic library, and as such, part of dyn-lib. Python keeps all its binary modules there as a convention, but there's nothing preventing us from writing one ourselves:
#include <Python.h>
// define a method with a standard signature
static PyObject* hello(PyObject* self, PyObject* args) {
printf("hello world\n");
Py_RETURN_NONE;
}
// define the module's methods
static PyMethodDef ExampleMethods[] = {
// method name, function, signature and docstring
{"hello", hello, METH_VARARGS, ""},
{NULL, NULL, 0, NULL} // empty line indicates the array's end
};
// define the module
static struct PyModuleDef examplemodule = {
PyModuleDef_HEAD_INIT,
"example", // module name
NULL, // module documentation
-1, // fixed module size, or -1 for standard modules
ExampleMethods
};
// initialize the module
PyMODINIT_FUNC PyInit_example(void) {
return PyModule_Create(&examplemodule);
}
At this point we can compile it from example.c to example.so with the necessary flags:
/home/user$ gcc example.c -o example.so -shared -fPIC \
-I/home/user/.pyenv/versions/3.12.4/include/python3.12
And we end up with a binary that can be imported and used seamlessly:
>>> import example
>>> example
<module 'example' from '/home/user/example.so'>
>>> example.hello()
hello world
Path Manipulation
To recap, Python can import .py files and .so binaries from various directories and zip archives found on sys.path, which we can edit to expand (or shrink) its lookup horizon. However, that's something we can only do in an interactive session, or when writing our own code — if we'd try to execute a readymade script and it crashes with an ImportError because it can't find its dependencies, are we really supposed to start patching it sources?
To avoid this, Python also keeps track of the PYTHONPATH environment variable. If it's set, it should be a colon-separated list of directories, which will be used to populate sys.path as soon as the interpreter launches, before it attempts to execute any code. So if we have some dependencies in /dir1 and others in /dir2, none of which is a standard location, we can do:
/home/user$ export PYTHONPATH="/dir1:/dir2"
To make sure they are scanned for imports — something we can see even in an interactive session:
/home/user$ python
>>> import sys
>>> sys.path
['', '/dir1', '/dir2', ...]
Environment variables, however, are quite ephemeral; if we forget to set them on some account, or accidentally override them with a different value, our imports will cease to work. A more persistent solution, then, is to add a .pth file to one of the standard locations Python looks through, like site-packages: when a file with this extension is detected, its lines are parsed and added to the path as additional directories to consider. So, if we unset PYTHONPATH and put /dir1 and /dir2 in dirs.pth instead:
/home/user$ unset PYTHONPATH
/home/user$ echo "/dir1\n/dir2" > dirs.pth
/home/user$ mv dirs.pth .pyenv/versions/3.12.4/lib/python3.12/site-packages
/home/user$ python
>>> import sys
>>> sys.path
['', ..., '/dir1', '/dir2']
We'll see them again, although this time at the end, after site-packages itself.
The Modules Cache
As you can see, import lookup is far from trivial; so once a module is found and loaded, it makes sense to cache it. If a.py imports os and b.py, which also imports os — it doesn't make sense to look for os twice, so Python keeps a dictionary with all the already-imported modules in sys.modules:
>>> sys.modules
{...,
'sys': <module 'sys' (built-in)>,
'os': <module 'os' from '.../os.py'>,
'numpy': <module 'numpy' from '.../site-packages/numpy/__init__.py'>}
This might lead to some confusion if we change a module's code on disk, re-import it, and expect our changes to take effect. Consider, for example:
>>> open('m.py', 'w').write('x = 1')
>>> import m
>>> m.x
1
>>> open('m.py', 'w').write('x = 2')
>>> import m
>>> m.x
1
The reason for that is that after our first import, m is safely tucked in sys.modules:
>>> sys.modules['m']
<module 'm' from '/home/user/m.py'>
So import notices it and doesn't bother looking for and loading it again — it simply reuses the same object. If we were to remove it, though:
>>> del sys.modules['m']
>>> import m
>>> m.x
2
We see that our changes did happen — we just needed to nudge Python to notice.
Dynamic Module Attributes
The module cache presents another interesting opportunity: if a module replaces itself in this dictionary (relying on the fact that it's stored under its own __name__), then we can put anything there:
# a.py
class A:
pass
import sys
sys.modules[__name__] = A()
Tricking import to return any object of our liking:
>>> import a
>>> a
<example.A object at 0x...>
This can be useful for more than just trolling. Suppose a module needs to initialize a lot of complicated data structures, and as a result, importing it (which executes its code) can take a while — significantly slowing the start time of any module that uses:
# heavy.py
x = ... # takes 1 second
y = ... # takes 1 second
>>> import heavy
# hangs for 2 seconds
Instead, we can resort to lazy loading, initializing the heavy components on demand. It might still take a while to get them ready, there's no way around that — but at least it will only happen if and when they becomes necessary:
>>> import heavy # instant
>>> heavy.x # takes 1 second
...
>>> heavy.y # takes 1 second
...
How can we do something of this sort, given that a module's code is executed before it becomes available? Easy — we replace it with an object whose __getattr__ is primed to load these names only when necessary, and cache their results:
# heavy.py
class LazyLoader:
def __init__(self):
self.cache = {}
def __getattr__(self, key):
if key in self.cache:
return self.cache[key]
if key == 'x':
value = ... # takes 1 second
elif key == 'y':
value = ... # takes 1 second
else:
raise AttributeError(key)
self.cache[key] = value
return value
import sys
sys.modules[__name__] = LazyLoader()
Of course, __getattr__ is not the only behavior we can overload — but it's by far the most common scenario in which I've seen modules replace themselves. After all, this way the imported object still looks like a module — anything else, like making our "module" callable or subscriptable, will violate the principle of least astonishment; users don't expect the name after an import-statement to be used this way, and it's usually a bad idea to surprise them.
So, just like with metaclasses, Python found a way to integrate this use case into the language: modules can simply define a global __getattr__ function, which gets invoked with no further trickery when a particular attribute is missing:
# dynamic.py
def __getattr__(self, key):
return key
>>> import dynamic
>>> dynamic.x
'x'
>>> dynamic.y
'y'
Packages
So, no matter how heavy our module gets, we can always load it piece-wise; but the bigger it grows, the harder it becomes to manage in a single file. We can always break it into several files and put them all in a zip archive, of course — but that's no way to live, which is why import supports a third type of importable: besides sources files and binaries, it can handle entire directories, known as packages.
Don't confuse this with the directories in sys.path — those were the places to be searched when looking for modules; now, we're talking about (sub)directories that can be found in such places. So far, all the modules I've focused on were single files, except one: numpy, being an enormous project, is exactly such a package. That's why its path looked unusual:
>>> numpy
<module 'numpy' from '.../site-packages/numpy/__init__.py'>
Because it's not a numpy.py file in site-packages — it's a numpy subdirectory. And to qualify as a package, a subdirectory needs to have a special file called __init__.py, which is exactly what we see above.
Let's take it for a spin: suppose we want to define two classes, A and B, and we would like to keep them in separate files for tidiness's sake. We can create a package p/, put an (empty) __init__.py file there, and add A in a.py and B in b.py. Then, we can do:
>>> import p.a
>>> p.a.A()
<p.a.A object at 0x...>
>>> import p.b
>>> p.b.B()
<p.b.B object at 0x...>
Seeing as submodules of the same package are often times interrelated, which is why their files are grouped to begin with, let's make B a subclass A:
# p/b.py
import p.a
class B(p.a.A):
pass
It's a bit cumbersome, having to specify p.x.X every time, but there's a subtle point to why this is important. If we would simply have two files, a.py and b.py, then this would work just fine:
# b.py
import a
class B(a.A):
pass
So called local imports got a pretty bad rep in Python 2, because the way they were implemented was pretty broken. Invoking python b.py would work — but only as long as both files were in our current working directory; from anywhere else (python p/b.py), our code would be unable to find a — unless, of course, we'd add p/ to sys.path. And if we were working with many different packages (as 3rd party libraries often are), we would have to add all their parent directories to the path; and if they had subpackages with subsubmodules, we'd have to add all of those as well.
In Python 3, it's not nearly as bad: modules resolve local imports relative to their own directory, so b.py would be able to find a with no problem regardless of the directory we're currently in. Still, there's a bigger problem: if some module names repeat — for example, there's a module named a.py outside of p — then the one to be imported first would prevail:
>>> open('a.py').write('x = 1')
>>> import a
>>> a.x
1
>>> import p.b
AttributeError: module 'a' has no attribute 'A'
So the fact that a package binds all its submodules in its namespace lets our modules have unique fully qualified names, clanky though those may be. And in fact, there's a way to avoid them altogether while still enjoying the protection of a package.
Relative Imports
Specifying such fully qualified names is known as an absolute import. Just like an absolute path, say /home/user/example.py, specifies the whole journey one needs to make from the root of the filesystem to arrive at a specific file, so does such an import specify the full route to some module, but in the format of home.user.example.
However, we also use relative paths like ./example.py (if it's in the our current working directory) or ../example.py (if it's one directory up) — so Python, too, supports relative imports. In p/b.py, for example, we could write:
# p/b.py
from .a import A
class B(A):
pass
That is: "look for a on the same level as you". And if we had a submodule p/utils/c.py where we'd like to have class C inherit from B, we could do:
# p/utils/c.py
from ..b import B
class C(B):
pass
That is: "look for b one level above you".
Note that unlike paths, relative imports don't use slashes, just dots, with each dot bumping us up one level. If we'd want to get C from inside p/tests/utils/test_c.py, we'd do from ...utils.c import C, equivalent to (./)../../utils/c.py (from test_c's perspective).
Also note that these dots can appear entirely on their own: in b.py we could also do from . import a and use it as a module, referring to its class as a.A; and similarly, in test_c.py we could do from from ... import utils.c and use utils.c.C, as well as from ...utils import c and use c.C. There's not one right way to navigate around a package any more than there is one right way to navigate around a filesystem — Python simply lets us walk up and down our structure as we wish, and we can choose whatever makes the most sense to us.
So in addition to protection from name collisions, a package lets us import stuff from directories "above" us (as long as they're part of the same package) — and it lets us do so rather short and elegant way.
Having said that, I'd be remiss not to tell you some people don't like relative imports. Their main complaint is that according to the zen of Python, explicit is better than implicit, so spelling out the full path to each module is always desirable. What's more, if the package structure changes, such relative references can go out of whack in unpredictable ways. There is some truth to it, I guess; but personally, I like them. When I worked on a module, orienting myself this way helps me reason about its position relative to the other components it depends on, rather than having to hold the big picture of the entire project structure in my head. It also helps me notice elaborate imports with many dots, indicating that the cohesion of my package is degrading; I prefer related functionality to stay close to each other, so high fragmentation can prompt me to reorganize my code. That's actually another benefit of using a package — it's more than just a collection of related modules, in that it gives us an opportunity to articulate their overarching interfaces, as well as any subinterfaces between them.
Exposing the Package API
To illustrate this point, consider the package from a user's perspective. So far, we talked about how it lets us split off our functionality in a reliable and maintainable fashion; but for anyone using it, it's actually worse — they have to know and remember where to look for what.
A module, at least, provides a single entry point to access everything; with a package, we have to import p.a for A, p.b for B, and p.utils.c for C. It would be much nicer if we could just import p, and have p.A, p.B and p.C already there, at its root level. Unfortunately, this isn't the case:
>>> import p
>>> p.A
AttributeError: module 'p' has no attribute 'A'
>>> p.a.A
AttributeError: module 'p' has no attribute 'a'
>>> dir(p)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
Note only does p have no A — it doesn't even expose its own submodule, a; and if we use the builtin dir function to inspect what attributes it does provide, we find pretty much nothing except for some standard package attributes. Only after explicit imports does the package realize it has some components:
>>> import p.a
>>> import p.b
>>> dir(p)
[..., 'a', 'b']
There's a reason for this. When we import a module file, Python goes ahead and executes its code — it's the resulting global scope that becomes available to us via its object. When we import a package directory, what exactly code does it have to execute and expose? Importing p.a, p.b, or p.utils.c explicitly is necessary for Python to even allocate a __dict__ and fill it with something, so what did we expect?
But it turns out that when we import a package directory, Python does run something: namely, the __init__.py file that it must contain — which we left empty for now, so we got a similarly empty package. If we put something there:
# p/__init__.py
def hello():
print('hello world')
Then this is precisely what will be executed for and exposed through import p:
>>> import p
>>> p.hello()
hello world
Which presents us with an interesting opportunity. Yes, we can go ahead and write some actual functionality in __init__.py — but we can also hoist other parts of structure up to this level, effectively using this entry point to organize the API we would like to expose, instead of having users dig through our internal files:
# p/__init__.py
from .a import A
from .b import B
from .utils.c import C
And now:
>>> import p
>>> p.A()
<p.a.A object at 0x...>
>>> p.B()
<p.b.B object at 0x...>
>>> p.C()
<p.utils.c.C object at 0x...>
We can even define the __all__ list of exposed names:
# p/__init__.py
# same as before
__all__ = ['A', 'B', 'C']
Which is used by the wildcard import, import *, to determine what names this * means; there might be some internal details defined in __init__.py, which we're not interested to make part of our public API. Like I said, the wildcard import is somewhat discouraged, but strangely enough __all__ is considered a best practice, if only because it articulates our intentions (and is used by some auto-documentation tools to infer the package interface).
This can be useful not just for the top-level package interface, but for the interfaces of its subpackages as well; even if their functionality remains internal, it can still be clearly demarcated. In utils/__init__.py, for example, we can have from .c import C, so any other subpackage that wants to access it needn't go further than utils:
# p/__init__.py
from .utils import C
# p/tests/utils/c_test.py
from ...utils import C
This isn't merely cosmetic: it's not just that we saved writing an extra .c, even though for longer named and deeper nested components it's more noticeable — it's that we articulated what sub-API utils provides to the rest of the package, which hopefully makes the overall structure of our packages clearer (even if only to ourselves).
Circular Imports
One annoying problem that arises with packages and is worth addressing is that of circular imports. Suppose that classes A and B are related in such a way, that each can operate on the other — I don't like overcomplicating examples when it's unnecessary, but to give you a feeling of what I mean, a User might have a get_posts method that returns its posts, and a Post might have a get_author method that returns its user. So, we might have:
# p/a.py
from .b import B
class A:
def get_b():
return B()
And:
# p/b.py
from .a import A
class B:
def get_a():
return A()
However, if we try to import p now, we run into a problem:
>>> import p
ImportError: cannot import name 'A' from partially initialized module 'p.a' (most likely due to a circular import) (/home/user/p/a.py)
Namely, p's __init__.py tries to import A from a, which requires executing its code; but as part of this execution, a.py tries to import B from b. This, in turn, executes code that tries to import A from a again, at which point Python detects a circular reference that would cause an infinite import loop and raises an error.
At first, it might seem a bit intractable. If we search this error, we're most likely to find people ranting about how such circularity indicates we must have organized our code poorly, but from my experience it often makes sense. And besides, there's an easy solution — all we have to do is move any such imports to the bottom:
# p/a.py
class A:
def get_b():
return B()
from .b import B
(And the same for p/b.py). Now, when p tries to import A and evaluates a's code, it still triggers an import of b — but this happens after the A class was defined, so when b tries to import A from a, Python is perfectly happy providing it with the result, despite the overall module still being only "partly initialized".
Note that this only works because the code that uses B isn't actually executed; it's part of a function that hasn't run yet, so the fact it's imported after its definition doesn't matter — by the time it's invoked, its globals (and B among them) will already be complete. If we'd need it elsewhere, like in A's class signature to inherit from it, it's unlikely we'd run into a circular import — not unless B also inherits from A, which doesn't make much sense anyway. The one place it might cause an actual problem is if we'd want to annotate our code, like so:
# p/a.py
class A:
def get_b() -> B:
return B()
from .b import B
Because here, B actually has to be resolvable at the time of get_b's definition, which is impossible. Luckily, Python provides a solution for that, too — we simply need to add one hacky line at the top of our module:
# p/a.py
from __future__ import annotations
class A:
def get_b() -> B:
return B()
from .b import B
__future__ is a special, built-in importable, which doesn't really provide us with functionality; rather, it reconfigures the Python interpreter. As such, it often stores changes that haven't made it into the language yet — in case of annotations, the ability to delay their evaluation until the module's execution has been complete. Since they're not actually critical for the code to work, any unfamiliar name (like B is, at the time of get_b's definition) is stored as a string, which is good enough for type checkers that can resolve it later. We can see this for ourselves:
>>> import p
>>> p.A.get_b.__annotations__
{'return': 'B'} # not a class object
As an aside, this is useful for more than circular references. If we'd want to annotate a class factory method, like Point's from_polar, we wouldn't be able to do this:
class Point:
@classmethod
def from_polar(cls, r: float, t: float) -> Point:
# ...
Since the class is mid-definition and unavailable at the moment its method annotation needs to resolve it; from __future__ import annotations helps in this case as well.
Anyway, circular imports: they're a fact of life when dealing with complicated, interrelated modules, and the way to address them is by making sure that at the time one circular import happens, the other module is defined-enough to service it. In cases when both modules actually need each other's contents at the same time — yeah, something is definitely wrong with our design; but how exactly would we solve this even with both definitions in the same file?
Package Execution
One thing packaging seem to break, however, is module execution. We saw that with if __name__ == '__main__' we can tell whether our code is running as part of an import, or being executed directly — but if we make it part of a package and add a relative import, the latter scenario no longer works:
# p/a.py
from .b import B
if __name__ == '__main__':
print('a')
/home/user$ python p/a.py
ImportError: attempted relative import with no known parent package
Because of the error message, it seems like the fault is with the relative import; but actually, an absolute one wound't work either:
# p/a.py
from p.b import B
if __name__ == '__main__':
print('running module a')
/home/user$ python p/a.py
ModuleNotFoundError: No module named 'p'
The problem is that once we add something to a package, we can't run it as a script anymore — doing so effectively launches it outside its broader context, so it's unable to orient itself and link correctly with the rest of its surrounding code.
This is not fatal, however: such modules can still be executed, they just need to be invoked in a slightly different way. Rather than specifying their path, we make sure their package is importable (e.g. its in our current directory, or on our PYTHONPATH), and then run:
/home/user$ python -m p.a
running module a
The -m option lets Python know we want to execute some module, passing in its fully qualified path; and this, in turns, gives Python the opportunity to load the rest of the package properly for everything to work. In fact, we can even execute the package itself if we add the appropriate entry point: just like having __init__.py makes it importable, so does having __main__.py makes it executable:
# p/__main__.py
print('running package p')
/home/user$ python -m p
running package p
Whenever we run a Python command like pip or pytest, this is actually what happens behind the scenes; it's equivalent to python -m pip or python -m pytest, which in turn runs those packages' __main__.py.
As an aside, the same convention works for zip archives, too: if we add a __main__.py file to our zip, Python is willing to execute it:
/home/user$ echo "print('hello world')" > __main__.py
/home/user$ zip example.zip __main__.py
adding: __main__.py (stored 0%)
/home/user$ python example.zip
hello world
Namespace Packages
If you ever forget to add an __init__.py file to a package, you might discover that this directory is still importable, and resent me for going on for so long about all the intricacies of packages when there was a simpler option all along — so let me explain.
Such a setup is called a namespace package, and it was added in Python 3.3 for a particular purpose. Most of the time, we should still write regular packages: it's really not such a big deal to add an empty __init__.py file, and better yet — a non-empty one, which exposes our API in an orderly fashion; when we're working on code that is complex enough to justify splitting into separate files, exporting its functionality in a single entry point is really worth the negligible effort it takes; without it, we're back to imoprting p.a for A and p.b for B.
But sometimes, we can intentionally sacrifice this clarity for extensibility. Suppose a company, like ACME corporation, decided to open source its various Python libraries — they have their our database, webserver and so on. But these names are rather generic; so they can either come up with separate brands for each project, or simply group them under the acme package: acme.database, acme.webserver and so on. Users, however, might not need all of this functionality — and in any case, in this scenario, it's reasonable to expect them to import each framework in a target way. As for the developers, different teams might be working on different products, and prefer to have their own dedicated repository for each.
In other words, all we're looking for is to group a bunch of modules (or subpackages, more likely) under the same namespace — and preferably in such a way that different parts of it can reside in different directories. The team working on the web functionality might want to use a stable version of the database package almost like a 3rd party library: one that they install in site-packages, while their own framework remains under development, e.g. in /home/user/dev. In this case, importing acme presents a problem: the different locations override each other, since Python can only cache one importable per name.
So, people developed various workarounds to wire everything together — but the different methods were not compatible with each other, and usually required having identical, boilerplate __init__.py files, which was somewhat error prone. Instead, namespace packages were introduced: since there's no unifying API and we don't need a single entry point anyway, importing directories without __init__.py effectively merges them together, without executing any additional code. If we have:
# /dir1/acme/a.py
class A:
pass
In one directory, and:
# /dir2/acme/b.py
class B:
pass
In another directory — then as long as neither acme defines __init__.py, and both directories are part of the lookup path, we can do:
>>> import acme.a
>>> import acme.b
And Python will be able to reconcile their different origins without any one of them overriding the other:
>>> dir(acme)
[..., 'a', 'b']
But this is a very particular use case — not a shortcut for defining proper packages; so I hope you find the long discussion about their nuances justifiable.
The Import System
Having covered the various ways we can import and use modules and packages, we can discuss how Python's machinery actually works. Like the garbage collector, it's pretty complicated — so I'll try to provide a condensed overview, highlight the important bits, and focus on how it can be extended, without getting bogged down by too many details.
Let's start with the import-statement itself. Like we mentioned before, it actually does two things: first, importing the relevant module, and then assigning it to a name. The first part is where all the magic happens, and it's carried out by the builtin __import__ function, so the following statements are equivalent:
>>> import sys
>>> sys = __import__('sys')
>>> import datetime as dt
>>> dt = __import__('datetime')
>>> from datetime import datetime
>>> datetime = __import__('datetime').datetime
>>> from os.path import join as join_paths # doesn't work
>>> join_paths = __import__('os.path').join
The way __import__ works is by first checking the module cache, sys.modules; if the name is there, its jobs is done:
>>> sys.modules['a'] = 1
>>> __import__('a')
1
If not, this is where the heavy lifting begins. The actual importing is managed by two types of objects: finders and loaders. A finder is expected to answer the question "do you know where some module (e.g. 'a') is?" — if it doesn't, the next finder in line is queried, and if none of them does, a ModuleNotFound error is raised. If some finder does know where to find it, though, it doesn't necessarily have to know how to load it — all it has to do is return a module specification, assigning the task to a specific loader.
Given the variety of importables and their locations, this separation makes a lot of sense. One default finder, for example, knows all the built-in module names; another, arguably the "main" one, looks through sys.path to see if it can locate files or directories of this name before handing down its verdict. And we can add our own, custom finders, looking for a module in a database, for example, or checking if it's available on GitHub; all we need to do is implement an object with a find_spec method that receives the module name and some other arguments, and returns an appropriate ModuleSpec (or None, if it's unable to find this module).
Then, loaders come into play: the "main" one, SourceFileLoader, knows how to load modules from source files; but we have others like SourcelessFileLoader, which can load cached bytecode modules, or ExtensionFileLoader, which knows how to load extensions modules like a binary .so file. Again, we can dream up our own loaders — it's just an object with an exec_module method that receives a new, empty module object, and executes the necessary code into it.
The reason I said this design makes sense, is that regardless of where we find the result (e.g. on the filesystem, in a zip file, or on the web), the way we actually load it is a separate concern: it might be a Python file, or a binary library, or even some encrypted format we'd have to decipher — so providing us with the option to mix and match finders and loader is pretty clever. Other times, especially when customizing the import system ourselves, we might implement a single class to do both the finding and the loading — in which case it's commonly called an importer.
A GitHub Importer
To really internalize all this confusing terminology, let's build one ourselves. Suppose we want to import code directly from GitHub; it's probably a bad idea for security reasons, but it's a good exercise. I'll do it in a single class that will double as both the finder and the loader, so we'll be dealing with an importer here; our syntax is going to be:
>>> from github.dan_gittik.example.hello import hello
This should give us the hello function from https://github.com/dan-gittik/example/, which has a hello.py module inside.
To do that, there's one more thing we need to know: when a fully qualified name contains multiple dots, the import machinery is invoked repeatedly on each part. When we import p.utils.c, for example, Python imports p first, then p.utils, and finally p.utils.c — so since we're effectively hacking this notation to describe a URL, we're going to have to fake finding the initial parts (github, github.dan_gittik and github.dan_gittik.example) until we get the complete module name. GitHub conveniently provides the raw file contents at https://raw.githubusercontent.com, followed by /<username>/<repo>/<branch>/<path> — so for the sake of this example we'll assume that the branch name is main, and that the path eventually checks out; for the intermediate imports we'll just load empty pass-through packages:
>>> from importlib.machinery import ModuleSpec
>>> from urllib.request import urlopen
>>> class GitHubImporter:
... def find_spec(self, name, path, target=None):
... if name != 'github' and not name.startswith('github.'):
... return None
... dummy = ModuleSpec(name, self, is_package=True)
... if name.count('.') < 3:
... return dummy
... _, user, repo, *path = name.split('.')
... user = user.replace('_', '-') # my username is dan-gittik, so…
... path[-1] += '.py' # assume this is a python file
... url = '/'.join((
... 'https://raw.githubusercontent.com',
... user,
... repo,
... 'main',
... *path,
... ))
... response = urlopen(url)
... if response.code != 200:
... return dummy
... code = response.read()
... return ModuleSpec(name, self, loader_state=code)
A finder should define find_spec, which accepts the fully qualified module name, its path (if it's a package — more on that later), and the old module (if it's being reloaded — a case that's practically irrelevant). It returns a ModuleSpec containing the name and the loader (in our case, self), as well as some other optional information.
So, for modules other than github or starting with github., we abort; and unless we've accumulated sufficient details to construct a viable URL, we return a dummy package spec. Otherwise, we cache the module's code for the loader to use, and then:
>>> class GitHubImporter:
... # same as before
... def create_module(self, spec):
... return None # use default module creation
... def exec_module(self, module):
... code = module.__spec__.loader_state
... if code is not None:
... exec(code, module.__dict__)
A loader should define create_module, which receives a spec and returns a new module object, or None to have the import machinery create it on our behalf (basically, the same as types.ModuleType(spec.name)); and then, exec_module, which uses the code we've left for it (if it's not a dummy package) to fill up that module's global scope.
To register this importer, we need to add it to the sys.meta_path list — and, lo and behold:
>>> sys.meta_path.append(GitHubImporter())
>>> from github.dan_gittik.example.hello import hello
>>> hello()
hello world
We've just found, loaded and executed code that's not even on our machine!
A GitHub Path Hook
Of course, this is very hacky — it only works for single modules, and encoding a URL as an import path is pretty cringy; so the import system provides somewhat of an alternative. If we rely only on the default importers in sys.meta_path:
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>,
<class '_frozen_importlib.FrozenImporter'>,
<class '_frozen_importlib_external.PathFinder'>]
We'll see that it includes one for builtins (like sys, gc and time), one for frozen modules (basically, regular modules that were compiled into the interpreter to help with its bootstrapping, like importlib itself), and one that uses sys.path. But that last one, PathFinder, doesn't just look through directories and load files — it actually provides us with the opportunity to add another, simpler hook.
What it does, in fact, is invoke whatever's in sys.path_hooks on every entry in sys.path, and see if it knows what to do with it; if it does, it should return the actual finder to use, and if not, raise an ImportError:
>>> sys.path_hooks
[<class 'zipimport.zipimporter'>,
<function FileFinder.path_hook.<locals>.path_hook_for_FileFinder at 0x...>]
That's how Python supports imports from zip files: the zipimporter returns a specialized finder for paths that end with .zip, and only then does the FileFinder (not PathFinder!) look at the rest of the paths. So whatever GitHub repository we want to import, we can add to sys.path (yes, the full URL and everything):
>>> sys.path.append('https://github.com/dan-gittik/example/')
And simply register a path hook that will know what to do with such an entry:
>>> def github_path_hook(entry):
... if not entry.startswith('https://github.com'):
... raise ImportError()
... # do something
>>> sys.path_hooks.append(github_path_hook)
This hook can clone the repository into some temporary directory, and return something like FileFinder(tempdir, (SourceLoader, ['.py'])) to "crawl" over it and load any Python files within it; but that requires having git installed and running some specific commands, so a simpler implemention would just download the repository as a zip file (another nice functionality provided by GitHub) and pass it to a zipimporter instead:
>>> import encodings.idna # necessary for urlopen to download a zip
>>> import tempfile
>>> from zipimport import zipimporter
>>> def github_path_hook(entry):
... # same as before
... repo = entry.split('/')[-1]
... path = tempfile.mktemp() + '.zip'
... response = urlopen(entry + '/archive/refs/heads/main.zip')
... with open(path, 'wb') as f:
... f.write(response.read())
... return zipimporter(path + '/' + repo + '-main') # path inside zip
And suddenly, we can do:
>>> from hello import hello
>>> hello()
hello world
# and just so you believe me it's coming from GitHub:
>>> __import__('hello')
<module 'hello' from '/tmp/tmpjglyrakm.zip/example-main/hello.py'>
Which would work for arbitrarily complex packages, since it brings the entire repository along — and we didn't have to implement any non-standard importers or ugly hacks.
Package Paths
Generally, though, it's not recommended to use __import__ — not only does it look funky, what with its dunderscores, but it doesn't work well for packages:
>>> __import__('os.path')
<module 'os' from '.../os.py'> # that's not os.path
Instead, importlib provides the more elegant import_module:
>>> importlib.import_module('os')
<module 'os' from '.../os.py'>
>>> importlib.import_module('os.path')
<module 'posixpath' from '.../posixpath.py'>
It can even resolve relative imports given their package:
>>> importlib.import_module('.a', package='p')
<module 'p.a' from '/home/user/p/a.py'>
Which brings to our last question: from the import machinery's perspective, what's the difference between modules and packages? The answer is surprisingly underwhelming: they're pretty much exactly the same — packages are just modules that have one additional special attribute, __path__, which simply holds the paths relative to which their imports are resolved. In p's case, for example:
>>> import p
>>> p.__path__
['/home/user/p']
Means that an import like .a would hone in on /home/user/p/a — resolving to the file /home/user/p/a.py, and if it's a directory, /home/user/p/a/__init__.py. We can even trick Python and do relative imports from packages that don't even exist:
>>> import types
>>> q = types.ModuleType('q')
>>> q.__path__ = ['/home/user/p'] # steal p's submodules
>>> sys.modules['q'] = q # and pretend to be a real package
>>> a = importlib.import_module('.a', package='q')
>>> a.A
<class 'q.a.A'> # whaaat
As an aside, most packages will have a single entry in their __path__ list — that is, the directory in which their __init__.py is located; but it being a list is what enables namespace packages to be distributed across the filesystem and still coalesce:
>>> open('/tmp/c.py', 'w').write('x = 1')
>>> q.__path__.append('/tmp')
>>> c = importlib.import_module('.c', package='q')
>>> c.x
1