Writing/The Unbearable Lightness of Coding
Chapter 22 · 64 min read
Type Hinting: The Dubiously Good, the Hypothetically Bad, and the Undeniably Ugly
Recent versions of Python saw a lot of commotion around type safety, type hinting, and generally its type system — the set of rules governing how objects of different types can and cannot interact with each other.
When we were talking about inheritance, we saw that Python is a dynamically typed language, meaning its names aren't associated with particular types. As a result, software design is not nearly as limited, and elaborate patterns (like inheritance itself) aren't nearly as necessary. Instead, Python endorses duck typing: the idea that if a function expects an argument to walk like a duck and squawk like a duck, then anything that qualifies will be treated like a duck, no questions asked. If we have:
>>> def add(x, y):
... return x + y
Then we can use it on numbers and strings both; anything that implements __add__ (or __radd__, if it's y), really:
>>> add(1, 2)
3
>>> add('hello ', 'world')
'hello world'
Strong Typing
Python, however, is still a strongly typed language: values of different types don't necessarily mix, unless a logic on how to mix them is defined explicitly. If we try to add a number to a string, we get an error:
>>> 1 + 'hello'
TypeError: unsupported operand type(s) for +: 'int' and 'str'
As opposed to weakly typed languages like Javascript, which automatically cast (or coerce) one type into another whenever the two meet:
> # in nodeJS
> 1 + 'hello'
'1hello'
Sometimes, this can be useful; converting user input (which always comes in strings) is a common scenario in web browsers, for which Javascript was designed, and it makes it as simple as:
> +'123'
123
> !'true'
false
Other times, however, it can lead to confusing situations, which is the premise of the famous "Wat" talk by Gary Bernhardt. It's a short, fun watch — but the gist of it is:
> [] + [] # array plus array is:
'' # empty string
> [] + {} # array plus object is:
[Object object] # object
> {} + [] # object plus array is:
0 # zero
> {} + {} # object plus object is:
NaN # not a number
Which really does beg the question "wat". Besides being bizarre, it can lead to a lot of subtle bugs that don't manifest immediately, but ripple ever weirder behavior throughout the execution. It's a serious drawback, so I understand people who've spent hours deubgging such nonsense, and dislike weakly typed languages as a result; getting a TypeError in these cases, like one does in Python, is objectively clearer.
People who dislike dynamically typed languages, however, are missing a fine point, at least in my experience. Their primary concerns are usually type safety and performance: the first, about how certain sanity checks and guarantees can be made by the compiler when it know each name's exact type; and the second, about certain optimizations that are possible when these details are known in advance.
Type Checking
The fine point is that these considerations can be addressed in a dynamically typed language — there's just no need to enforce it. In this chapter, we'll see how it works for type safety, leaving performance for later.
In principle, we don't have to take Python code at face value, run it in good faith, and possibly crash with an error. A separate program, known as a type checker, can go over our modules, and use type inference to build a map of what value is assigned to what name, how those name are passed around, and so on — coming up with pretty much the same conclusions as those a compiler of a statically typed language might arrive at. Catching situations like 1 + 'hello' is trivial, and harder cases, like:
>>> x = int('1')
>>> y = [x]
>>> y[0] + 'hello'
Can be inferred as well. Whenever they can't, we can use annotations to associate names with their expected types, and a type checker like mypy will integrate this information into its analysis. This approach is called type hinting, because where inference fails, we provide a hint to keep it going — and this way, we really do cover all the possible type safety checks. Even if we're dealing with dynamically evaluated code, a simple variable annotation (added in Python 3.6) is enough to incorporate it:
>>> x: int = eval('[1, "hello"][0]')
Unless, of course, we really don't know what the type of the value might be:
>>> x = eval(f'{[1, "hello"]}[{i}]')
In which case, we want it this way for some reason — so having a compiler refuse to run our code because it might be wrong is really not very helpful. Our first example can thus be annotated:
def add(x: int, y: int) -> int:
return x + y
And if anywhere in our code we have:
# example.py
print(add('hello ', 'world'))
Rest assured, mypy will catch it:
$ python -m mypy example.py
example.py:4: error: Argument 1 to "add" has incompatible type "str"; expected "int" [arg-type]
example.py:4: error: Argument 2 to "add" has incompatible type "str"; expected "int" [arg-type]
Found 2 errors in 1 file (checked 1 source file)
But should we decide that this is what we want after all, we can mark it accordingly:
# example.py
print(add('hello ', 'world')) # type: ignore
And mypy will gloss over it silently. Note how none of this has anything to do with the actual execution: example.py will print "hello world" regardless of what mypy thinks about it. All we need, from a type checker as well as a compiler, is to tell us about problems, so we can fix them — not to block us until we do.
And indeed, the reason compilers can't get over type errors is not out of principle, or to guarantee safety — older languages didn't use types because it was "better" or "more correct", but because they were too low-level to support anything but. A compiler has to know the type of each name, just so it can generate machine code that stores its value properly; and machine code that adds integers is not at all similar to machine code that concatenates strings. In Python, this is all moot: the interpreter does everything on our behalf, and everything is an object anyway — so types are not necessary. If we still want them, they're there for us, and we can make sure they're compatible; but if we don't, we can omit or ignore them, and everything works exactly the same — all of the benefit, none of the trouble.
Compulsive Typing
Only there's plenty of trouble, and it isn't talked about nearly enough. The current fashion in the Python community, and most resources you can find on the topic, are so indiscriminately and uncritically in favor of type hinting that I find it a bit alarming. When Python just started, one of its most liberating features was that it didn't have those pesky, limiting types — and now we're effectively adding them back in.
So before we dive into the syntax and its potential, I'd like to convince you to take all of it with a grain of salt. I'll also admit outright that I think type hints are greatly overhyped; and while that's just, like, my opinion, man — I hope you'll agree that their advantages, such as they are, should be evaluated against their costs, which turn out to be quite significant.
It's a bit of a nuanced argument, going beyond technical considerations of software design and into the realms of psychology and sociology; so let's build it gradually, and start with a recap of what we already know. The claim that dynamically typed languages can't guarantee type safety is dead wrong — we can guarantee it, and we've seen how to do so with annotations. Thinking about it this way, however, implicitly accepts the critics' presuppusition that type safety is indeed important. In other words, yes, we can do it — but the fact that we can do something doesn't mean that we should.
Of course, there is some value in type checking; that's why Python 3.5 introduced annotations to begin with, why typing functionality keeps being extended still, and why I have a whole chapter dedicated to the subject. But all of it comes at a cost; and I find that because type hints are optional, they create an illusion that we can have our cake and eat it, too — like, "hey, if you don't like them, don't use them". Unfortunately, they have an infectious quality to them: when annotated values interact with unannotated ones, the resulting code loses its type information, rendering our type checking void. This leaves us with a nagging feeling of incompleteness — like, we're already using a type checker, so we might as well make it work well. The compulsiveness of the urge varies between people, but it's always there: we feel compelled to annotate those other values, too; and once we do, anything they interact with needs to be annotated as well, so before we know it, we're annotating the entire codebase. At this point, we become so committed to this style that even a single unannotated parameter might feel like a serious violation, jeopardizing our entire effort.
So while it's true that type hints are optional, they're not really much good when they're partial — there's a strong psychological pull towards an all-or-nothing solution. I think this is perfectly illustrated by the Any annotation: many tutorials will tell you that when the type of some value can't be determined, the "correct" way to annotate it would be:
>>> from typing import Any
>>> def f(x: Any):
... print(x)
Or, if we're dealing with a heterogeneous dictionary:
>>> from typing import Dict
>>> def f(d: Dict[Any, Any]):
... for key, value in d.items():
... print(key, value)
Except, all that information is practically useless: annotating with Any is completely equivalent to omitting the annotation altogether (and instead of Dict[Any, Any], a simple dict would suffice). Even if our annotation is "non-trivial", like Dict[str, Any], as long as we're not actively relying on the keys' stringiness, none of this is actually helpful — it just adds clutter to code that would otherwise be exceedingly simple.
Another example is the recently added Self annotation. It was intended for scenarios like factory methods, which return objects of their own class:
>>> class A:
... @classmethod
... def create(cls) -> A:
... return cls()
Normally, this would raise a NameError, because at the time of create's definition, A is still mid-definition (and thus, undefined) itself. If you recall, we solved this problem before by adding from __future__ import annotations, which evaluated unfamiliar annotations as simple strings, like 'A'. Now, we have a less hacky alternative:
>>> from typing import Self
>>> class A:
... @classmethod
... def create(cls) -> Self:
... return cls()
Which is well and good, except a lot of people took to annotating every self in every method like so:
>>> class A:
... def f(self: Self):
... print('hello world')
Because, well, we should annotate self with something, right? But of course not — it's obvious to both humans and type checkers what the first argument of any given method is; so all we've done, really, is wrote the same thing twice, out of pure principle. The same goes for:
>>> class A:
... def __init__(self, x: int) -> None:
... self.x = x
... def __bool__(self) -> bool:
... return self.x > 0
... def __repr__(self) -> str:
... return f'{self.__class__.__name__}({self.x})'
Gee, __init__ returns None, __bool__ returns a boolean, and __repr__ returns a string? I'm sure glad we cleared that up; all these functions would literally throw an error if we'd try to return anything else.
"Alright," you might say, "but that just means people should learn to use type annotations properly". My point, however, is that the pull towards completeness is so strong, people would add useless annotations to their code rather than leave even a single value unannotated, lest it looks unintentionally so — a psychological phenomena, not a technical one, which we'll return to shortly.
The Cost of Typing
In other cases, no matter how advanced our typing skills are, annotations can still greatly overcomplicate our code. Suddenly, a function with three arguments whose signature used to fit in a single line has to be spaced out consdierably, because one of them is Optional[Union[int, str]], another one is Dict[str, List[int]], and yet another is Callable[[Tuple[Float, ...]], Iterator[bool]].
And let me tell you, this cognitive load accumulates. In C++, for example, one of the ways to bypass the limits of type enforcement is to write templates: code that uses generic types like T and S that can be substituted for actual ones (so long as their constraints are respected), thus allowing similar functionality to be shared among different cases. This results in signatures like this one:
template <typename T, typename S>
std::pair<T, S> firstItem(const std::map<T, std::vector<S>>& d, const T& k);
While in Python we can implement the whole function in less than half the characters:
def first_item(d, k):
return k, d[k][0]
Unfortunately, I'm seeing more and more template-like code in Python, simply because annotating everything with the right types is hard. If we insist on doing so everywhere, a completely innocuous function like the one above can turn out to be surprisingly tricky: it receives a dictionary that can map any type T to a list of any other type S, and a key of the first type, T; and it returns a pair of that key's and value's types, T and S — so to express all of this before Python 3.12, we had to do:
from typing import Dict, List, Tuple, TypeVar
T = TypeVar('T')
S = TypeVar('S')
def first_item(d: Dict[T, List[S]], k: T) -> Tuple[T, S]:
return k, d[k][0]
Or, if we'd want to annotate an optional decorator:
from typing import Callable, Optional
def f(decorator: Optional[Callable[[Callable], Callable]]] = None):
...
By Python 3.12, we at least got some cleaner syntax to do the same:
def first_item[T, S](d: dict[T, list[S]], k: T) -> tuple[T, S]:
return k, d[k][0]
type Decorator = Callable[[Callable], Callable]
def f(decorator: Decorator | None = None):
...
Namely, the abilities to declare type variables like T and S as part of function and class definitions (new to Python 3.12); use native types like list and dict instead of special typing classes like List and Dict (available since Python 3.9), or | instead of Union (with | None instead of Optional; 3.10); as well as a type-statement to properly define a type alias like Decorator (3.12).
Like I said, this whole domain is under active construction, and I dare say it's turning out as elegantly as one could hope — and yet, annotating first_item nearly doubled the amount of characters used for this tiny snippet. Granted, I chose this example to serve my point, but it's still a valid one: just take a look at codebases with extensive type annotations and ones without, and tell me if you see a difference in readability.
Designing Around Complexity
People who use types are not blind to this complexity, of course; but my impression is that instead of reconsidering the paradigm, they double down on it, and find ways to design around it. For example, there are various libraries popping up for type-managed objects, from the builtin dataclasses module to popular 3rd party solutions like pydantic, where we can define:
>>> @dataclass
... class A:
... x: int | str | None = None
... y: dict[str, list[int]]
... z: Callable[[tuple[float, ...]], Iterator[bool]]
So as to have simpler signatures elsewhere:
>>> def f(a: A):
... ...
From a software design perspective, however, this is terrible. One of Python's key strengths is the versatility of its builtin types — things like lists and dictionaries let us avoid such dedicated containers, passing in simple values without requiring intimate familiarity with special taxonomies, multiple imports, elaborate conversions and the like.
So when the Zen of Python teaches us that "flat is better than nested", understanding it to mean we shouldn't indent our statements too deeply is only a superficial reading of it; functions that receive structures-of-arguments instead of arguments, and more generally, the proliferation of objects whose sole purpose is to store other objects — all of this amounts to conceptual nesting, making our interfaces and abstractions needlessly layered and complicated.
It also used to be the case that when we weren't sure about some object, we could pass it into the builtin help function to quickly check its type, signature and documentation. Try doing it now with popular type-heavy libraries like FastAPI, and you get:
>>> import fastapi
>>> app = fastapi.FastAPI()
>>> help(app.get)
get(path: typing.Annotated[str, Doc('\n The URL path to be used for this *path operation*.\n\n For example, in `http://example.com/items`, the path is `/items`.\n ')], *, response_model: Annotated[Any, Doc("\n The type to use for the response.\n\n It could be any valid Pydantic *field*
# ... cropped ...
>>> len(str(inspect.signature(app.get)))
12211
More than 12,000 characters — for a function's signature! For comparison, its actual documentation (cropped, at the very bottom) is 300 characters long — because parts of it are actually stored in the arguments' annotations, to be parsed out later by some dedicated documentation-building tool. And hey, FastAPI is free to do whatever it likes — but as far as the principle of least astonishment goes, this is pretty astonishing.
As a last example, I recently had to pass validate(payload or {}) into a function; however, because everything had to be part of a special hierarchy, I figured:
>>> class A(Base):
... def __init__(self, payload=None):
... self.payload = validate(payload or {})
Should do the trick. Unfortunately, it didn't; the codebase relied heavlity on pydantic, so I had to fall in line:
>>> class A(Base):
... payload: Annotated[
... pydantic.Json,
... pydantic.BeforeValidator(validate),
... ] = Field(default_factory=dict)
Except this didn't work either, because apparently a default factory can't be used with a compound annotation, and the way to appease it (after some struggle) turned out to be:
>>> class A(pydantic.BaseModel):
... payload: Annotated[
... pydantic.Json,
... pydantic.BeforeValidator(validate),
... Field(default_factory=dict),
... ] = {} # this is overridden by a fresh factory dict
The particularities don't really matter; my point is that instead of programming in the flexible and tasteful language I grew to love, I find myself appeasing half-baked, declarative semi-languages — which don't even solve that interesting or challenging a problem to begin with.
The Dubious Value of Typing
This last bit is actually quite important. So far, we've seen that despite being optional, type annotations are infectious; that despite all the toy examples being clean and pleasant, actual code ends up being quite complicated; and that this encourages practices that are clumsy, idiosyncratic, and only there to mitigate complexity we've introduced ourselves by insisting on types. So whatever it is type safety gives us, it better be worth all this hassle.
But what does it give us, really? One claim is that it prevents bugs by catching type incompatabilities early — and here, people arguing the point would often add that it's especially useful for large projects, where a lot of people are working on a lot of code, so nobody can keep track of what exactly goes where.
This argument is a bit of a non sequitur, but let's take it for a spin anyway. Python worked fine for more than twenty years without any type checking, and spawned a lot of hugely sophisticated software in this timespan, which works very well without a single hint. Did its developers really spend all that much time on type errors, or trying to figure out what each parameter means?
Consider the bugs type safety catches: most often it's a TypeError, telling us quite explicitly that an object of type X cannot do behavior Y; or it's an AttributeError, telling us that object of type X does not have attribute Y. In both cases we have a clear traceback, showing us where the error happened and whence the value came, so even if we resort to the oldest and most primitive technique of print debugging, it's the kind of issue we can resolve in under a minute.
As for the informative aspect of annotations, while x: int indeed provides us with valuable information about x, it's really an issue of x being an uninformative name to begin with. See if you can figure out the types of the following names: is_active, num_errors, cost, name, messages, name_to_cost, callback, decorator — and in the context of an actual codebase, it gets even easier (for example, the precise signature of the callback would be obvious from its invocation). When a parameter really is too complicated for its name to explain it, adding a comment or a documentation string can clarify as much (and arguably, much more) than any annotation would.
In fact, I'd argue that what large projects really need is good documentation and comprehensive tests — both of which we'll discuss at depth in the next chapter. If we're serious about making our code as clear and correct as possible, types are a weirdly indirect, incomplete and declarative way to achieve this: a lot of errors can occur even when they are perfectly aligned (like division by zero, or indexing a missing key), and a lot of nuances are better articulated in words, or by showing examples (of which tests, incidentally, are an excellent kind). What's more, in a strongly typed language like Python, TypeErrors and AttributeErrors that stem from type inconsistencies are the first thing we're going to catch when we run our test suite — so in effect, this practice encompasses a lot of what type safety guarantees, anyway.
Another (somewhat better) argument in favor of type hinting is that it improves the static analysis capabilities of our code editor. IntelliSense, the clever syntax highlighting and autocompletion that most IDEs provide, does a lot of type inference to figure out what's going on in our program even as we're writing it — and where it falls short, type hints are exactly the help it needs to fill in the gaps. I haven't used this resolution of coloring in the book, but when we write something like this in VSCode:
def f(s: str):
print(s.trim())
The method should be highlighted according to its stature; the fact that it isn't might alert us to a problem, as we recall that in Python, trimming a string is actually called strip:
def f(s: str):
print(s.strip())
What's more, in writing this function I'd probably type s.t, at which point the IDE would suggest to to autocomplete it to title or translate, and I'll realize this isn't the method I'm looking for — so I'll delete the t, list through the possibilities and find strip, avoiding this mistake altogether.
In other words, type hinting lets IntelliSense run type checks in the background, which allow it to highlight and autocomplete code for us, avoiding silly mistakes like typos, omissions of mandatory parameters, or confusion over the order of return values. It effectively tells us if some of our code doesn't make sense — or rather, it tells us when it does, so when it doesn't, we might notice the coloring or completions are off.
This is nice, I'm not gonna lie — but if we're being honest, it's also quite immaterial. As we've seen, the bugs it catches are generally trivial, and besides, I'm not sure that's the engineering culture we should be striving for. The best programmers I've come across were often the most ascetic, using vim or emacs or Notepad++ with barely any plugins even as they worked on operating system internals; and as I keep saying, we landed on the moon using assembly language (and no syntax highlighting at that), so let's not pretend it's more essential than it is. Like I said, such crutches and cushions are nice, and I don't want to to shame anyone who uses them, or ascribe "honor" to anyone who avoids them on principle — dogmatism is immature regardless of the stance taken. But sacrificing readability and software design because the colors look pretty and we don't care to remember what our own code does? Really?
To be clear, the reason I appreciate minimal tooling is not because of "real programmers use a magnetized needle and a steady hand" kind of hubris; I just find that it fosters an accountability for good design. Any tools we use to manage complexity naturally increase our tolerance for it — so unless we're mindful and disciplined, it can compound and spiral out of control. An IDE that teleports us across files is neat, but it can lead us to have a myriad files across differently structured directories, where otherwise we would be forced to come up with a tidier and more cohesive layout. Similarly, relying on IntelliSense to complete our arguments reduces our need for deep interfaces with concise signatures — but those are important for much more substantial reasons than tactical productivity. Knowing what our code does and what its parameters mean is, after all, the very foundation of our enterprise — hardly an annoying chore to delegate or automate. If we do, how will we develop the gut feeling that something is too complicated, and the imagination to rework it in a better way?
Typing Historiography: A Hypothesis
At this point, I'd like to apologize if my analysis was in any way offensive — and risk one more exhortation, before finally moving on to discuss how types actually work.
The reason I feel strongly enough about the subject to go on about it for so long, is that I'm worried we're educating a new generation of programmers that typing is a "best practice" — without them having their own experience in statically typed languages, the shit one has to deal with when using types, and the cumbersome patterns that are borne of desparate attempts to overcome it. Without enough confidence in their own intuition, they might conclude that the fault lies with their competence; instead of blaming their tools, which were supposedly designed by smarter people, they might commit to learning those peoples' methods for handling such tools, where I would much rather see them come up with new and better ones — both cutting their teeth in the process, and expanding our mutual horizon of possibilities as a result. The alternative attitude of just going along with complexity often leads to zealous narrow-mindedness: to people that don't really understand why they're doing something the way they do, but have spent so much time mastering it that they can't admit the sunk cost, and simply write off anyone who thinks differently as an uneducated amateur.
Like I said, the reason compilers provided type safety to begin with was because they had to have type information anyway — in which case, it makes sense to make lemonade and perform any extra checks and optimizations possible. As for inheritance, templates and antiquated design patterns: those were conceived under truly immutable constraints — a reality that simply doesn't apply to Python.
So how come type hinting is so popular? How can so many people be so wrong, and why would I think I'm right? Arguments aside, this is a matter of personal opinion, as we can write excellent software either way. Personal opinions, however, are primarily informed by emotions; any logical explanation we come up with after the fact is usually a rationalization. Looking at the emotional landscape behind such debates, then, can shed light on what's actually going on — but in doing so, we invariably run the risk of projecting our own beliefs and insecurities unto others. That is to say: I have no way of proving the following speculations, and none of them are said in judgement, anyway; my only goal is to provide a way of looking at the question that will help you disengage from existing preconceptions just enough to make your own call.
The truth is, Python was criticized for not having a type system from its very conception; presumably, by people who came from statically typed languages, and, having spent a lot of time mastering them, found it frustrating that their expertise was irrelevant, or that small bugs they were used to relegating to the compiler kept cropping up. This wasn't much of an issue, however, until someone managed to change Guido's mind: the whole thing was spearheaded by a relatively small group of people, one of whom has a Ph.D. on the subject and was working alongside Guido at the time. I imagine good-natured brainstorming between colleagues carries more weight than distant, abstract and often adversarial disputes on the internet — but for one reason or another, that's what got the ball rolling. Interestingly, in the discussions that followed, people whom I consider the most design-minded thought leaders of the community (and whom you might have noticed time and again in the references, so maybe we just carry similar biases) — Raymond Hettinger, David Beazley, Ned Batchelder — were exactly the ones who've expressed concern; and to the degree they've accepted the changes, did so very slowly and cautiously.
In the meantime, a seemingly unrelated story was unfolding in the background: Typescript saw a meteoric rise in popularity, effectively supplanting Javascript in many of the biggest, most influential projects in the web development world. Note that in this case, there are many good reasons for the shift: Javascript is a weakly typed language, and was famously developed in a rush, by one person, over the course of ten days. Typescript, in comparison, was a much more thought through, collective and comprehensive effort, providing a rich set of features way beyond a type system; so, hardly the same situation as we have in Python. Still, I can't shake the feeling that in some people's minds, the two trends got somehow intertwined, and a general feeling that "a proper language should have proper types" began to take hold.
To that, add the psychological pull I described earlier; a tendency I'm inclined to fold into a broader phenomena I call "IDE procrastination". It is my experience that people sometimes try to avoid hard work — but to feel somewhat less guilty about it, they look for creative ways to feign productivity, like focusing on simpler, less urgent tasks. Or, as the term suggests, fiddling endlessly with their IDE: tweaking its plugins and configurations, so that when they do get to write code, they're going to be super-ultra-efficient at it — when in fact, they could have written it ten times over in the time they've spent in preparation. Ensuring type safety feels like this kind of endeavor: figuring out proper annotations is just challenging enough to feel like we're solving something important and making progress, while also being far enough removed from the actual task we're trying to avoid. Together with type hints' infectious nature, and the satisfaction of seeing the type checker's errors go away (however inconsequential those may be), we get a perfect strom: one can spend hours ironing our signatures and defining tidy, type-managed (and completely trivial) classes, instead of developing actual tests, writing sensible documentation, or god forbid, developing the system — all without feeling bad, because type safety is just such a crucial, cutting-edge contribution.
If I sound bitter, it's because I am (and hey, I did apologize in advance); I don't like the direction in which the Python ecosystem is going, and I don't hear too many voices echoing my feelings. If you have a choice of two similar libraries — one of them typed, so you get nicer syntax highlighting, autocompletion, and a clean mypy bill — I don't blame you for going with the typed one. But this consideration is only skin deep: how cumbersome its design might be, how slowly it might develop as a result, how hard it might be to monkeypatch and extend, or how little use it might make of Python's incredible dynamicity — such factors are much harder to quantify in the moment, and only become evident in the long run, after all your code is already annotated itself.
When I first wrote this chapter, it didn't have this rather long rant. But then Python 3.12 came out, and I saw that they added the type-statement I mentioned earlier:
type Decorator = Callable[[Callable], Callable]
Do you realize what it means, to add a whole new statement to a language as terse and mature as Python? Much better and much more important functionality was consigned to the standard library, or even 3rd party packages — contextlib's contextmanager and suppress, functools.wraps and itertools.islice, pytest, click and numpy, just to name a few — and this crap makes it into the core of the language? What's more, it's a purely aesthetic consideration; beforehand, we had to write:
from typing import TypeAlias
Decorator: TypeAlias = Callable[[Callable], Callable]
Which, given how ugly typing is in general, is really not that big a deal. Maybe it's petty, but I find this move incredibly dirty: it effectively cements typing's status in a language that managed to avoid it for so long — because hey, if we have special syntax just for this purpose, it must be really important, doesn't it?
Type Annotations
There; as you might have gathered, I don't like typing — but it's part of Python now, for better and worse, so any self respecting resource on the language should address it, however sincerely I might recommend that you avoid it.
So, how do we annotate names? We've seen three cases for this: first, in function signatures (both the parameters and the return value); then, in class attributes; and finally, in plain, local variables:
>>> def f(n: int) -> str:
... return str(n)
>>> class A:
... n: int
... s: str
>>> n: int = 1
>>> s: str = 'hello world'
We can use all the native types here: bool, int, float, complex, str, bytes, tuple, list, set, dict and so on. And when the value is a container, like a list or a dictionary, we add square brackets to further specify its contents:
>>> x: list[int] = [1, 2, 3]
>>> d: dict[str, int] = {'x': 1, 'y': 2}
Before Python 3.9, the native container types didn't support it, so we had to import equivalent (and capitzlied) classes like typing.List and typing.Dict. Anyway, note how in the case above it's quite easy to infer the containers' types from their contents — but when they start out empty, as they often do, a proper type hint can be crucial for the type checker to know what to expect later on:
>>> x: list[int] = []
>>> d: dict[str, int] = {}
When we're dealing with a tuple, the annotation serves a double purpose — it specifies both the types of its values, and its length, depending on how many such types it holds:
>>> def f() -> tuple[int, str, bool]: # returns a triplet
... return 1, 'hello world', True
And if the tuple is of variable size (but then, we have to make the assumption its values are of the same type, otherwise there really is no way to know its exact content), we do:
>>> def f() -> tuple[int, ...]: # returns any number of integers
... return tuple(int(s) for s in input().split())
If a name can take values of several different types, those can be separated with the bitwise-or operator:
>>> x: int | str = 1 if condition else 'hello world'
A particularly common case of this sort is when the value is optional — that is, has a specific type, but can also be None:
>>> x: int | None = 1 if condition else None
And note how we're using None for its own type; there's no need to figure out where exactly it comes from (i.e. types.NoneType). Just like with empty containers, type hints can be particularly important for such names, since they often start out empty:
>>> n: int | None = None # no way to know it's an int unless we say so
Before Python 3.10, this syntax wasn't supported; combining a few types had to be done with something like typing.Union[int, str], and combining specifically with None was achieved via e.g. typing.Optional[int].
If we expect an object, we can annotate it with object — or, if we know its (presumably custom) class, use that class as a type annotation:
>>> obj: A = A()
If, however, we expect the class itself, using a plain A will make it seem like we expects an A instance, whereas what we really mean is A's own type:
>>> cls: type[A] = A # or just type, if the base class is unknown
If we expect a function, we've seen that typing.Callable can be used as a stand in for the function's type (types.FunctionType) — either plainly:
>>> x: Callable = lambda: 1
Or, if we want to further specify the types in its signature, accompanied by a list of its arguments' types, followed by that of its return value:
>>> x: Callable[[int, int], float] = lambda x, y: x / y
And when the arguments are too dynamic to describe, we can at least specify the result:
>>> x: Callable[..., int] = lambda *args, **kwargs: len(args) + len(kwargs)
Generic Types
As you see, there are some cases that don't map well to type annotations — but people keep figuring out clever ways to articulate them anyway. In particular, there are a lot of situations where the specific types can't be discerened, and yet there is a clear relationship between them. Recall our first example:
>>> def add(x, y):
... return x + y
If we wanted to use it for integers and for strings both, we might have conceived of:
>>> def add(x: int | str, y: int | str) -> int | str:
... return x + y
This description, however, is too lax: it implies that adding two integers might return a string, or that adding an integer and a string is permissible. A better way to approach this would be to say that both x and y should be of the same type, and that the return value is guaranteed to be of that type as well — whatever the actual type might be.
What we need, then, is a type variable, also known as a generic: those T and S we've seen before. In previous versions of Python, we would have done:
>>> from typing import TypeVar
>>> T = TypeVar('T')
>>> def add(x: T, y: T) -> T:
... return x + y
Or, since we're looking to constrain it to strings and integers only:
>>> T = TypeVar('T', bound=int | str)
>>> def add(x: T, y: T) -> T:
... return x + y
Of course, in some cases, we have no such constraints, and the type can be anything at all; a function that takes a list and returns its first element can only be described as:
>>> T = TypeVar('T')
>>> def first(xs: list[T]) -> T:
... return xs[0]
The same can be used in class definitions, like when we have a data structure — say, a priority queue — that aims to hold values of some particular type:
>>> from typing import Generic
>>> class PriorityQueue(Generic[T]):
... def __init__(self) -> None:
... self.items: list[T] = []
... def enqueue(self, item: T, priority: int = 0) -> None:
... # implementation
... def dequeue(self) -> T:
... # implementation
And if we have several independent relationships, like in a class that mimics a mapping:
>>> K = TypeVar('K')
>>> V = TypeVar('V')
>>> class Mapping(Generic[K, V]):
... def __getitem__(self, key: K) -> V:
... # implementation
Note that this is exactly what list and dict (and before them, List and Dict) do, using generic types to accommodate whatever values we actually end up using. Also note that as a result, whenever they begin empty — like our PriorityQueue does — their concrete type can't be inferred, so any variable of this sort will need an explicit annotation, just like empty lists and dictionaries did:
>>> queue: PriorityQueue[int] = PriorityQueue()
This can be achieved somewhat more elegantly through this clever syntax:
>>> queue: PriorityQueue[int]()
And just to prove that regular lists and dictionaries are generic containers, consider that this code works (although here, I find the alternative notation, using their native syntax, to be somewhat more elegant):
>>> x = list[int]() # same as x: list[int] = []
>>> d = dict[str, int]() # same as d: dict[str, int] = {}
In Python 3.12, it became possible to declare such type variables as part of the function or class definition:
>>> def first[T](xs: list[T]) -> T:
... # same as before
>>> class PriorityQueue[T]:
... # same as before
>>> class Mapping[K, V]:
... # same as before
And if we want to constrain them to particular types like before:
>>> def add[T: int | str](x: T, y: T) -> T:
... return x + y
Polymorphic Typing
Object-oriented languages, particularly typed ones, are especially useful due to their polymorphism: the ability to pass in more specific instances that are-a certain superclass. Typing, therefore, respects inheritance: if B is a subclass of A, a name annotated with A accepts a B object (and one annotated with type[A] accepts B):
>>> class A:
... pass
>>> class B(A):
... pass
>>> def f(a: A):
... print(a)
>>> f(B()) # type-wise, this is OK
<B object at 0x...>
This makes sense, because an normal subclass should only expand on its parents behavior, while still being usable anywhere its parents would. In principle, we could do:
>>> class A:
... def f(self, x, y):
... print(x, y)
>>> class B(A):
... def f(self, x):
... print(x)
>>> def f(a: A):
... a.f(1, 2)
>>> f(B()) # type-wise, this is still OK
TypeError: B.f() takes 2 positional arguments but 3 were given
And mypy wouldn't even catch it — but only because nobody really does it, anyway. Like I said, a subclass should maintain its parents signatures to the dot, only adding additional methods (or additional, optional parameters to existing one) — that is, being more specific on demand, but still perfectly viable in the general cases.
An interesting exception to that is numbers: conceptually, an integer is more specific than a float (in the sense that 1 is-(sort-of-)a 1.0); and both are more specific than complex numbers (as in 1.0+0.0j). Technically, they're not actual subclasses — an int doesn't have the hex method a float does — but mypy allows for a certain ducktyping here, specifically for this (and a few other) extremely common cases. It means we can confuse it:
>>> def f(x: float):
... print(x.hex())
>>> f(1) # type-wise, this produces no error, but...
AttributeError: 'int' object has no attribute 'hex'
But this case is niche enough to disregard, especially since the alternative would be to annotate int | float instead of just float in any function that expects a real number (or otherwise, be unable to pass regular integers into it).
More generally, though, we don't want to rely on strict inheritance, but rather on compatible behavior. Think of a function that expects an iterable, whether a tuple, a list, a set, or even a dict (which yields its keys, after all). Luckily, we can do this:
>>> from typing import Iterable # or from collections.abc import Iterable
>>> def iterate(xs: Iterable) -> None:
... for x in xs:
... print(x)
What Iterable cares about is not that xs is-a one type or another, but that it behaves-like one — namely, like an iterable, implementing the __iter__ method. Another way of thinking about it is that it adheres to a certain protocol, and we can articulate such requirements ourselves:
>>> from typing import Protocol
>>> class P(Protocol):
... n: int
... def s(self) -> str:
... ...
Note that s's body is a literal ellipsis (...), because it makes no sense for a protocol to define actual behavior; all it does is specify how any given object should behave — in this case, have an integer attribute n and a method s that returns a string. Then, we can annotate a function like so:
>>> def f(p: P):
... print(p.n + 1)
... print(p.s().upper())
And pass in any object that can do this, regardless of its actual type. All that Iterable does, then, is define (and thus, require) the magic __iter__ method, which should return an object of another protocol — Iterator, defining both __iter__ and __next__.
Incidentally, this is exactly what a generator does, so the way to annotate one (at least, as long as we're using it for iteration) would be:
>>> from typing import Iterator # or from collections.abc import Iterator
>>> def infinity() -> Iterator[int]:
... counter = 1
... while True:
... yield counter
... counter += 1
This can come in handy with a lot of other behaviors as well; and indeed, typing is full of such readymade protocols. Sequence and Mapping, for example, stand for list-like and dictionary-like objects that can be indexed like you might expect — and even more particular behaviors, like SupportsInt (can be converted to integer, i.e. defines __int__) or SupportsAbs (can be passed to abs, i.e. defines __abs__) are provided.
And notice how a lot of these protocols are also generic: while we can use Iterable if all we care about is the object's iterability, we can also use Iterable[int] (or Sequence[str], or Mapping[str, int]) if we intend to use its items in this particular way.
If we use our generator not as an iterator but as a coroutine, though — calling its send to pass in inputs, and possibly expecting a return value from its eventual StopIteration — then Iterator is not going to cut it; we need to use Generator instead, providing it with the types yielded, sent in, and returned:
>>> from typing import Generator
>>> def coro() -> Generator[int, str, bool]:
... x = yield 1
... return x.lower() == 'x'
That's all that Callable does, too, albeit in finer resolution: it requires that __call__ be present, and if the types of its arguments or return value are specified, demands that those be aligned as well. This also means that more complicated callables, likes ones accepting positional-only, keyword-only or variadic parameters, can be described using a custom protocol:
>>> class CustomCallable(Protocol):
... def __call__(self, x, /, y, z=3, *args, w=4, **kwargs):
... ...
>>> def f(c: CustomCallable):
... c(1, 2, 3, 4, 5, a=1, b=2, c=3)
And of course, it's not just the signature that can be articulated — the arguments and return value can be annotated as well, giving us unprecedented control over how our objects should behave if they are to pass the type checker's scrutiny. Some other useful protocols to know in this regard are Pattern (for regular expressions), Match (for the resulting match objects) and ContextManager:
>>> from typing import ContextManager
>>> @contextlib.contextmanager
>>> def cm() -> ContextManager[int]
... print('setup')
... try:
... yield 1
... finally:
... print('teardown')
Which saves you from defining the lovely __exit__(exception: type[BaseException], error: BaseException, traceback: types.TracebackType) -> None | bool yourself.
Type Variance
Unfortauntely, for rather nuanced reasons, this flexibility doesn't always work as we might expect. If we have a name that holds a list[Animal], we might reasonably think it would accept dogs: list[Dog] — and yet it doesn't, because that would allow:
>>> def add_animal(animals: list[Animal]):
... animals.add(Animal())
>>> dogs: list[Dogs] = []
>>> add_animal(dogs)
>>> dogs[-1].bark() # AttributeError: dogs bark, but not all animals do
In fancy typing jargon, this is described as lists being invariant: they, and other mutable containers, can't afford to be permissive about the types of objects they hold, so rather than isinstance(x, Animal) they perform a strict type(x) is Animal check.
Immutable containers like a tuple, however, can play nicer — they are said to be covariant, and a name expecting a tuple of Animals won't complain over receiving a tuple of Dogs. The Sequence and Mapping protocols we've seen earlier, for example, are made to be covariant (so Sequence[Animal] will accept any sequence of Dogs) — but that means they only require behaviors that don't change the sequence, like __getitem__. If we make use of the __setitem__ behavior, too, we're looking for MutableSequence and MutableMapping, and those are invariant — they'll expect exactly the types we specify in them, and nothing else; a tradeoff inherent to type theory in general.
This is actually a bit more nuanced than simply mutability, and harkens back to םור discussion about how a proper subclass should expand its superclass's behavior, in no way making it more limited. If A defines f and B inherits from it and overrides it, B.f should be broader than A.f, as in "usable in more cases" — for example, add new optional parameters, like we mentioned.
But even when we don't change the parameter list, changing the annotations can be similarly significant. Suppose A.f returns an Animal; if we make B.f return a Dog, we increase the number of situations in which it can be used — no longer just a.f().eat() and a.f().poop(), but also a.f().bark(), which wasn't possible before. In other words, moving the return value's type down the inheritance tree tends to broaden the function. If we'd want our generic classes to support it, then, we need to do:
>>> T = TypeVar('T', covariant=True)
>>> class Litter(Generic[T]):
... def get_runt(self) -> T:
... # implementation
This way, a function that expects Litter[Animal] will accept Litter[Dog]:
>>> def priority_feed(Litter[Animal] litter) -> None:
... litter.get_runt().feed()
>>> litter = Litter[Dog]()
>>> priority_feed(litter) # mypy is happy
If we wouldn't specify T as covariant, a type checker would complain that we're attempting to pass Dogs where strictly Animals are expected.
Interestingly, though, this isn't the case with arguments: if A.f would accept an Animal (rather than return it), then making B.f accept a Dog would actually make it usable in less cases: of a.f(Animal()) and a.f(Dog()), b.f would only support the latter. In fact, the exact opposite holds true — if A.f would accept Dogs and we make B.f accept Animals, then we would be making it broader despite moving up the inheritance tree (or rather, precisely because of it).
This is called contravariance, and in cases a generic is used as a parameter annotation, it's what we need if we want inheritance to be respected. For example, a generator's second type variable (let's say S in Generator[Y, S, R]), which stands for the type of the value sent into it (so in a way, an argument), happens to be contravariant. If we have:
>>> def gen1() -> Generator[None, Dog, None]
... dog = yield
Then given g1 = gen1() we could do g1.send(Dog()); but if we have:
>>> def gen2() -> Generator[None, Animal, None]
... animal = yield
Then with g2 = gen2() we could do both that and g2.send(Animal()). This means that if we have a function that expects:
>>> def f(g: Generator[None, Dog, None]) -> None:
... # do something, example got too convoluted to stay narratable :P
We would want it to be able to accept:
>>> f(gen2()) # that's Generator[None, Animal, None]
And for that to happen, we'd have to define S = TypeVar('S', contravariant=True) (if we were implementing Generator, that is). Interestingly, Generator's first variable, Y, pertaining to the yielded value's type, is actually covariant (because, well, it's a return value of sorts): gen2 that yields a Dog is actually broader than gen1 that only yields an Animal, isn't it?
Granted, all of this is pretty confusing — but really, the reasoning is straightforward: when we change the outwards-facing interface of a function, namely its return value, then the more specific we make it, the more use cases can utilize its specificities — so subclasses are welcome. But when we change its inwards-facing interface, that of arguments, then the less specific we make the signature, the better; it means more use cases can invoke it, and superclasses are the way to go. By default, type variables are invariant, meaning they will match types precisely; if we want them to accommodate inheritance, we have to tell them how — covariantly, by going down the inheritance tree and accepting more specific types (for an outwards-facing interface like return values or immutable attributes), or contravariantly, by going up the inheritance tree and accepting more general types (for inwards-facing interfaces like arguments).
I mentioned immutable attributes here because this way of thinking can shed some light on the constraint we started with. Suppose we were implementing a generic container of our own, like the priority queue we've seen before:
>>> class PriorityQueue(Generic[T]):
... # same as before
... def enqueue(self, item: T, priority: int = 0) -> None:
... # implementation
... def dequeue(self) -> T:
... # implementation
What is a function that expects PriorityQueue[Animal] going to do when given a priority queue of Dogs? If it uses dequeue, that would mean T has to be covariant for Dog to qualify as a compatible type; but if another function that expects PriorityQueue[Dog] is using enqueue, and we would like to pass in a queue of Animals, then T would have to be contravariant; unfortunately, it can't be both. In other words, the generic T is part of both the outwards-facing and the inwards-facing interface of the class as a whole, which is tightly linked to the fact that this class is mutable in respect to T: it both ingests Ts and emits them. When it's one or the other, we can widen our scope accordingly; when it's both, we have no choice but to fix it on exactly one concrete type (as is the case here, or with mutable attributes).
This is a pretty advanced topic in type theory, but it keeps popping up when working with generics, so I hope I was able to explain it well. The bottom line is simple: covariant for return values and readonly attribute, contravariant for arguments, and invariant for generics that are used as both, implying mutability. Conveniently enough, in Python 3.12 they've added an algorithm to infer this automatically — so if we're using the new syntax of class[T], these decisions will be made on our behalf; all that remains is to not be surprise when they manifest in actual usage and type checker errors.
More Unusual Cases
Even with generics, protocols, generic protocols and a variety of variances under our belt, there are still a lot of cases in Python that elude us and require even more special treatment. One such case is a function that doesn't return:
>>> from typing import NoReturn
>>> def f() -> NoReturn:
... raise ValueError()
And another one is a class attribute. You might think that this would do:
>>> class A:
... n: int
But this is not actually a class attribute; it has no value, so it ends up in the class's __annotations__ dictionary only:
>>> A.n
AttributeError: type object 'A' has no attribute 'n'
>>> A.__annotations__
{'n': <class 'int'>}
And in fact, it's understood to describe the instance attributes that are to come. If they are set immediately in __init__, it's usually enough to have:
>>> class A:
... def __init__(self, n: int) -> None:
... self.n = n
For the type checker to figure out what's what; but if this is done later (or automatically, as is the case with dataclasses or pydantic models), it's a necessary prognostiaction:
>>> class A:
... n: int
... def set_n(self, n: int) -> None:
... self.n = n
So if we want an actual class attribute, we need to annotate it with ClassVar, and put in square brackets whatever its value's type is:
>>> from typing import ClassVar
>>> class A:
... n: ClassVar[int] = 1
Another unusual case is the situation in which we have a value that can hold one of several literal values, and in particular a constant whose literal value never changes:
>>> from typing import Literal, Final
>>> switch: Literal['on', 'off'] = 'on'
>>> PI: Final[float] = 3.14
Which can be useful for a type checker to guarantee const correctness, or that only a valid set of known, predefined values is passed into a function (a scenario that can also be achieved by defining them in an Enum, and using it as an annotation instead).
Then, there are type aliases and new types. If we have an expression that is cumbersome to repeat, like the optional decorator we've seen before or something like list[list[dict[str, str]], we can define it in a variable:
>>> Decorator = Callable[[Callable], Callable]
>>> Conversations = list[list[dict[str, str]]]
And use it henceforth. However, in older version of Python, type checkers had some trouble figuring out that this is a type annotation, particularly with forward references such as:
Animals = 'list[Animal]'
class Animal:
@classmethod
def all() -> Animals:
# return all animal instances
In which case, it really does look like Animals is simply a name bound to a string; so to mark it explicitly, we would do:
from typing import TypeAlias
Animals: TypeAlias = 'list[Animal]'
# same as before
And the convention stuck even in later versions, because it's helpful to clarify our intent. In any case, the whole practice was rendered moot by the vile type-statement I mentioned earlier, which lets us do:
>>> type Animals = 'list[Animals]'
It even lets us define inline generics, much like we would in functions and classes:
>>> type NativeSequence[T] = tuple[T] | list[T] | set[T]
A similar case happened when people wanted to define a type alias that could be used in runtime — not just for annotations, but to wrap (however symbolically) actual values; for example, a UserID that's actually just an integer. One could do:
>>> class UserID(int):
... pass
>>> def delete_user(uid: UserID) -> None:
... # implementation
>>> uid = UserID(1)
But that creates unnecessary overhead, creating objects that really only exist for type readability — so NewType was introduced instead:
>>> from typing import NewType
>>> UserID = NewType('UserID', int)
>>> uid = UserID(1) # this is fine; it's basically uid = 1
One last special case I'd like to cover is that of a dictionary — an object with a notoriously fluid structure, seeing as we can change its keys (and their values' types) on a whim. If we want to require that a dictionary has exactly some keys with exactly some values, we can do so by creating a TypedDict instance, or defining a subclass of it:
>>> from typing import TypedDict
>>> Config = TypedDict('Config', {'host': str, 'port': int, 'backlog': int})
>>> # or:
>>> class Config(TypeDict):
... host: str
... port: int
... backlog: int
Now, a name annotated with Config will only accept dictionaries with this particular structure; any additional, unrelated keys will cause an error. If we don't mind such extras, and mostly want to make sure that some bare minimum is present, we can loosen the restriction by adding total=False to its definition:
>>> Config = TypedDict('Config', {...}, total=False)
>>> class Config(TypeDict, total=False):
... # same as before
Variadic Typing
And there are still more cases that don't fit any of the above solutions; that's what a dynamic language Python is. Suppose we were writing a decorator:
>>> def double(f):
... def wrapper(*args, **kwargs):
... return 2 * f()
... return wrapper
Here, we have a clear relationship between the signature of f and that of wrapper — namely, they should be the same, even though the latter is expressed with the omnisignature. To encapsulate this dynamic, we need an entirely new concept:
>>> from typing import TypeVar, ParamSpec # used to be in typing_extensions
>>> P = ParamSpec('P')
>>> R = TypeVar('R')
>>> def double(f: Callable[P, R]) -> Callable[P, R]:
... def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
... return 2 * f()
... return wrapper
That is, we capture the signature in a special parameter specification P and a return value type variable R, and then use them to annotate wrapper's args and kwargs (and return value) in a way that a type checker will understand.
Another complication is that decorators can often inject an argument into the function, thus skewing its signature and rendering our parameter specification inaccurate. To mitigate such a case, we need to declare our intent of argument concatenation (adding, for example, a Request object to a webserver handler):
>>> from typing import Concatenate
>>> def handler(f: Callable[Concatenate[Request, P], R]) -> Callable[P, R]:
... def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
... return f(Request(), *args, **kwargs)
... return wrapper
As for second-order decorators, those too can be salvaged with hardcore generics:
>>> F = TypeVar('F', bound=Callable[..., Any])
>>> def multiply(m: int) -> Callable[[F], F]:
... def decorator(f: Callable[P, R]) -> Callable[P, R]:
... def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
... return m * f()
... return wrapper
... return decorator
And if we have a semi-parameterized decorator — this is my personal favorite — here is what we do:
>>> from typing import overload
>>> # for bare invocation:
>>> @overload
... def trace(f: F) -> F:
... ... # this is a stub, so it's a literal ellipsis
>>> # for parameterized invocation:
>>> @overload
... def trace(*, log: Callable[[str], None] = print) -> F:
... ...
>>> # actual implementation:
>>> def trace(f: Callable[..., Any] | None = None,
... *, log: Callable[[str], None] = print) -> F:
... if f is None:
... return lambda f: trace(f, log=log)
... # implementation
That is, define protocol-like stubs for the two different ways in which trace can be invoked — on a function directly, or with the log parameter first — and decorate them with overload, so that the type checkers knows to associate them with the actual implementation later on. Isn't typing just wonderful?
Luckily, the new syntax of inline generics makes it somewhat easier, at least for the more common cases; if we include a type variable (conventionally, P) preceded with two stars, it is understood to be a ParamSpec:
>>> def double[R, **P](f: Callable[P, R]) -> Callable[P, R]:
... def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
... return 2 * f()
... return wrapper
Now, where we have two stars, we usually have one star as well — an indeed, there is such a concept as variadic generic, implemented via a TypeVarTuple. The idea here is that sometimes we might be dealing with multidimensional containers with a variable number of axes, such as:
>>> arr2d: Array[float, float]
>>> arr3d: Array[float, float, float]
And to support this, we need to define the Array class like this:
>>> from typing import TypeVarTuple
>>> Shape = TypeVarTuple('Shape')
>>> class Array(Generic[*Shape]):
... def __add__(self, other: Array[*Shape]) -> Array[*Shape]:
... # implementation
Which effectively lets us capture (and then explode) any number of type variables, providing a guarantee that however many there are, only arrays of the same dimensionality can be added, and they produce a result of the same shape. Note that this is not as well-defined as the rest of the topics we covered, and different type checking libraries might leverage this syntax in different ways — for example, instead of providing the types of each dimension, providing it once in the beginning and using the remaining spaces to lay out its exact size:
>>> D = TypeVar('D')
>>> Size = TypeVarTuple('Size')
>>> class Array(Generic[D, *Size]):
... def __add__(self, other: Array[D, *Size]) -> Array[D, *Size]:
... # implementation
>>> arr: Array[float, Literal[600], Literal[800]] # 600x800 array of floats
In any case, it's the availability of the syntax that matters — and it fits nicely into the new form of:
>>> class Array[*Shape]:
... # same as before
>>> class Array[D, *Size]:
... # same as before
So yeah, in principle, all three cases can be utilized with e.g. class A[T, *Ts, **P].
Typing Nuances
At what point does it become funny when I say that even this is not enough, and we need to keep helping the type checker out? If we have a function like this:
>>> def f(s: str | None = None):
... if s is not None:
... print(s.upper())
Then mypy is generally smart enough to understand that within that context, s is going to be a string and not None; however, should we complicate it a little:
>>> def f(s1: str | None = None, s2: str | None = None):
... if s1 is not None or s2 is not None:
... print((s1 or s2).upper())
This branching logic would be too much to handle; to accommodate the type checker's limited understanding, we'd have to rewrite it along the lines of:
>>> def f(s1: str | None = None, s2: str | None = None):
... if s1 is not None:
... print(s1.upper())
... elif s2 is not None:
... print(s2.upper())
Other times, we're not dealing with None but with different types, and might do something like:
>>> def f(x: int | str):
... if isinstance(x, int):
... print(x + 1)
... else:
... print(x.upper())
And again, mypy is able to follow and infer correctly that x must be an integer in the then-clause and a string in the else-clause. This is called type narrowing: various clues that let the type checker hone in on the specific type relevant to the circumstances. It's actually surprisingly clever — it will know, for example, that:
>>> def f(x: int | str):
... if isinstance(x, int):
... return
... # x can't be int at this point
Or that:
>>> def f(o: object):
... t = type(o)
... if issubclass(t, A):
... # o is-a A
But when it doesn't, it is on us to tell it so explicitly, using assert-statements:
>>> def f(o: object):
... assert isinstance(o, int)
... # o must be an integer
Ugly though it may be. We can also use the special typing.cast function, which has no effect at runtime, but in the type checker's eyes coerces one type into another — even letting us change them midway:
>>> from typing import cast
>>> def f(o: object):
... o = cast(int, o)
... # o is an integer
... o = cast(str, o)
... # o is an string
Incidentally, this used to be the way we dealt with decorator annotations before ParamSpec came along:
>>> F = TypeVar('F', bound=Callable[..., Any])
>>> def double(f: F) -> F:
... def wrapper(*args, **kwargs):
... return 2 * f()
... return cast(F, wrapper)
That is, giving up on annotating wrapper, but then casting it forcefully at the last moment into whatever signature f had.
Another scenario where we might want to help the type checker narrow down types is when we have some predicate function, like is_uppercase — one that returns a boolean, but whose result has certain implications on the argument's type (in this case, it being a string). Normally, if we do:
>>> def is_uppercase(x: int | str) -> bool:
... return isinstance(x, str) and x.isupper()
>>> def f(x: int | str):
... if is_uppercase(x):
... # type checker doesn't know that x must be string
So instead of annotating is_uppercase's return value with a boolean (although that it is), we substitute it for a TypeGuard that indicates what the argument's type is if this boolean turns out to be true:
>>> from typing import TypeGuard
>>> def is_uppercase(x: int | str) -> TypeGuard[str]:
... return isinstance(x, str) and x.isupper()
>>> def f(x: int | str):
... if is_uppercase(x):
... # type checker knows that x must be string!
Type Checking and Stubs
This chapter is getting pretty long, in part because typing is such a sprawling subject, and in part because I find it important to show, not just tell. If you look back on the examples so far — despite them being extremely simplified and laconic, their typing information can get incredibly cumbersome, fighting against an impossible tide and effectively juxtaposing an additional language on top of the existing syntax, cluttering it with opaque symbols. Still, it's also utterly brilliant — the lengths they're willing to go to and the checks they're able to perform are no laughing matter, and I deeply respect the software design that went into it, even if I think it harms the software design of everything that uses it.
So before we finish, I'd like to tell you about what I think is the worst feature of the typing library, and the one I think is the best, and then talk about how they actually make all this magic happen and wrap up.
The ugliest, most horrible idea to come up from this mess is TYPE_CHECKING: a flag that is true during a type check and false at runtime. The main motivation behind it, as I understand, was to allow circular imports, which annotations so often require — something we were able to achieve with from __future__ import annotations by simply placing them at the bottom (and personally, I like it more, since it places the least important information last, rather than starting the module with a long and rather irrelevant collection of import-statements). Still, if you want to do it the typing way:
# a.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from b import B
class A:
def f(self, b: 'B') -> None:
pass
# b.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from a import A
class B:
def f(self, a: 'A') -> None:
pass
And notice how we can't use the actual A and B, because in runtime these names are not going to be defined at all, so any code that expects them to will crash.
The reason I find this so ugly, is that this escape hatch quickly became the go-to place for a myriad of increasingly complicated ad-hoc hacks. A Queue, for example, didn't support generic subscripting for a while, so code like this:
from queue import Queue
class A(Queue[int]):
# implementation
Would crash. The solution?
from typing import TYPE_CHECKING
from queue import Queue
if TYPE_CHECKING:
BaseQueue = Queue[int]
else:
BaseQueue = Queue
class A(BaseQueue):
# implementation
And if it's a generic:
from typing import TYPE_CHECKING, Generic, TypeVar
from queue import Queue
T = TypeVar('T')
if TYPE_CHECKING:
class BaseQueue(Queue[T]): pass
else:
class BaseQueue(Generic[T], Queue): pass
class A(BaseQueue):
# implementation
Nothing like conditional class definition with multiple inheritance to start your day with; not to mention struggling with situations like this one:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from _typeshed import SupportsRichComparison
def f(x: SupportsRichComparison) -> None:
# implementation
Which has to be under TYPE_CHECKING because SupportsRichComparison is actually undefined at runtime — meaning this code will crash with a NameError, so we need to add from __future__ import annotations after all.
As for my favorite typing feature, it's called a stub file: kind of like a C header, declaring all the names and their types without providing any of the implementation, which means that in the file where we do provide the implementation, we can rid ourselves of all this clutter altogether. Its primary intent was to allow a non-intrusive way of annotating older libraries without having to rewrite their code — but I don't see a reason not to use it as the de-facto standard for adding types to everything, far from sight and far from heart (unless, that is, you still think that embedding it in your code makes it more readable).
The way we create a stub file for e.g. a.py is by creating a similar a.pyi next to it, where all we put is hollowed out names, signatures and class definitions, like so:
x: int
def f(s: str | None) -> None:
... # literal ellipsis
class A:
x: bool
def __init__(self, x) -> None:
...
In fact, if a.py already defines x, f and A, we can use mypy's stubgen tool to autogenerate a stub draft like this one:
from typing import Any
x: Any
def f(s: Any) -> Any: ...
class A:
x: Any = ...
def __init__(self, x) -> None: ...
For us to edit and fill up with actual types. There are smarter tools, like monkeytype, which can actually run our code (or our tests) to generate a stub file with the types we use in practice rather than Anys; but regardless, once a stub file is present, we can use mypy's stubtest to check if it conforms our code, revealing if there are any inconsistencies between how we defined our types and what our code actually uses.
Implementing Type Checking
Having said so many bad things about typing, I thought I'd finish the chapter on a more positive note. Say what you will about its philosophy, it is an engineering marvel — because the new syntax notwithstanding, most everything we've seen is implemented over Python's regular data model, ironically taking its versatility and dynamicity to new heights. Annotations, after all, are just objects, and a type checker like mypy is just a library that looks at them and draws some conclusions, however advanced its metaprogramming capabilities are.
So how does it work? Obviously, it would be impossible to cover all the tricks typing had to come up with, but as per tradition, I'd like to give you a taste. Even when we see something like list[int | str] or a Protocol, ask yourself how can such a construct do what it does — in the first case, maybe by defining a metaclass to overload __getitem__ and __or__ (or using the class's __class_getitem__ if it's just the former), and in the second, by crawling over object __dict__s and function signatures, making sure all their bits match.
A particularly interesting case is the behavior of generics. When we do:
>>> class A(Generic[T]):
... pass
What does it mean? Let's investigate:
>>> A.__bases__
(<class 'typing.Generic'>,)
>>> dir(A.__bases__[0])
[..., '__class_getitem__' ...]
OK, so it adds __class_getitem__ to A's inheritance tree, letting us do something like A[int] later; but what is that?
>>> A_int = A[int]
>>> type(A_int)
<class 'typing._GenericAlias'>
>>> dir(A_int)
[..., '__args__', ..., '__or__', '__origin__', ...]
So we see the familiar __or__, allowing for A[int] | None to compound further; but what are the other two attributes?
>>> A_int.__args__
(class <'int'>,)
>>> A_int.__origin__()
<class 'A'>
So the type arguments (coming from __class_getitem__) are stored as a tuple (in case we have a situation like A[int, str, bool]), and we keep a reference to the original class, too. Then, if we do:
>>> class B(A[int]):
... pass
>>> B.__bases__
(<class 'A'>,)
This information magically disappears; after all, what we mean in practice is that B inherits from A, not from some _GenericAlias. The actual parent, necessary to preserve type information, is stored elsewhere:
>>> B.__orig_bases__
(A[int],) # that's our _GenericAlias from before
How can this be? Well, if you recall our chapter about metaclasses, we touched on the pretty niche method (which we'd see in the full dir(A_int) if I hadn't cropped it for dramatic effect) called __mro_entries__ — it was actually added for this express purpose. When such a generic alias is inherited, it replaces itself with the original class, sneakily adding itself back in __orig_bases__ instead. There's similar patchwork happening when we do:
>>> a = A[int]()
>>> type(a)
<class 'A'> # the int went away; however:
>>> a.__orig_class__
A[int]
This time, by overloading something like the metaclass's __call__ or the class's __new__ to, once again, switch the original class back in at the last moment, and keep the type-informed alias in a different attribute. These attributes, then, are precisely what a type checker needs to determine whether types are being used in a compatible fashion: first it will notice Generic in the object's MRO, then hone in on the specifics via __orig_bases__ or __orig_class__, and then run a bunch of type theory algorithms to see if given these annotations the code makes sense, accounting for such things as protocols, variance and so on. This is also the reason type checkers can't really handle unannotated or Any names: lacking this information, their abilities to do static analysis like parsing the program's abstract syntax tree and traversing it for conclusions are quite limited (although these, too, are done: consider how after an if-statement like x is not None, mypy knows that inside this context, the | None part of the annotation can be discarded).
Still other scenarios can be hijacked by other clever means: for example, the isinstance and issubclass calls can actually be the magic __instancecheck__ and __subclasscheck__ methods (although mypy usually uses the more advanced __subclasshook__). Imagine, for a second, we were implementing a union of types:
>>> class Union:
... def __init__(self, *types):
... self.types = types
... def __instancecheck__(self, value):
... return isinstance(value, self.types)
This would effectively trick isinstance(x, Union(int, str)) to return not whether x is an instance of Union, as it normally would, but whether it's an integer or a string:
>>> u = Union(int, str)
>>> isinstance(1, u)
True
>>> isinstance('hello world', u)
True
Incidentally, the UnionType that results for e.g. int | str does something similar (albeit by different means), which means that instead of:
>>> isinstance(x, (int, str))
We can now use:
>>> isinstance(x, int | str)
And there's my silver lining, anyway; this is the only typing feature I actually use in my own code, when given the choice. All's well that ends well, I guess.