Writing/The Unbearable Lightness of Coding
Chapter 20 · 19 min read
The Walrus Operator: the Syntax and the Drama
One things we haven't covered about assignment, and I promised we'd get back to, is the walrus operator, :=.
I decided to split it into a separate chapter so we can elaborate on it a bit more; not because it's so useful, but because it goes beyond syntax, well into software design, programming philosophy and the climate of the Python community, and provides quite a few interesting lessons.
The Syntax
The assignment syntax we learned so far was all statements: code that changes the program state, but has no value in itself. Python 3.8 added what's known as assignment expressions — code that both assigns a value, and evaluates to the value assigned, like in Java or C. Instead of writing:
>>> x = list(range(100))
>>> if len(xs) > 10:
... print('too many items:', len(xs))
too many items: 100
Which would have to calculate the length of the list twice, we can now write:
>>> if (n := len(xs)) > 10:
... print('too many items:', n)
too many items: 100
That is, capture len(xs) into n, while still being able to use its value as part of a broader expression. This might seem like a silly example, and definitely not reason enough to add a whole new construct to the language — so let's look at some more substantial examples from the real world. When we talked about indefinite loops:
>>> while True:
... data = connection.recv()
... if not data:
... break
... connection.send(data)
We discussed how contentious their structure is, even though it lets us avoid code duplication (specifically, writing data = connection.recv() before and inside the loop). Now, we can do the same thing with no while-true, and in just two lines:
>>> while (data := connection.receive()):
... connection.send(data)
We also covered regular expressions, in which the first thing we do is a match a string to pattern, and then — if there's a match — do something with it, like extract the matched groups:
>>> match = re.search(r'(.*) = (.*)', 'x = 1')
>>> if match:
... key, value = match.groups()
Well, how about:
>>> if match := re.search(r'(.*?) = (.*?)', 'x = 1'):
... key, value = match.groups()
Or, suppose we want to wrap a list of values in a class, and also filter them according to some method or attribute that's only available in their object. We can do it like this:
>>> result = []
>>> for value in values:
... a = A(value)
... if a.is_valid():
... result.append(a)
Which requires a full-fledged loop, seeing as in a list comprehension we'd have to instantiate each object twice:
>>> result = [A(value) for value in values if A(value).is_valid()]
Because the if-clause is evaluated before the the iteration's value, so we can't use it in the condition. That is, unless we can create it there and then:
>>> result = [a for value in values if (a := A(value)).is_valid()]
Not the most readable code, perhaps — it takes a moment to realize what a for value in values means — but also not the worst; and once you get used to it, arguably more elegant than a trivial five-line loop (which, personally, I prefer).
So to recap: the walrus operator lets us condense code, which would otherwise require an assignment and a followup, by embedding that assignment as an expression in that very followup. It makes code a bit harder to follow, since normally we'd define our values and then use them, and here we're defining them as part of their usage — but a counterargument might be that in simple and/or common enough scenarios, bundling the two is tolerable, and definitely makes the code shorter. This, in turn, makes the code easier to follow, so there must be some reasonable balance we can strike.
Fibonacci Fun
Of course, we can also abuse the walrus operator; it provides tremendous opportunities to terrorize fellow programmers. To give an example, recall how nice and clean Fibonacci's function looks in Python, thanks to parallel assignment:
>>> def fib(n):
... a, b = 0, 1
... for _ in range(n):
... a, b = b, a + b
... return a
And how, while we can write it in one line:
>>> def fib(n):
... return n if n <= 1 else fib(n - 1) + fib(n - 2)
We probably shouldn't, because its exponential runtime complexity makes it so unbearably slow that we had to write a memoization decorator to make it usable.
Well, what if I'd tell you there's a way to implement it in one line, while also using the iterative rather than the recursive approach, so its runtime complexity remains linear? Consider how we can spell its values out manually, using a list and the walrus operator:
>>> [
... (a := 0),
... (b := 1),
... (b := a + (a := b)),
... (b := a + (a := b)),
... (b := a + (a := b)),
... (b := a + (a := b)),
... (b := a + (a := b)),
... ...
... ]
[0, 1, 1, 3, 5, 8, ...]
So what if we were to do:
>>> def fib(n):
... return [(a := 0), (b := 1), *[(b := a + (a := b)) for _ in \
... range(n)]][n]
That is, create a list with 0 and 1, and then explode a sublist which keeps building up on top of those values, reassigning them as it goes, until n — only to finally be index the result at its th place? The space complexity here is worse than in the original implementation (as it accumulates n numbers, rather than just keeping the relevant two), but it runs fast, and fits in one line (formatting limitations notwithstanding)!
Except it's horrible, of course; show it to someone without this breakdown, ask what it does, and watch their brain wrinkle. Personally, I like such quirky code — it's nice to stretch our imaginations every now and then, even if we don't necessarily want it to end up in production. I also like it because it annoys the purists — you know, the people who find it infuriating that such a thing is even possible, and because of whose righteousness we don't have constructs such as goto.
The Origins of Walrus Noteriety
Assignment expressions originated in ALGOL, which used the same walrus-looking operator, :=, for all assignment. The idea then carried over to C — but, seeing as C was inspired primarily by B (which used a plain = for assignment, and didn't have assignment expressions), it was decided to marry the two concepts, keeping the shorter syntax, but equipping it with this additional functionality. It turned out to be a terrible idea, because when people were trying to write a condition like this:
if (x == 1) {
// ...
}
A small typo, namely omitting one =, would result in perfectly valid code:
if (x = 1) {
// ...
}
That is, assign 1 to x, and if the value of this assignment is true (which it always is), make the condition happen. Clearly, a logic very different from the original intention — and a minute detail to notice, leading to very sneaky bugs. So much so, in fact, that programmers started writing:
if (1 == x) {
// ...
}
Just to mitigate potential typos, relying on the fact that assignment to a constant (1 = x) would result in a clear compilation error. It's somewhat jarring — "if, to 1, x equals" — so it became known, quite aptly, as Yoda notation. I think it's also part of the reason some people grew to dislike assignment expressions so much — just try spending a few hours debugging completely valid code, only to discover you missed a single character.
Note that in Python, this problem is avoided altogether by having different syntax for the different cases. = can only be used in assignment statements, so there's no danger of confusing it with anything that expects an expression, which might contain ==. Still, like with Dijkstra's "Goto Considered Harmful", a vague sense that some smart, important people disliked this idea clung in the air long after the history itself was forgotten, and without its full context, post-rationalizations began to appear, like claiming that this:
>>> while (data := connection.receive()):
... connection.send(data)
is utterly unreadable code, or one that violates the single-responsibility principle, since it does two things at once. But guess what — this code is readable enough, and we have no problem with x, y = 1, 2 doing two things at once. for x, y in a actually does at least four: defining a for-loop, assigning a value to x, assigning a value to y, and passing a to iter (which calls its __iter__ and then its __next__, which might do a hundred things more). That is to say, the whole point of abstractions is to do more work with less code — to do one thing, and one thing only, we'd have to be working with binary data and assembly code.
Duff's Device
In fact, by an large, figuring out how to use an existing tool (like assignment) for a new purpose (like condensing an indefinite loop) is one of the main ways by which innovation happens. It's called exaptation — a name coming from evolution, where certain traits might develop in an animal that perform a function not originally "optimized for" by natural selection. For example, we suspect wings evolved to regulate body temperature, and then turned out to be useful for flying — it doesn't really make sense that flight was their original purpose, seeing as it would take a long time to perfect while being useless to countless generations in the meantime, which is not how evolution works.
To give another fun example of such thinking (and to further annoy the purists, if any of them are still reading this book), consider Duff's device. In C, copying data from array from to array to might look like this:
do {
to[i] = from[i];
} while (++i < size); // like (i = i + 1) < size
But this code can be made more efficient, if we know that the array size is divisible by e.g. 4. In that case, we can do:
do {
to[i++] = from[i]; // like ++i, but evaluates the value before assigning
to[i++] = from[i];
to[i++] = from[i];
to[i++] = from[i];
} while (i < size);
This is called loop unwinding, and the reason it's faster than a regular loop is that it involves fewer evaluations of its condition and fewer jumps. Ideally, if we knew how many iterations the loop would take, writing them explicitly would be the fastest — but since the whole point of having loops is to support a variable number of iterations through a parametrizable condition, this wouldn't work. Finding some middle ground, though, like copying data in direct chunks of 4, 8, or some other number of bytes per iteration, can give us a boost in performance — and with functionality as basic as copying an array, it might well be worth it.
The problem, then, is dealing with the remainder: if size doesn't happen to be divisible by 4, how can we account for the last size % 4 bytes, making sure to copy them as well? We can add another loop that does so explicitly — but in 1983, Tom Duff, working on the software used for animating the special effects in Star Wars, came up with this devious contraption instead:
switch (size % 4) {
case 0: do { to[i++] = from[i];
case 3: to[i++] = from[i];
case 2: to[i++] = from[i];
case 1: to[i++] = from[i];
} while (i < size);
}
Take a moment to digest it. It interlaces a switch-statement with a do-while loop, in a way that seems like it shouldn't be valid — and yet in C, it is. If we think of loops and switch-cases in terms of gotos jumping to labels, it's not that inconceivable: if size % 4 is, say, 2, then we effectively start the loop from the middle, and after the first iteration (which takes care of the remainder) go back to its beginning, doing the rest of the iterations (of which there is a divisible number) in full.
Why am I telling you this? I mean, Duff's device is hardly a popular topic to include in a book on Python, so it's a valid question. But besides its ingenuity, which I find delightful, and its extensive use of assignment expressions, which I find relevant, my main motivation is to drive a certain attitude across. Software is not a mathematical abstraction developed by rational, uninterested minds — like everything else in human society, it's ultimately a human affair, carried out by emotional and biased humans, and it's important to always keep this in mind.
Pretending that this is a matter of pure reason, with no psychology and no politics involved, is both naïve and dangerous. Naïve, because it's simply not true, and doesn't account for the possible outcomes of any given discussion — and dangerous, because some of these outcomes can be profoundly negative. If we really were rational, uninterested minds, there wouldn't be a problem with voicing any opinion under any circumstances: we would either be right, in which case everyone would immediately accept it, or we'd be wrong, in which case they'd point out why, and we would give up on our perspective instantly and wholeheartedly.
But of course, this is not how things work; and when the walrus operator was proposed, it caused a huge, heated, and at times hostile debate within the Python community. Its story is arguably more interesting, and definitely more consequential, than the operator itself — and that's what this chapter is really about.
Guido's Farewell
So far, I tried to introduce the walrus operator without passing judgement on whether it's good or bad, or if you should use it or not. In fact, reading it again, my words feel like I've defended it, implying that it's a welcome addition to the language, when actually I think quite the opposite.
To live up to my own standards, I won't say you should never use it; I'm sure there are instances where it's a valid option. And from what I've seen, most of the code the uses it isn't unreadable — it's just less readable than the more explicit (albeit verbose) alternative. As a result, I barely use it myself, but nor do I comment about it when reviewing a student's submission or a colleague's code, unless I truly believe it adds unnecessary complexity. In other words, I don't mind it much, at least as a language construct; on the human level, though, I resent it very, very much.
Let's start with the software level. There's an idiom in software design about how code is read more than it is written — meaning that its readability is more important than how easy it was to write. After all, we only write it once, but we read it many times over the course of development and maintenance. What the walrus operator does is, honestly, neither new nor interesting — it's not an abstraction so much as a shortcut, which lets us collapse several statements into one expression. That expression, in turn, is less readable; you could argue that it's a matter of taste, but however you spin it, one terse snippet that does two things is objectively more complex than two separate snippets that do one thing each. What is a matter of taste is whether this complexity is justifiable, providing some compesnating merit — which in this case, at least in my opinion, it doesn't. Following our idiom, shaving off a few seconds of typing isn't worth adding a few seconds of confusion to anyone trying to decipher it later, since the latter matters considerably more.
This issue was discussed at length as part of PEP 572; and like I said, the discussion got so unpleasant that it lead Guido van Rossum, Python's creator, to resign from the Python project after more than 25 years. It probably wasn't the only reason, but it sounds like it was the last straw. This, in my opinion, is not worth any language construct, especially such a measly, insignificant shortcut — and why I'm taking the time to address it at length, and hopefully in a respectful enough manner to appeal to both sides of the argument. That people take themselves and their reasons so seriously as to sabotage the integrity of the community is not just misguided, but actively detrimental: a lot of Python's clarity and consistency can be attributed to this one person, reconciling and consolidating the various contributions into a coherent, well-informed and elegant style. Losing this person — and frankly, over such bullshit — is not only entirely ungrateful, but a huge waste for the language at large.
Was It Even Worth It?
Another curious twist to this story is that our idiom, about code being read more than it is written, is often attributed to Guido himself — it's even mentioned in some early PEPs and Python's official documentation. So how come he changed his mind and approved the walrus operator, despite some pretty vocal opposition from overly, um, enthusiastic members? Like I said, personally I'm not a fan of this decision — but nor do I consider the matter nearly important enough to drive a rift between people otherwise united around a common goal. Still, it's interesting to address the reasoning behind it.
The PEP lists a couple of "compelling" examples, which I find intriguing in terms of software design, and programming philosophy at large. According to the authors, Guido went over the codebase of Dropbox, where he was working at the time, and discovered that "programmers value writing fewer lines over shorter lines". For example, instead of writing:
match = pattern.match(data)
if match:
result = match.groups()
Or even:
match = pattern.match(data)
result = match.groups() if match else None
Some programmer wrote:
result = pattern.match(data).groups() if pattern.match(data) else None
Which matches the regular expression twice, and can be considerably less efficient. Another example was matching one of two patterns:
match1 = pattern1.match(data)
match2 = pattern2.match(data)
if match1:
result = match1.groups()
elif match2:
result = match2.groups()
else:
result = None
This is also less efficient than it could be, since it matches the second pattern even if the first pattern was found. The authors even provide the correct version, presumably to demonstrate that it's understandably cumbersome:
match1 = pattern1.match(data)
if match1:
result = match1.groups()
else:
match2 = pattern2.match(data)
if match2:
result = match2.groups()
else:
result = None
But wait; this isn't the only alternative, is it? What do you think about:
result = match_one_of(pattern1, pattern2, data)
And before you say that I'm cheating, consider that moving this logic into a small, dedicated function actually ends up being quite elegant:
def match_one_of(pattern1, pattern2, data):
match1 = pattern1.match(data)
if match1:
return match1.groups()
match2 = pattern2.match(data)
if match2:
return match2.groups()
return None
You could even generalize it:
def match_one_of(data, *patterns):
for pattern in patterns:
match = pattern.match(data)
if match:
return match.groups()
return None
Whoa. So you see, at least according to the examples in the PEP, most code that is complex enough to "benefit" from the walrus operator is also complex enough to benefit from a small, dedicated function. In fact, it seems like something in the very logic of the statement is more complex than usual — which is why we'd even consider an operator that does both assignment and evaluation. But complexity doesn't go away from being expressed in terser code; to clean it up, we need to add an actual layer of abstraction, like a function with an informative name and a clear signature.
Anyway, going back to the previous example, there's something even more interesting at play here. When a programmer writes something like:
result = pattern.match(data).groups() if pattern.match(data) else None
It's odd to claim that they "value fewer lines over shorter ones". This would imply that programmers are inexplicably born with some innate desire, some individual aesthetic preference for writing code; and since we're all liberal and enlightened, we should accommodate everyone equally, right?
But it's not really a matter of an innate tendency — it's a matter of habit, of education, of attention and proficiency. This code is objectively bad, on many levels: not only is it less efficient and less readable, but it actually duplicates code, so it's more error-prone and harder to maintain. From my experience, when a programmer writes something like this, it's not because they value it — it's because they don't really care, or simply don't understand, what exactly it does. Not that they're stupid, or lazy — they're simply not very well-informed, whether about regular expressions, software design, or both. So when the authors of the PEP advocate using the walrus operator "sparingly", only when it makes code "clearer" — in a way, it contradicts their examples, because it's safe to assume that the programmer in question wouldn't use it any better than they did regular expressions.
What really caught my eye, though, is that particular phrasing, about what programmers value. It indicates a very subjectivistic and individualistic worldview, which is endemic in our society, especially in the politically-correct culture of Sillicon Valley and Big Tech. Just because some people prefer something, or do something, doesn't mean it's OK — this is not a judgement or an accusation, it's simply a matter of fact. Some preferences can be bad, and anyway, most preferences are arbitrary habits; they're not an expression of our authentic self, because Homo Sapiens didn't evolve to care about assignment operators. If anything, they reflect our culture and education, which is always somewhat biased and imperfect, so the adult thing to do would be to admit it and account for it.
This might sound like irrelevant philosophizing, but I think it's a good lesson in software design. When we make decisions that involve other people, we have to consider their inputs, but also their contexts — and we have to do so with as little prejudice as possible, but also with a sober and unapologetic attitude. Whether we decide to include the walrus operator or not is besides the point; but deciding to include it because it would improve bad code, misses the actual reason of that code being bad — namely, that the programmer didn't write it well. If we can address something like this honestly and directly, without blame, and maybe even with compassion (I also wrote, and write, a lot of bad code; more so than good one, arguably), we can reach an amazing level of discourse as a community.
And whatever my opinion, I respect Guido and trust his intuition enough that if he chose to include it, there are probably some good use cases for it, even if it's not the ones listed. Or at least, it wouldn't be that bad — definitely not bad enough to justify the escalation that led to his resignation. Such a mindset, of having an argument instead of a discussion, stems in a way from the same worldview: that programmers (and people in general) are independent individuals, each entitled to their own opinion. This worldview is not so much wrong, as it misses the forest for the trees: having a diverse, mature and cooperative community is far more important to Python's future than this or that operator.
The moral of the story, at least in terms of software design, is this: rather than picking a side, it's worth asking why are there even sides to begin with. Sides, you see, are always a result of human dynamics, not of truth; so it should be addressed and amelioriated with humans in mind. It's never about "more correct" code, and definitely not the subjectivist catch-all that "it's all a matter of taste". Acidity and bitterness are a matter of taste; anything more complex, including the food we like, is a matter of culture, and we all have a say in recreating the cultures we take part in — so we can, in principle, aim to make them better as a whole, rather than making them more about us.