Error Handling: Exceptions, Tracebacks and Frames

Having covered Python's entire data model, there are still several miscellaneous topics I'd like to cover: and the first one of those is error handling.

Of course, we've seen it all along — it's impossible to cover something as complex as interrelated as Python in a completely linear fashion. But I hope you won't mind my going over its syntax again, in a bit more detail; especially since there are quite a few nuances and software design consideration we haven'treally addressed.

Raising Errors

Errors are, unsurprisingly, objects — but ones whose class inherits from BaseException or one of its subclasses, most commonly Exception. Such objects can be raised with a raise-statement:

>>> class MyException(Exception):
...     pass

>>> raise MyException('something went wrong')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MyException: something went wrong

Which cases interrupts the standard flow, propagating until someone catches and handles it. In an interactive session, that would be the Python interpreter, which will print the error's class name, argument, and traceback — something I've been omitting from the code snippets until now:

>>> def f():
...     g()
...     print('never happens')

>>> def g():
...     raise MyException('something went wrong')
...     print('never happens')

>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in g
Exception: something went wrong

Catching Errors

In a regular program, such an unhandled errors will bring it crashing down — so the responsible thing to do would be to handle them with a try-except clause:

>>> try:
...     f()
... except MyException:
...     print('everything is ok')
everything is ok

An except-clause only happens if an error was raised that is an instance of the exception it expects. It also work with a tuple of such exceptions:

>>> try:
...     raise ValueError()
... except TypeError, ValueError:
...     print('caught type or value error')
caught type or value error

But if we'd like the handling logic to be different, we can chain multiple except-clauses:

>>> try:
...     raise ValueError()
... except TypeError:
...     print('caught type error')
... except ValueError:
...     print('caught value error')
caught value error

It can even have an as-clause, if we'd like to inspect what exactly happens:

>>> try:
...     raise ValueError('something bad happened')
... except ValueError as error:
...     print('%s, but now everything is ok' % error)
something bad happened, but now everything is ok

Re-Raising Errors

In some cases, it makes sense to catch some error and then re-raise it again; that is, let it keep propagating, since all we wanted to do is log it or perform some cleanup:

>>> f = open('file.txt', 'w')
>>> try:
...     f.write(data)
... except OSError:
...     print('write failed') # log it
...     f.close()             # clean up
...     raise                 # re-raise
write failed
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
OSError: [Errno 27] File too large

Or, we might raise a different error in its stead:

>>> f = open('file.txt', 'w')
>>> try:
...     f.write(data)
... except OSError:
...     f.close()
...     raise RuntimeError('backup failed')
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
OSError: [Errno 27] File too large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: backup failed

Notice that in this case, the errors are chained: the RuntimeError is aware that it happened in the context of an OSError; in fact, it keeps it in its __context__ attribute, which is how the interpreter is able to reconstruct such a detailed traceback.

In some cases, it's wonderful: it lets us drill down into what exactly went wrong on the lowest level — any exception raised from it would just mask the original cause. Other times, though, we know that something can go wrong, and wouldn't necessarily want to expose our internal implementation — in which case, we can do:

>>> f = open('file.txt', 'w')
>>> try:
...     f.write(data)
... except OSError:
...     f.close()
...     raise RuntimeError('backup failed') from None
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: backup failed

Effectively settings the exception context ourselves; and if we set it to None, suppressing it altogether.

Raising the Right Error

To illustrate where we'd want to do this, consider the following configuration class:

>>> class Config:
...     def __init__(self, **values):
...         self.values = values
...     def __getattr__(self, name):
...         return self.values[name]

Admittedly, it's pretty silly to just wrap a dictionary like so — but suppose it provides some useful functionality, in addition to which we'd like it to store some arbitrary values:

>>> config = Config(x=1, y=2)
>>> config.x
1
>>> config.z
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __getattr__
KeyError: 'z'

There's something slightly wrong with this scenario — and it's not the fact that z is missing, which is legitimately the case. I'll give you a hint: the builtin getattr function is used to access an attribute given its name, and accepts an optional argument to default to if it's missing:

>>> getattr(config, 'x', 0)
1
>>> getattr(config, 'z', 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __getattr__
KeyError: 'z'

Now, that's definitely wrong: if z is missing, we specifically requested 0, and it looks like getattr blatantly ignored us. But actually, it didn't: it just expected a missing attribute to raise an AttributeError, not a KeyError, which is what a dictionary raises if it can't find some key. A more correct implementation would be:

>>> class Config:
...     # same as before
...     def __getattr__(self, name):
...         try:
...             return self.values[name]
...         except KeyError:
...             raise AttributeError(name) from None

>>> config = Config(x=1, y=2)
>>> getattr(config, 'x', 0)
1
>>> getattr(config, 'z', 0)
0

In which case, we wouldn't want to chain the AttributeError we raise to the KeyError raised by the dictionary — revealing the "root cause" in this case can only confuse people, since it's a valid and predictable situation in which we're simply "correcting" the error type.

Asking Forgiveness, Not Permission

In fact, we could have implemented __getattr__ in a seemingly cleaner way:

>>> class Config:
...     # same as before
...     def __getattr__(self, name):
...         if name not in self.values:
...             raise AttributeError(name)
...         return self.values[name]

That is, raise the right error if the key is missing, and return it if not; no need for all the extra nesting and weird from Nones, right?

Not so — it might look nicer, but it's actually less safe; specifically, it we use this implementation in multithreaded code, it might very well break and raise a KeyError. The same is true for this code, which attempts to open a file:

>>> path = 'file.txt'
>>> if os.path.exists(path):
...     f = open(path)
... else:
...     print('file does not exist')

The problem is that in a concurrent application — as well as in a filesystem, which is part of a concurrent operating system — things can be happening in parallel; and while it's not very probable, it can (and does) happen that our check succeeds, and a moment later (before the rest of our code is executed), something else happens: another thread removes the key from the configuration, or another process deletes the file we're trying to open, causing our code to crash despite its best efforts to mitigate precisely this case. Compare it to:

>>> try:
...     f = open(path)
... except FileNotFoundError:
...     print('file does not exist')

While somewhat clunkier, it absolutely guarantees that if we can open the file, we will — and if not, we won't (even if someone deleted it just a milisecond before our attempt).

There's a nice joke by the comedian Emo Philips that helps with remembering this idiom: "when I was a kid," he tells, "I used to pray every night for a new bike. Then I realised, the Lord doesn't work that way — so I just stole a bike and prayed every night that he'd forgive me."

So is the case with error handling in Python: exceptions are such a fundamental feature of the language that it's common practice to ask for forgiveness when some operation raises them, rather than permission to do said operation. This prevents a whole class of issues and security vulnerabilities known as time-of-check to time-of-use, or TOC/TOU, two of which we just saw: basically, something happening right after we checked that we can use a resource, before we've gotten around to actually using it.

Exception-Oriented Programming

If you're coming from other languages, like Java or C++, this idea might feel a bit uncomfortable: exceptions are notoriously expensive deployments, since they allocate objects, unwind the stack and forgo any instructions that CPU might have cached. In Python, however, this is not the case — or rather, it's negligible, since everything is an object, the stack is managed differently, and the opcodes the interpreter executes don't match to machine instructions anyway. In other words, Python operates at such a level of abstraction that there's no real performance penalty for using exceptions — so in many cases, it's entirely common practice to rely on them. Making sure some variable is defined, for example, is done like so:

>>> try:
...     x
... except NameError:
...     x = 1 # assign default value

An equivalent if-statement will have to check the locals(), the globals(), and ideally any closure our code might have — an operation that would be considerably more expensive. Similarly, checking if some functionality is available:

>>> try:
...     from os import getuid # missing on windows
... except ImportError:
...     def getuid():
...         return 0

Is much more robust that trying to determine what operating system we're using and whether it has a concept of a user ID (quick: does Solaris? Does Haiku?). So again, rather than being clever about it, it's simpler to try and just say "oops, sorry" if we fail.

Catching the Right Error

However, to handle errors properly, we need to catch the right error. Suppose we pass some variadic arguments to a function with perfect forwarding:

>>> div(*args, **kwargs)

If the arguments don't match the function signature, it's going to raise an error — but which? If we're not sure, and just wrap everything with try-except Exception:

>>> try:
...     div(*args, **kwargs)
... except Exception:
...     print('invalid arguments')

Then any exception the function raises — even if it's ZeroDivisionError that can happen naturally during division, or a NameError that can happen if we make a typo in the function's name and it doesn't exist at all — all of these errors will be suppressed, prompting the user to reconsider their arguments (which might be entirely appropriate).

In fact, Python supports a catch-all except-clause, where we don't specify the expected exception at all:

>>> try:
...     div(*args, **kwargs)
... except:
...     print('invalid arguments')

Which is even worse, since it catches things that aren't even errors, like KeyboardInterrupt or SystemExit. If this function gets stuck, and we press control+C to terminate the program, or if we're debugging it and call exit() to abort the code when it reaches a certain point — such a catch-all clause will actively resist us, ignoring our attempts to shut the process down.

For that reason, using except with no exception class is generally discouraged; the only legitimate use case I can conceive of is if we want to do something, like logging what happened, and then re-raise the error — in which case, recording that our server stopped because somebody deliberately closed it can be a reasonable thing to do. So in a way, it's good that we have it — but we should not use it lightly, since in the vast majority of cases what we actually mean is probably except Exception.

Or, better yet, except a specific exception: going back to our question from before, when we call a function with arguments that don't match its signature, it raises a TypeError — so whenever we have a suspicion what might go wrong, the responsible thing to do is to calibrate our code to address that, and only that. Anything else that might go wrong, any "unknown unknowns" we might've missed, is better off propagating than being mistaken for something that it isn't and brushed off on that account.

To Err Is Human

That last bit is actually quite important. A lot of programmers seem to hold unto the notion that errors are bad — that all code should behave correctly, always. In a perfect world, that might be achievable; but in the one we currently live in, I think a more mature and pragmatic attitude is to embrace the fact that mistakes happen, and try to learn from them rather than sweep them under the rug.

Python is, after all, an interpreted language: it executes our code as it comes, which has enormous benefits — but also means we can get ourselves into an unfamiliar corner, where something misbihaves in a way we hadn't ever considered. This can happen even in compiled languages like C++: for all the compiler's type checking and validation, programs still crash with segmentation faults or unhandled exceptions.

There's a funny story — I don't know if it's actually true — about the development of the original Macintosh system in the 1980s. Close to its release date, the programmers working on it noticed a problem: every time the system would shut down, it would raise an exception, and despite extensive debugging, nobody could figure out why. Frustrated and running out of time, they figured out a creative solution: they changed the error message to "Thanks for using Macintosh, have a nice day!" — so when the system would switch off, it would still effectively crash, but do so in such a manner that it looked like its intended behavior.

The moral of the story is not that we should give up on fixing problems, of course; but that we should be accept the fact that problems are going to arise and act accordingly. Our considerations might differ, of course: if we're working on a pacemaker or a nuclear reaction, our tolerance for mistakes might be lower — but seeing as most of us are not, a shift in our attitude towards them might be the most practical solution. It's like the serenity prayer, popularized by alcoholics anonymous, says: "grant me the serenity to accept the things I cannot change, the courage to change the things I can, and the wisdom to tell the difference."

Another anecdote to that effect comes from my own experience: we were working on a system whose database would occasionally fill up all its storage and crash due to insufficient space. The problem was that to clear some space properly, we couldn't just delete some files, which might corrupt the database's state; we had to launch it and run an orderly cleanup, except that in such a state it wasn't able to launch at all, so every time was an emergency, and we felt that one day we might run out of tricks. The obvious solution of allocating more storage wasn't an option either, since some organizational politics have completely alienated our IT department — so eventually we decided to allocate an 1GB file, full of random noise, with the apt name of DELETE_ME_IN_CASE_OF_EMERGENCY. Hardly the kind of solution that would impress Dijkstra — but given our circumstances, it worked surprisingly well.

Anyway, all these stories aren't just some idle philosophy: it illustrates how some consideration are truly beyond our control. Even if your code is pitch perfect, the system itself might run out of storage space or memory, encounter a disk fault, or crash due to bugs that exist in the Python interpreter itself, or the operating system — those things exist, as I can attest to have seen myself on several occasions. In these situation, accidentally catching a MemoryError we never thought of can be worse than letting it crash your system, because at least then we know something went terribly wrong, and can seriously consider what can be done to ameliorate it, even if it's an equally terrible hack. What's more, mishandling such errors is a bad strategy: if the system truly is out of memory, the fact that we can ignore it and carry on doesn't mean the rest of our code will work as expected — in such a state, the system will keep thrashing and malfunctioning, and it's better to get fewer, clearer signals as to why, rather than a 24-hour log full of all sorts of indirect warnings.

The Exception Hierarchy

Having said that, of course we shouldn't give up on error handling with a "come what may" attitude; all I mean is that we should handle what we can with a precision that will reveal to us what we hadn't considered, so we can learn and adjust with forethought. To do that, however, we need to familiarize ourselves with Python's exception hierarchy: it has quite a few different errors, and all of them mean something different.

At the root, we have the BaseException: it's never actually raised itself, but serves as a superclass for anything raisable. From it, we derive two types of exceptions: Exception and a few non-exception classes we'll cover momentarily.

Now, raise Exception() is something you might see — but it's the vaguest error possible, and Python itself never raises anything like it. So in effect, Exception also serves as a superclass: and conveniently, all its subclasses have names that end with Error, whether its SyntaxError, TypeError or ValueError. As their name might suggest, they deal with errors: something undesirable happened, and we need to accommodate it if we wish to continue.

The non-exception classes, on the other hand, pertain to situation where nothing bad happened, and we just want to get out of the current execution flow. The two significant examples are SystemExit, raised when we call exit(), which can be used to abort a program or close an interactive session, and KeyboardInterrupt, which happens when a user pressed control+C, or the program receives an interrupt signal in some other way. If you recall our GotoEnd context manager, where we added support for a local return, consider what might happen if we have:

>>> with GotoEnd() as goto:
...     try:
...         data = open(path).read()
...         if not data:
...             goto.end()
...     except Exception:
...         data = default_data
...     # more code

That is, if the file doesn't exist or we can't read it, we want to use some default data, but if the file is empty we want to abort the entire block. The way we implemented GotoEnd, the ContextExit that end raises is an instance of Exception, so our code will continue with the default data even if the file is empty; this is one of those rare cases when we'd actually want to inherit from BaseException, since what we raise is not actually an error.

Alternatively, we could expect the user to catch a more specific OSError, just like we discussed — in which case ContextExit would propagate as intended; but since we can't really guarantee how others will use our code, we might as well make it as precise as we possible can.

The Else Clause

This is actually a good opportunity to introduce another, less well-known feature of the try-except structure: the else-clause, which happens if the try-block ended successfully and no exception was raised.

Like the loop's else-clause, it's not immediately clear why it's useful; if we want to print the number of entries in a JSON file:

>>> text = open('file.json').read()
>>> try:
...     data = json.loads(text) # might fail
...     print('file has', len(data), 'entries')
... except Exception:
...     print('not a json file')

We can simply put it after whatever we expect to fail, like json.loads. This, however, doesn't account for the fact that whatever line we add after can also fail — for example, true, 1 and null are all valid JSON contents, but they don't have a length — so we might catch the wrong error, raised by len (or even print), and handle it incorrectly.

Once again, proper exception classes can help us: json.loads raises a JSONDecodeError, while len raises a TypeError; but sometimes, the two can coincide. So instead, the else-clause provides us with the opportunity to separate what we're actually hedging against from everything else:

>>> try:
...     data = json.loads(text)
... except OSError:
...     print('not a json file')
... else:
...     print('file has', len(data), 'entries')

And if we fail in the else-clause, it's a whole other error, unrelated to and unprotected by the try, whose sole focus is on loading the JSON. You might wonder, still, how is it different from simply putting it after the try-block altogether:

>>> try:
...     data = json.loads(text)
... except OSError:
...     print('failed to open file')
... print('file has', len(data), 'entries')

And in most cases, it is not. But consider our previous example again:

>>> with GotoEnd() as goto:
...     try:
...         data = open(path).read()
...     except Exception:
...         data = default_data
...     if not data:
...         goto.end()
...     # more code

We want to abort if the file is empty, and otherwise use some default data. This way, however, we also abort if the default data is empty, which isn't necessarily what we wanted — the way to address it properly would be:

>>> with GotoEnd() as goto:
...     try:
...         data = open(path).read()
...     except Exception:
...         data = default_data
...     else:
...         if not data:
...             goto.end()
...     # more code

That is, try something; if it fails, account for that; and if it succeeds, do something else that might raise its own, unrelated exceptions. In other words, code appearing after the whole try-block might run after the except-clause happened, but the else-clause runs instead of it; if they deal with similar concepts, this separation might come in handy.

It's a bit of a contrived example, I know — but error handling is all about nuance, since usually we're dealign with exceptional behavior that doesn't match our initial expectations anyway. Here's a more mundane example:

>>> try:
...     x = numbers[0] # might be empty
...     x += 1
... except Exception:
...     x = 0

This takes the first number, increments it by 1, or defaults to 0 if numbers is empty. But what if numbers is not empty, but its first element is not a number either, so it's the += 1 that raised the exception? This would've caught it:

>>> try:
...     x = numbers[0]
... except Exception:
...     x = 0
... x += 1

But then, if numbers is empty, x would end up as 1 instead of 0. We can do x = -1 to offset this quirk, I guess; but this feels substantially more elegant:

>>> try:
...     x = numbers[0]
... except Exception:
...     x = 0
... else:
...     x += 1

Error Types

You know what else can work well, and arguably be even more elegant? This:

>>> try:
...     x = numbers[0]
...     x += 1
... except IndexError:
...     x = 0

Because accessing a missing element will raise an IndexError, while adding 1 to a value that can't accommodate it will raise a TypeError — which brings us back to Python's exception hierarchy. Knowing what to catch can improve our ability to navigate tricky edge cases, especially so we don't catch something we didn't intend to; and knowing what to raise can make our code much more predictable to others, assuming they're paying similar attention to their error handling code.

But if you look at the number of standard errors, there are almost 50 of them, which can feel intimidating:

>>> len([k for k in __builtins__.__dict__ if k.endswith('Error')])
49

Most of them we're not likely to use, and half of them we're not likely to see much — but it's still a lot, so let me try to break them down and go over them in a way that will make sense.

First, the most common exception that we're going to use most of the time: ValueError. Basically, it means there's a problem with some value — we got an invalid argument that doesn't meet some criteria. Several of its subclasses have to do specifically with unicode operations, so we won't be raising them ourselves, but we might very well see them when converting strings to and from bytes: UnicodeEncodeError, UnicodeDecodeError and UnicodeTranslateError, all subclasses of UnicodeError.

Second, there are errors that arise from object behavior, which we should imitate when implementing magic methods ourselves. The most common one here is AttributeError, for when something goes wrong with dot notation — most often accessing a method or attribute that don't exist. If you're coming from other languages, this is also how a null dereference usually looks like in Python: None being passed into something that expects it to have this or that field, which it doesn't. Anyway, this is what we should raise if we can't complete __getattr__, __delattr__, __get__ or __delete__ (as well as __setattr__ and __set__, if the assignment is somehow invalid).

Similarly, there are the two LookupErrors, KeyError and IndexError, for bracket notation: when we're dealing with containers, either mappings like dictionaries or sequences like lists, this is what comes up when a key is missing or out of bounds, so this is what we should raise in __getitem__ and __delitem__.

And then there's RuntimeError, for when something unexpected happens during our operation and doesn't fit any other criteria. One of its subclasses, NotImplementedError, we raised in abstract methods to force their implementation; the other, RecursionError, we might see if our function calls itself recursively too many times, usually due to an infinite loop, but we'd probably never raise ourselves.

Other than these 5 errors — ValueError, AttributeError, KeyError, IndexError and RuntimeError (well, and NotImplementedError, but that doesn't really count) — there isn't much else for us to complain about; so the third group is errors we'll see a lot, but probably never raise ourselves:

A NameError happens when we interact with an invalid name, usually one that is not defined (or in case of its subclass, UnboundLocalError, which we've seen in the past — not defined yet). A TypeError happens when we interact with an object in an invalid way, like passing arguments that don't match the function signature, or performing an operation that the object doesn't support. Rather than raising it directly, though, we should return NotImplemented and let Python see if it can't figure out a better solution. Specifically, please don't raise it when an argument doesn't match its expected type: this is a classic ValueError, and has nothing to do invalid behavior.

Then there's ImportError, which happens when a module can't be imported (usually via its more specific subclass, ModuleNotFoundError); AssertionError, which happens when an assert-statement evaluates to false; and SyntaxError (including IndentationError and TabError) which happens when our code is not valid Python. But again, nothing that we'd normally raise ourselves — not unless we're implementing an importer, a testing framework, or some sophisticated code generation, in which case we'll probably be very familiar with them anyway.

The fourth group is OSError and all of its subclasses, of which there are 15: basically, something went wrong in our interaction with the "outside world", the operating system. It can be FileNotFound or FileAlreadyExists; and if we're dealing with a file- or directory-specific operation, IsADirectoryError or NotADirectoryError; or if we're dealing with a privileged operation, PermissionError; or if we're dealing with a timed operation, TimeoutError; or if we're dealing with communication, ConnectionError. The nice thing about those errors is that they're not really Python-specific: they all include a number, errno, and a corresponding error message, which we can find more information about online.

Finally, there are the errors we rarely, if ever, see: MemoryError for when we run out of memory, or SystemError for when Python encounters an internal problem — two cases in which there isn't much we can do, anyway; and ZeroDivisionError (a subclass of ArithmeticError), which has a very particular use case (and personally, I think it should have been a ValueError).

That should cover most of them — and hopefully, it wasn't too bad. Here's a condensed table for reference, see if you can recall what each of them does (I've grayed out the ones mention, and highlighted the important ones):

No Error No Cry

We can, of course, define our own error types — but this has several drawbacks, so we should do it only when we're absolutely convinced it's necessary. Suppose we have a function that receives a user ID and deletes the corresponding user (from its cache as well as from the system):

>>> def remove_user(uid):
...     uid = int(uid)
...     if uid not in users:
...         raise UserDoesNotExist()
...     # delete user from system
...     del users[uid]

>>> class UserDoesNotExist(Exception):
...     pass

In my opinion, it would be better to raise a standard ValueError in this case (or at least, make UserDoesNotExist subclass it instead of Exception). The first drawback of custom errors is that they add cognitive load: there are now more parts to our system that need to be considered, without much compensating merit to this complexity. The second, somewhat related drawback is that any custom errors we define are in addition to the standard ones, not instead of them — we can't hope to remove all the ValueErrors our function might raise (unless we catch and replace them manually), because Python itself still resorts to them (e.g. if uid can't be converted to an integer, that's what int will raise anyway).

Last, but not least, is that I subscribe to a different design philosophy altogether: trying to define errors out of existence. John Ousterhaut, who developed the TCL language, laments in his book, A Philosophy of Software Design, about his own implementation of the unset keyword in TCL: pretty much like in Python, trying to delete a nonexistent name raises an error:

>>> del x
NameError: name 'x' is not defined

And you kind of have to ask yourself, why? On one hand, this is legitimate behavior — trying to delete something that doesn't exist isn't entirely correct; but at the same time, is it that exceptional? If the contract of this operation is that after its carried out, the name x will not exist anymore, then we might as well ignore deletion of names that didn't exist to begin with: after everyone is said and done for, the name indeed doesn't exist, so the contract is fulfilled without burdening the user with 3 extra lines.

The same holds for function: if uid is not in users, we can simply return, and no harm is done. To be clear, this idea is complementary to exceptions — I'm not arguing against them; if a non-numeric value is passed in, it's clearly a mistake (and int will be sure to complain about it), so the function still raises exceptions — it just doesn't do so for cases in which there's no added value to such aggressive behavior.

What I'm trying to say is this: when we encounter an exceptional case, raising an error is absolutely reasonable. We don't know how to proceed, so we abort our flow entirely; it's both responsible and informative. But in some cases, we get to define what constitutes exceptional behavior — and there are other ways of handling edge cases. Consider another function, which returns a user from the same cache:

>>> def get_user(uid):
...     uid = int(uid)
...     if uid not in users:
...         raise ValueError()
...     return users[uid]

The reason I don't just return users[uid] is that I wouldn't want to raise an unexpected KeyError if the user is missing — providing an invalid user ID is a ValueError. However, as the function's designer, we can decide to soften its interface: when a corresponding user doesn't exist, we can return None instead. Any code that users our function can then do:

user = get_user(uid)
if not user:
    return

Instead of:

try:
    user = get_user(uid)
except ValueError:
    return

Especially since we've actually conflated two errors into ValueError: was this a nonexist user ID that caused the error, or an invalid one altogether? Using UserDoesNotExist would actually be clearer on this point — but so is returning None, really; any ValueError raised by int still propagates.

As another example, consider taking a slice of an array or a substring of a string in Java: we call Array.copyOfString(a, from, to) or s.substring(from, to), but both can raise IndexOutOfBoundsException if the expected range doesn't fit within the boundaries of our target object; we need to crop them correctly beforehand, where it would perfectly reasonable behavior for the language to do so on our behalf. And indeed, this is what Python does: with slice notation, both a[from:to] or s[from:to] return their the best effort to crop the target thusly, even if it resulting range turns out to be empty — and from my experience, it's much nicer to work with as a result.

Python, however, is not entirely consistent in this regard. Both lists and strings have an index(item) method, returning the offset of the item within them — or raising a ValueError if it's missing. But a similar indexOf behavior in Javascript figured out a softer solution — it returns -1 to indicate the same thing, no error needed. Python strings, in fact, have an equivalent find(item) method which does exactly that — but lists don't have it for some reason, which is just odd. Of course, a special return value and an exception are somewhat interchangeable — but think what it entails from the users' perspective. In one case, they can do:

index = s.find(substring)
if index == -1:
    raise ValueError()

But if they want the opposite:

try:
    index = a.index(item)
except ValueError:
    index = -1

It's not just an extra line — there's additional mental load just with having to deal with this flow control: capturing a return value and checking it with an if-statement is rather trivial in comparison to a try-block that resets the value on error.

One last example is how we delete an item from a dictionary. Calling del d[key] raises a KeyError if the key is missing, just like getting d[key] would. There's something pleasant about this symmetry, so I'd give it a pass if it weren't for the following: calling d.get(key) is a softer version of the latter, returning a default None; and similarly, we have d.pop(key) — except, it still raises a KeyError is the key is missing! To achieve a so called "soft delete" in Python, we have to do d.pop(key, None), and there's something rather inelegant about the whole thing: not just in the "astonishment" over the difference, but also about the more desirable behavior being less achievable. Why couldn't d.pop(key) (or getattr(obj, name), for that matter) return None by default, and have an optional parameter in the (rarer) case where we do want an error, like d.pop(key, must_exist=True)? As usual, software design is not an exact science, so there are no right answers — but since we're talking about the right ways to raise errors here, I thought I'd mention that a rather valid option is to simply not raise them at all.

Warnings

If you're still ambivalent, I don't blame you: silently ignoring a user ID that doesn't exist and "pretending" to have deleted it can feel somewhat dishonest. Interestingly, though, Python provides a mechanism that can be used to address this and other cases.

Warnings are similar to errors, except, well, less severe. We "raise" them by calling warnings.warn, and optionally passing a more specific class (just like with errors), this time subclassing Warning:

>>> from warnings import warn
>>> warn('this is a warning') # default is UserWarning
<stdin>:1: UserWarning: this is a warning
>>> warn('this feature will be removed in version 2', DeprecationWarning)
<stdin>:1: DeprecationWarning: this feature will be removed in version 2

The nice thing about warnings is that users can decide what to do with them for themselves, by setting what's known as a warning filter. The default behavior simply prints them to the standard error stream — but we can also suppress some of them (or all of them) by setting it to ignore, or make them raise actual errors by setting it to error.

This filter can be set by Python's -W flag or the PYTHONWARNINGS environment variable — they have the same format and do the same thing, essentially setting this information to sys.warnoptions. The format, in turn, is action:message:category:module:lineno, where we can skip any filtering options that are not applicable to our case, or omit them entirely if the whole tail is irrelevant. To give some examples:

/home/user$ python -W error::RuntimeWarning
# raises runtime warnings as errors
/home/user$ python -W "ignore:.*deprecated.*" example.py
# suppresses warning the contain the string "deprecated"
/home/user$ export PYTHONWARNINGS="always:::p.a,ignore::DeprecationWarning"
/home/user$ python -m p
# suppresses depreciation warnings, but always prints warnings from p/a.py

The same can be done in code using warnings.filterwarnings:

>>> warnings.filterwarnings('error', category=RuntimeWarning)

Or even as a context manager, in which case the specified filter applies to that particular context and resets after it's done:

>>> with warnings.catch_warnings(action='ignore'):
...     # suppresses all warnings

We can even collect all the warnings raised in such a context in a list:

>>> with warnings.catch_warnings(action='always', record=True) as ws:
...     warn('this is a warning')
>>> ws
[<warnings.WarningMessage object at 0x...>]
>>> ws[0].message # this is the actual warning object
UserWarning('this is a warning')

Or, if we want to redirect them to any stream other than stderr, we can overwrite the warnings.showwarning function to do so:

>>> def log_warning(message, category, filename, lineno, file, line):
...     with open('/tmp/log.txt', 'a') as f:
...         f.write('%s: %s\n' % (category.__name__, message))

>>> warnings.showwarning = log_warning

So really, warnings are a very interesting (and somewhat underused) mechanism that lets us have soft errors: one that we can decide dynamically whether we want to raise, log, or just ignore, and do so with pretty unprecedented resolution.

So, if in doubt about whether we should or shouldn't raise an error (for example, if the UID being deleted doesn't exist), we always have this option:

>>> def remove_user(uid):
...     uid = int(uid)
...     if uid not in users:
...         warn('user %d does not exist' % uid)
...         return
...     # delete user from system
...     del users[uid]

If the user prefers to deal with exceptions, they can set warnings to error (in general, or particularly in the context of running our code); and if they don't care, they can suppress it entirely. Even when they don't, a nice feature of warnings is that the same warning is never issued twice, so we're not going to spam the user either; it's too bad the warning hierarchy is not as elaborated as the errors one, and that there's no dedicated syntax to raise them — like I said, I think they're extremely handy, especially when we design our code to avoid unnecessary errors; but for some reason, they're often overlooked.

Exception Groups

So sometimes, we can avoid raising an error altogether: what about the opposite situations, when we have more than one error to raise?

Imagine we'd like to convert a list of items to integers:

>>> def to_ints(xs):
...     return [int(x) for x in xs]

Normally, the first item that we encounter for which it doesn't work will raise a ValueError, and that's that; if it's removed, another such value can be hiding down the road — so wouldn't it be cool to raise all of them at once?

Or, imagine we're looking for some filename in a predefined set of directories (like sys.path): it's not in the first directory, and in the second directory we don't have permissions to read it — so it's two different errors, which we'd normally have to suppress in order to arrive at the conclusion that no such (accessible) file exists:

>>> def find(filename):
...     for directory in directories:
...         with contextlib.suppress(FileNotFoundError, PermissionError):
...             path = os.path.join(directory, filename)
...             return open(path, 'r')
...     raise ValueError('unable to find file: %r' % filename)

This discards all the useful information about what actually went wrong, and settles for a generic and not very descriptive error instead. For these and other reasons, Python 3.11 introduced a new and exciting feature: ExceptionGroup, which lets us bind and raise a group of exceptions, all at once:

>>> def find(filename):
...     errors = []
...     for directory in directories:
...         try:
...             path = os.path.join(directory, filename)
...             return open(path, 'r')
...         except (FileNotFoundError, PermissionError) as error:
...             errors.append(error)
...     raise ExceptionGroup('unable to find file: %r' % filename, errors)

When such an object is raised, we get this delicious breakdown:

>>> f = find('a.py')
  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 1, in <module>
  |   File "<stdin>", line 9, in find
  | ExceptionGroup: unable to find file: 'file.txt' (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |    File "<stdin>", line 6, in find
    | FileNotFoundError: [Errno 2] No such file or directory: '/dir1/a.py'
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |    File "<stdin>", line 6, in find
    | PermissionError: [Errno 13] Permission denied: '/dir2/a.py'     
    +------------------------------------

Which is incredibly detailed, and can be invaluable for debugging. And that's not even the best part: together with exception groups, the new except* syntax was introduced, which lets us catch and handle parts of the exception group, while letting the rest of it propagate on. So if we'd decide to ignore directories where the file doesn't exist:

>>> try:
...     f = find('a.py')
... except* FileNotFoundError:
...     pass
  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 2, in <module>
  |   File "<stdin>", line 9, in find
  | ExceptionGroup: unable to find file: 'a.py' (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "<stdin>", line 6, in find
    | PermissionError: [Errno 13] Permission denied: '/dir2/a.py'
    +------------------------------------

Or, we can catch both, see if we can fix the permissions issues, and retry:

>>> try:
...     f = find('a.py')
... except* FileNotFoundError:
...     pass
... except* PermissionError as group:
...     for error in group.exceptions:
...         with contextlib.suppress(PermissionError):
...             os.chmod(error.filename, 0o644) # makes file readable
...     f = find('a.py')
>>> f # it worked!
<_io.TextIOWrapper name='/dir2/a.py' mode='r' encoding='UTF-8'>

Exception Notes

This would also work with our to_ints example, although here it might be a bit less useful: we'll end up with a bunch of ValueErrors that we need to sieve through. It would be nice if we were able to include custom information with each error, especially since now we're throwing them en masse — so together with exception groups, Python added the option to enrich errors with notes. This way, we could mention problematic indices:

>>> def to_ints(xs):
...     errors = []
...     result = []
...     for i, x in enumerate(xs):
...         try:
...             result.append(int(x))
...         except ValueError as error:
...             error.add_note('item index: %d' % i)
...             errors.append(error)
...     if errors:
...         raise ExceptionGroup('conversion failed', errors)
...     return result

>>> xs = ['1', 'a', '2', 'b']
>>> ns = to_ints(xs)
  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 1, in <module>
  |   File "<stdin>", line 11, in to_ints
  | ExceptionGroup: conversion failed (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "<stdin>", line 6, in to_ints
    | ValueError: invalid literal for int() with base 10: 'a'
    | item index: 1
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "<stdin>", line 6, in to_ints
    | ValueError: invalid literal for int() with base 10: 'b'
    | item index: 3
    +------------------------------------

We can actually use this information to iterate over the ValueErrors in the group, extract their index from the note, remove it and retry; and to avoid the indices shifting inwards after deletion, let's do it in reverse:

>>> try:
...     ns = to_ints(xs)
... except* ValueError as group:
...     for error in reversed(group.exceptions):
...         i = int(error.__notes__[0].split(':')[1])
...         del xs[i:i+1]
...     ns = to_ints(xs)
>>> ns
[1, 2]

As aside, while notes are particularly useful with exception groups, we can use them to add information to regular errors, too; and as you can see from the __notes__ attribute, they accumulate. As an error propagates up the stack, we can imagine it being caught multiple times, with each level trying to resolve it — and if it can't, adding a note about its attempt before re-raising it further:

>>> def f():
...     raise Exception('f failed')

>>> def g():
...     try:
...         f()
...     except Exception as error:
...         error.add_note('g failed')
...         raise

>>> g()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in g
  File "<stdin>", line 2, in f
Exception: f failed
g failed

Tracebacks

There's something a bit redundant about that last example: we don't need each function in the call stack to add a note about its failure, since this is something we can see in the error's traceback anyway. In a more complex situation, such notes might contribute valuable information — but if this isn't the case, with errors as with comments, if we don't have anything to say, it's better to not say anything at all.

So what are tracebacks, then? Essentially, as the error starts propagating and unwinding the call stack, it collects the frames which it is unfolding in a chain of objects, the last (or uppermost) of which, once the error is caught, is exposed through its __traceback__ attribute. Collectively referred to as a traceback (because it allows us to trace the error's route back to its origin), each of these traceback objects contains a reference to its frame (via tb_frame), the line and instruction offset at which the error occurred (tb_lineno and tb_lasti), and the next, "deeper" traceback (tb_next), ever closer to the source. The associated frame objects, in turn, hold information about the code block through which the error has passed: in f_code we can find its code object (which, as you might recall, has co_name and co_filename), and f_locals holds the local scope as it was when the error occurred.

This wealth of information enables us to reconstruct a pretty detailed picture of where the error has travelled. A lot of the code I show happens in an interactive session, which doesn't really have a concept of a file or a line number (hence the "<stdin>" and the confused lines, which restart in every function from the beginning). But if we do have a file:

# example.py

def f():
    raise Exception('something went wrong')

def g():
    f()

Then upon catching an error, we can iterate over its traceback and print a pretty similar report:

# same as before

try:
    g()
except Exception as error:
    print('Traceback (most recent call last):')
    lines = []
    tb = error.__traceback__
    while tb:
        file = tb.tb_frame.f_code.co_filename
        line = tb.tb_lineno
        function = tb.tb_frame.f_code.co_name
        lines.append('  File "%s", line %d, in %s' % (file, line, function))
        tb = tb.tb_next
    print('\n'.join(reversed(lines))
    print('%s: %s' % (error.__class__.__name__, error))

Which gives us:

/home/user$ python example.py
Traceback (most recent call last):
  File "/home/user/example.py", line 8, in <module>
  File "/home/user/example.py", line 5, in g
  File "/home/user/example.py", line 2, in f
Exception: something went wrong

Of course, we don't actually have to do that; Python does it itself, and when it's dealing with files it even go ahead prints the corresponding line in the source code (which it gets by opening and reading the file, so it can't do in an interactive session):

/home/user$ python example.py # with try-except removed
Traceback (most recent call last):
  File "/home/user/example.py", line 7, in <module>
    g()
  File "/home/user/example.py", line 5, in g
    f()
  File "/home/user/example.py", line 2, in f
    raise Exception('something went wrong')
Exception: something went wrong

In recent versions, it even adds funky ASCII art to highlight the problematic operation itself; if we swap raise Exception() with e.g. opening a non-existent file, we'd get:

/home/user$ python example.py # with try-except removed
Traceback (most recent call last):
  File "/home/user/example.py", line 7, in <module>
    g()
  File "/home/user/example.py", line 5, in g
    f()
  File "/home/user/example.py", line 2, in f
    f = open('file.txt')
        ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'file.txt'

And if you're wondering how can Python possible know that — it's done by two new code object attributes we haven't discussed until now, co_exceptiontable (which maps where try-blocks begin and end), and co_positions (which maps the offset of each instruction within its line).

As an aside, if we ever want to print such a traceback ourselves without letting it propagate all the way to the top and crash our program, we can use the standard traceback module. Within an except-clause, we can simply do:

>>> import traceback
>>> try:
...     raise Exception('something went wrong')
... except Exception:
...     traceback.print_exc()
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception: something went wrong

Where Python retrieves the active error by calling sys.exc_info(); and from anywhere else (assuming we have an error object):

>>> traceback.print_exception(error)
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception: something went wrong

And if we want it as strings (e.g. to write to a log), traceback.format_exc and traceback.format_exception come to our rescue. Hell, it lets us print even if there's no error at all, with traceback.print_stack; just replace the problematic line in our file with this, and we get:

/home/user$ python example.py
  File "/home/user/example.py", line 9, in <module>
    g()
  File "/home/user/example.py", line 7, in g
    f()
  File "/home/user/example.py", line 4, in f
    traceback.print_stack()

Frame Objects

How can Python do that, with any traceback? The truth it, it doesn't need much more than the frames. We can always retrieve the current frame with inspect.currentframe, and its f_back to go "up" to the previous frame that effectively called into it.

Note how this chain points in the opposite direction from a traceback: with frames, we're looking from the innermost level outwards, whereas once an error has unfolded this nesting, tracebacks let us peek back inwards from outside.

Anyway, this lets us print the stacktrace with no error, using pretty similar code:

# example.py

import inspect

def f():
    lines = []
    frame = inspect.currentframe()
    while frame:
        file = frame.f_code.co_filename
        line = frame.f_lineno
        function = frame.f_code.co_name
        snippet = open(file).read().splitlines()[line - 1]
        lines.append('    %s' % snippet.lstrip())
        lines.append('  File "%s", line %d, in %s' % (file, line, function))
        frame = frame.f_back
    print('\n'.join(reversed(lines)))

def g():
    f()

g()

Whereupon we get:

/home/user$ python example.py
  File "/home/user/example.py", line 20, in <module>
    g()
  File "/home/user/example.py", line 18, in g
    f()
  File "/home/user/example.py", line 9, in f
    line = frame.f_lineno

And note how, since now we're using f_lineno, it hones in on the line it's called at.

Frame objects can be used for other creative purposes, too: one cool thing we can do is interpolate strings based on their enclosing scope, like so:

>>> def log(message):
...     frame = inspect.currentframe()
...     print(message.format(frame.f_back.f_locals))

>>> x = 1
>>> y = 2
>>> log('x = {x}, y = {y}')
x = 1, y = 2

We do this simply by going one level up, to whoever called the log function, and using its local scope to automatically format the message.

Or, if you've ever used pytest to write tests, you might have noticed it generates a pretty elaborate breakdown of what happened:

# test.py

def test():
    x = 1
    y = 2
    assert x == y

/home/user$ pytest test.py
___________________________________ test ___________________________________
    def test():
        x = 1
        y = 2
>       assert x == y
E       assert 1 == 2

test.py:4: AssertionError
========================= short test summary info ==========================
FAILED test.py::test - assert 1 == 2

Compare this to a regular assertion error, if we'd just call test at the bottom of our file and let Python handle it:

/home/user$ pytest test.py
Traceback (most recent call last):
  File "/home/user/test.py", line 6, in <module>
    test()
  File "/home/user/test.py", line 4, in test
    assert x == y
AssertionError

There's quite a difference: pytest actually parses our code, fetches the corresponding values from f_locals, and gives us a much better picture of what exactly transpired — which is invaluable to understanding how and why are tests are failing.

Error Messages

In fact, assertion statements are particularly opaque in this regard:

>>> x = 1
>>> y = 2
>>> assert x == y
AssertionError

Without a traceback, we don't even know what happened — Python first evaluates the expression, which turns out to be false, and this makes the assert-statement raise an error with no idea whatsoever as to what about it was false, or what it even was. In the olden days, if we wanted to get informative messages, we'd have to add them like this:

>>> assert x == y, '%r = %r' % (x, y)
AssertionError: 1 = 2

But of course, that's no way to live. With other errors, at least, we conventionally pass an error message — and since there's no special syntax for it, it's not as jarring:

>>> raise Exception('something went wrong')
Exception: something went wrong

In fact, Exception can receive as many arguments as we want, which it stores in args, and uses to print itself:

>>> e = Exception(1, 2, 3)
>>> e.args
(1, 2, 3)
>>> print(e)
(1, 2, 3)

Although, if it's a single value, it strips the tuple away — since the most common scenario is passing in a single string, this makes everything look nicer. Other exceptions, like OSError, can have additional fields (like errno, strerror, or filename), or define slightly different formatting. If we compare AttributeError to KeyError:

>>> print(AttributeError('x'))
x
>>> print(KeyError('x'))
'x'

We'll notice that in case of a key, quotes are added (that is, Python uses repr rather than str), just to reflect its "keyness" a little better. This is incredibly nuanced, in my opinion, and leads me to my last point in this chapter: error messages.

If you take one thing away from this long deliberation, let it be this. Exception hierarchies, groups and warnings are all nice — but since most of the time we'll just be using a ValueError anyway, passing some string into it, this message is effectively the most important bit of information to get right. There's nothing more frustrating than receiving and opaque error message that just tells us something is invalid argument or that some operation failed.

So, please, put serious thought into your error message; ask yourself, if I were to get it, what would I want to know? One rather obvious tip is to elaborate: we can embed any offending values to the description (preferably in their raw representation), just so the user can double-check they haven't accidentally passed in the wrong thing: seeing invalid value: 'a' might be just the clue we need when we'd thought we were passing an integer. When there are no relevant values to share, we can still expand on what exactly went wrong: operation failed: resource is busy at least lets us know what is going on — and suddenly it might dawn on us that this resource, despite not being user-generated, can be included in the message, so we know what exactly resource was busy (and maybe even why; do we know who else is using it?).

Another tip, which I can't stress enough, is to provide the user with advice on how to actually resolve this exception. Recall Scott Meyer's guideline, about interfaces that are easy to use correctly, and hard to use incorrectly: a system telling you what it does expect, or what needs to be done differently, is much to use (well, reuse) incorrectly. This can mean enumerating the acceptable options:

if arg not in options:
    options_list = ', '.join(repr(option) for option in options))
    raise ValueError('invalid argument: %r (expected one of: %s)'
                                           % (arg, options_list))

Or even adding a link to the appropriate online documentation, say if the object needs to be configured in some non trivial way in order for some method to work. This might blow up your exception raising code to multiple lines instead of a single raise-statement, or make your exception message run past the 80 characters per line recommended by PEP8 — but it's totally worth it. Just look at how nuanced and informative Python's own error reporting is — and how they keep adding new features to it still; this, I believe, plays a large role in how accessible (and popular) the language is.

Error Handling: Conclusion

Error Handling: Further Reading