Functions 3: Code as an Object

In the last couple of chapters, we talked a lot about functions: how their interfaces are structured, and how they themselves can interact in all sorts of clever ways.

After all this, we're finally ready to get to the key question — what actually are functions? We know that they are objects, but they're still a very special case; they're callable, have their own syntax, and come equipped to handle rich interfaces and complex invocation.

So this time, we'll cover how the function object is structured, see how its code object is organized, and even disassemble its bytecode and learn what exactly is going on when it runs. We'll also engage in some light metaprogramming, implementing a simple inversion of control and rewriting functions on the fly — so yeah, it's going to be fun.

For now, just recall how our discussion of functions started, with default arguments.

>>> def inc(x, delta=1):
...     return x + delta

>>> inc(1)
2

We saw that their actual values are often better to store separately, using the argument to express absence and only resolving it dynamically, at the last possible moment:

>>> default_delta = 1
>>> def inc(x, delta=None)
...     if delta is None:
...         delta = default_delta
...     return x + delta

>>> inc(1)
2

This way, other functions that use inc can pass through delta=None to get at the default behavior, without coupling their interfaces to its particular value. As for the value itself, we've extracted it out of the function body, so we can easily inspect and change it on demand.

We also addressed the drawbacks of having a loose, global variable just floating next to function, and saw that we can actually store it as part of the function object:

>>> def inc(x, delta=None)
...     if delta is None:
...         delta = inc.default_delta
...     return x + delta
>>> inc.default_delta = 1

>>> inc(1)
2

So far, so good.

Default Arguments, Revisited

But to be fair, this level of access was possible all along. You see, even if we use default arguments that are actual values, they are similarly stored in the function object, although in a different format. Like we said, a function is not simply a plain object — it represents an interface to some code, and that interface can include a lot of information, so the function object must have some native attributes that pertain to that.

Those attributes, considered low-level and internal, start and end with two underscores, conventionally called dunder — we already saw them in the class initialization method, __init__ ("dunder-init"), as well as in the function object itself. For example, we have the function's name in __name__:

>>> inc.__name__
'inc'

And we've seen the documentation string and annotations dictionary:

>>> def add(x: int, y: int) -> int:
...     'Returns the sum of and y.'
...     return x + y

>>> add.__doc__
'returns the sum of x and y'
>>> add.__annotations__
{'x': <class 'int'>, 'y': <class 'int'>, 'return': <class 'int'>}

Note how, even though this information is not needed to execute the function, it's still included in its object — because it's still part of its interface. As such, it can have additional meaning depending on the context, as we'll see in a moment.

As for default arguments, we actually glimpsed them once before as well, in __defaults__:

>>> inc.__defaults__
(1,)

It may feel a bit opaque and counterintuitive, but the way it works is that they're stored in a tuple, and assigned to the last positional parameters. With a signature like (x, delta), the single 1 would match the last parameter, delta, and that's indeed the case.

Through this native attribute, we can change a function's default values — although, since tuples are immutable, we can't just modify it and have to reassign the whole thing:

>>> inc.__defaults__ = (2,)
>>> inc(1)
3

We can even add a default argument to a parameter that doesn't have one, like x, and call inc with no arguments at all:

>>> inc.__defaults__ = (1, 2)
>>> inc()
3

Or remove the default arguments altogether, forcing both x and delta to be passed explicitly:

>>> inc.__defaults__ = ()
>>> inc(1)
TypeError: inc() missing one required positional argument: 'delta'

Honest Design

Likewise, if we have keyword-only parameters:

>>> def inc(*, x, delta=1):
...     return x + delta

>>> inc(x=1)
2

We can find them in a similar attribute:

>>> inc.__kwdefaults__
{'delta': 1}

And change them through there:

>>> inc.__kwdefaults__['delta'] = 2
>>> inc(x=1)
3

Notice how it's rather convenient to manage defaults through a dictionary, which is easier to modify and unpack than a tuple — so it would be fair to ask why aren't __defaults__ stored in a similar way.

And it turns out there's a good reason: keyword-only parameters, after all, are always passed by keyword, so their order doesn't matter — we can set or unset their default values as we wish. We can make x optional and delta mandatory, for all the sense it makes:

>>> del inc.__kwdefaults__['delta']
>>> inc.__kwdefaults__['x'] = 1
>>> inc(delta=2)
3

But with regular parameters, we have to account for the order, in case they are passed in by position and need to be matched this way. And in Python, to keep things simpler, we're not allowed to have an optional parameter before a mandatory one:

>>> def inc(x=1, delta):
...     return x + delta
SyntaxError: non-default argument follows default argument

The reason I'm dwelling on this is important; you probably won't touch this often, or ever — I just think it's really smart design. I used to find it clunky; to wonder why they haven't implemented some clever dictionary to expose the regular defaults, too, in an elegant and editable way.

But then, that dictionary would have all these implicit requirements, for what keys you can set when — and that would be surprising and error prone; dishonest design, if you will, that makes something complicated look easy. So when somebody looks at the defaults tuple and thinks "that's weird" — well, it's because it is a bit weird, so we better make sure we understand what's going on. We can't just set and unset keys arbitrarily; we have to make sure all the mandatory positional parameters come first, and the optional ones come later — so if there are default values, they'll be consecutive and last. At this point, "assigning an immutable tuple to the last position parameters", as we described a defaults update, doesn't sound that strange; in fact, it's a pretty clever solution to handle this particular weirdness in a pretty straightforward way.

The Code Object

What about the rest of the function's signature? Like, where are the argument names? So far, I've been introducing all the "peripheral" attributes, the defaults and the annotations, while sort of dancing around the main one. So here it is — the code object:

>>> inc.__code__
<code object inc at 0x...>

That the code is stored separately from the function might seem surprising — what is one without the other, after all? To explain it, we can imagine that they serve different purposes: the function object represents its interface with the world, while the code object is in charge of its actual operation. As we shall see, it's not exactly accurate — but it is the most plausible working assumption we have so far.

Anyway, this means that we can actually modify the code of an existing function. Suppose we have one that prints "Hello, world!":

>>> def hello():
...     print('Hello, world!')

We can use the builtin compile function to turn some different source code, which prints something else, into a code object:

>>> text = "print('something else')"
>>> code = compile(text, 'example.py', 'exec')

We need to give it some filename, which doesn't matter much, and a mode — 'eval' for expressions and 'exec' for statements and code blocks. In our case, it doesn't matter either, but generally we want the latter.

Now, we can swap this code in, and have the same function print something else, as promised:

>>> hello.__code__ = code
>>> hello()
something else

It's pretty crazy; but it only works because in this "function", there's no real interface to speak of. If there had been, some aspects of the interface and the code would bleed into each other, and this is where my neat little abstraction breaks down.

Looking for Parameters

If we want to find the function's parameter names — which are, undoubtedly, part of its interface — we'll have to go look in the code object; specifically, in co_varnames:

>>> def f(x, y):
...     print(x + y)

>>> f.__code__.co_varnames
('x', 'y')

That is, because they're also local variables in the code. Suppose we'd want to change the function to print their product instead of their sum; we can swap in the appropriate code:

>>> text = 'print(x * y)'
>>> code = compile(text, 'example.py', 'exec')
>>> f.__code__ = code

But it doesn't work; when we provide the arguments, Python complains it doesn't want them:

>>> f(2, 3)
TypeError: f() takes 0 positional arguments but 2 were given

And when we omit them, it complains about that, too:

>>> f()
NameError: 'x' is not defined

What's happening is that the code itself is different: the original one operated on a local scope, while the one we prepared, without more context, has no way of knowing that x and y are parameters. Naturally, it assumes they're global — so in fact, it doesn't expect any arguments whatsoever, and the the first error actually makes perfect sense.

Accordingly, if we set those names in the global scope, the second error goes away:

>>> x = 2
>>> y = 3
>>> f()
6

To sum up: function objects are separate from code objects, and their purpose is to provide an interface for them — but the two concepts overlap, since an interface and its code are invariably coupled. With that in mind, it shouldn't be so surprising that this is where we find the parameter names; and actually, any local variables as well, since from the code's perspective there's no difference between the two:

>>> def f(x, y):
...     z = x + y
...     print(z)

>>> f.__code__.co_varnames
('x', 'y', 'z')

To tell them apart, we have the co_argcount attributes; and since the parameters necessarily precede any local variables, we can effectively crop them out:

>>> f.__code__.co_varnames[:f.__code__.co_argcount]
('x', 'y')

Deciphering the Signature

Similarly, co_posonlyargcount tells us how many position-only parameters we have:

>>> def f(x, /, y):
...     z = x + y
...     print(z)

>>> f.__code__.co_varnames[:f.__code__.co_posonlyargcount]
('x',)

And co_kwonlyargcount tells us how many keyword-only arguments, starting after the regular arguments, we can expect:

>>> def f(x, *, y):
...     z = x + y
...     print(z)

>>> start = f.__code__.co_argcount
>>> stop  = f.__code__.co_argcount + f.__code__.co_kwonlyargcount
>>> f.__code__.co_varnames[start:stop]
('y',)

As for variadic parameters, we'd have to check co_flags, and do some bitwise arithmetic to see if a function has a variadic positional parameter and/or a variadic keyword parameter (as well as some index math, if we want to figure out which varname is what):

>>> def f0():
...     pass
>>> def f1(*args):
...     pass
>>> def f2(**kwargs):
...     pass
>>> def f3(*args, **kwargs):
...     pass

>>> import inspect
>>> for f in (f1, f2, f3, f4):
...     has_args   = bool(f.__code__.co_flags & inspect.CO_VARARGS)
...     has_kwargs = bool(f.__code__.co_flags & inspect.CO_VARKEYWORDS)
...     print(f.__name__, has_args, has_kwargs)
f0 False False
f1 True False
f2 False True
f3 True True

All in all, deciphering a signature from a code object is a bit tedious, and we'll learn a better way to parse it soon enough. But why is this even useful? If you think about, it's not entirely clear: if functions are a way to organize our codebase, why would we need access to all these internals, like checking the function's name? Isn't it obvious from, well, the function's name, as it appears in its definition?

A Brief Intro to Metaprogramming

And of course it is — when the code is read by a human; that's why in most programming languages, functions are not objects, and there's no easy way to access a function's internals programatically.

However, a language that does allow it, making its code programmatically accessible to itself, makes room for a whole new and fascinating paradigm to emerge. It's called metaprogramming: literally, writing code that is about other code. A silly example might be a function that receives another function and prints its name and docstring:

>>> def document(f):
...     print(f.__name__)
...     print(f.__doc__)

>>> document(add)
add
Returns the sum of x and y.

This is effectively one code object working on another, isn't it? And it's more or less what's the builtin help function does — although it also bothers to decipher and present the full signature:

>>> help(add)
Help on function add in module __main__:

add(x: int, y: int) -> int
Returns the sum of x and y.

This approach comes in useful in some common scenarios that are code-centric by definition: tests, for example, are code whose purpose is to make sure that some other code runs correctly. In this case, being able to analyze code is handy to say the least; especially when it fails, and we can automatically infer why, or at least where, like the pytest library does.

Similarly, documentation can be auto-generated by code that examines other code and summorizes it in a friendlier manner: listing the function names, parameters and descriptions in a small website, derived from programatically scanning the codebase.

You've probably seen such similar websites with library documentation and API references — even the Python documentation itself. The reason they all look alike is that they're auto-generated from the same templates, by the same tools: all the examples you read there are written, verbatim, in the actual source code, as docstrings — and then automatically assembled and repackaged by a library like sphinx.

Inversion of Control

More interestingly, metaprogramming unlocks new possibilities in software design. In many cases, the code we write is used by other programmers — that is, by other code. The most standard example of this is reusable libraries with generic functions, where a client uses the library by passing in some arguments, then deferring to its computation; like with math, where we pick the numbers and don't worry about the formula:

>>> import lib
>>> lib.formula(1, 2)
3

In such cases, our code calls into the library to do something on its behalf. A different scenario, known as inversion of control, reverses this relationship: it has the library call into our code — for example, when some event happens. This is typical of libraries that provide a complex framework, like running a webserver; they don't expect us to tell them what to do, but actually take the reins, and only need us to fill in some gaps, like providing a handler that would be called on incoming requests:

>>> def handle_request(request):
...     ...

>>> lib.run_webserver(handle_request)

We've seen that Python lets us pass functions as arguments, so this is easy enough to achieve; but what if the function needed access to some complex, expensive object, like a database connection? Constructing it on demand, like so:

>>> def handle_request(request):
...     db = lib.connect_database('...')
...     ...

Would recreate it for ever request. If we want to do it once and keep reusing the same copy, we either have to keep it as some unmanaged, global variable, or find a way to register it along with our handler, maybe as an extra argument:

>>> def handle_request(request, db):
...     ...

>>> db = lib.connect_database('...')
>>> lib.run_webserver(handle_request, db)

Python can handle this, too, using the omnisignature and perfect forwarding, so this definitely works. But it also introduces a needless detour, and subtle code duplication: the arguments we explicitly generate are the same as the argument in the function definition — in this case, db — so if one changes, the other needs to be updated accordingly, or the code will break. What's more, we're reaching into the library to generate an object that we then pass back into it — what if we found some better way to request them?

Dependency Injection

In a spin on a design pattern known as dependency injection, we can pass our function to the library, like before — but then, the library can actually inspect our function's signature, infer which arguments it expects, generate them, and pass them in of its own volition whenever that function is invoked.

Doing this is actually easier than it sounds, at least once we understand what the hell is it that we're trying to do; that's why I find metaprogramming in Python particularly delicious. Putting on our librarian's hat, suppose some client code provides us with a function, and we need to figure out how to call it. We have a pool of potential, supported arguments — let's do x=1 and y=2 to keep things simple, but the same holds true for a database connection. Our job is to inspect the function's parameters, filter only the relevant options that match by name, and invoke the function on this subset alone:

>>> def inject(f, options):
...     params = f.__code__.co_varnames[:f.__code__.co_argcount]
...     kwargs = {}
...     for name, value in options.items():
...         if name in params:
...             kwargs[name] = value
...     return f(**kwargs)

This works with no arguments, for x only, for y only, and for x and y both:

>>> def f0():
...     print('f0')
>>> def f1(x):
...     print('f1', x)
>>> def f2(y):
...     print('f2', y)
>>> def f3(x, y):
...     print('f3', x, y)

>>> options = {'x': 1, 'y': 2}
>>> inject(f0, options)
f0
>>> inject(f1, options)
f1 1
>>> inject(f2, options)
f2 2
>>> inject(f3, options)
f3 1 2

This nifty trick is actually used, in one way or another, by many sophisticated frameworks. In pytest, for example, such arguments are known as fixtures: if we add them to a test function, pytest will know to set them up before invoking this particular test case, and clean them up after it's over. If we were testing some writing functionality and wanted to do it ourselves, we'd have to generate some temporary path and make sure it's deleted regardless of the test's outcome:

>>> import os, tempfile
>>> def test_write_hello():
...     path = tempfile.mktemp()
...     try:
...         write_hello(path)
...         assert open(path).read() == 'hello world'
...     finally:
...         os.unlink(path)

But rather than repeating this clunky boilerplate every time, we can delegate it to pytest, simply by adding the tmp_path argument to the function's signature (and note how the argument name has to be exactly that, so pytest knows what we're talking about):

>>> def test_write_hello(tmp_path):
...     write_hello(tmp_path)
...     assert open(tmp_path).read() == 'hello world'

Parsing Signatures

In inject's implementation, all we cared about were the parameter names, so working directly with the code object wasn't too bad. However, when we're trying to do something more substantial with a function's signature, we probably shouldn't be deciphering its low-level internals like every time. Take this function, for example — it would be pretty tedious to figure all of its nuances:

>>> def f(x: bool, /, y: str, z: int = 3, *args: float, w=4, **kwargs):
...     pass

Luckily, Python comes equipped with a builtin metaprogramming module in its standard library, called inspect. In it, getargspec was the first function to parse signatures — but as new features piled up, they added getfullargspec, returning a named tuple with all the parsed information:

>>> import inspect
>>> inspect.getfullargspec(f)
FullArgSpec(args=['x', 'y', 'z'], varargs='args', varkw='kwargs', defaults=(3,), kwonlyargs=['w'], kwonlydefaults={'w': 4}, annotations={'x': <class 'bool'>, 'y': <class 'str'>, 'z': <class 'int'>, 'args': <class 'float'>})

And eventually the Signature class, which is the most refined of the three:

>>> s = inspect.signature(f)
>>> s
<Signature (x: bool, /, y: str, z: int = 3, *args: float, w=4, **kwargs)>

This does all the parsing for us, and organizes everything in a convenient object with a parameters dictionary — so for each parameter we can check its type, default value, annotation and so on:

>>> for name, param in s.parameters.items():
...     print(name, param.kind, param.default, param.annotation)
x POSITIONAL_ONLY <class 'inspect._empty'> <class 'bool'>
y POSITIONAL_OR_KEYWORD <class 'inspect._empty'> <class 'str'>
z POSITIONAL_OR_KEYWORD 3 <class 'int'>
args VAR_POSITIONAL <class 'inspect._empty'> <class 'float'>
w KEYWORD_ONLY 4 <class 'inspect._empty'>
kwargs VAR_KEYWORD <class 'inspect._empty'> <class 'inspect._empty'>

Note that for parameters that don't have a default value or an annotation, a special object is used to indicate this — inspect.Parameter.empty; so we can get a nicer report checking for this directly:

>>> for name, param in s.parameters.items():
...     default = param.default
...     if default is inspect.Parameter.empty:
...         default = '<no default>'
...     annotation = param.annotation
...     if annotation is inspect.Parameter.empty:
...         annotation = '<no annotation>'
...     print(name, param.kind, param.default, param.annotation)
x POSITIONAL_ONLY <no default> <class 'bool'>
y POSITIONAL_OR_KEYWORD <no default> <class 'str'>
z POSITIONAL_OR_KEYWORD 3 <class 'int'>
args VAR_POSITIONAL <no default> <class 'float'>
w KEYWORD_ONLY 4 <no annotation>
kwargs VAR_KEYWORD <no default> <no annotation>

At this point, we pretty much covered everything interface-related. Besides this, the code object also stores some metadata, like its name (well, its function's name), the file in which it was defined, or the first line number in that file at which the code starts:

# example.py

def f():
    x = 1 + 2

>>> from example import f
>>> f.__code__.co_name
'f'
>>> f.__code__.co_filename
'example.py'
>>> f.__code__.co_firstlineno
3

But of course, and maybe most importantly, it stores the essence of the function — the code that gets executed when it's invoked. You can find it in co_code:

>>> f.__code__.co_code
b'd\x01}\x00d\x00S\x00'

Except, of course, it doesn't look like code at all.

The Python Interpreter

Before we can make sense of this strange string, we need to understand how python works. I'm talking about the program itself, known as the Python interpreter — the command that takes a file with Python code and makes it happen, or opens an interactive session for us to experiment in.

Contrary to some popular explanations about how interpreters operate, Python doesn't just read the file line by line, parsing and executing them as it goes. In fact, it reads the entire file first, then breaks it into tokens, and organizes those in a structure called an abstract syntax tree, or AST. A simple program like x = 1 + 2 has to be broken down into the tokens x, =, 1, +, and 2, and then arranged in a hierarchy that represents how the tokens interrelate, which naturally assumes the shape of a tree:

TODO@image: ast.png

The AST is then compiled — translated into a series of simple, limited and low-level commands, known as opcodes; in this case, something along the lines of:

LOAD_CONST 1
LOAD_CONST 2
BINARY_ADD
STORE_NAME x

And it's only this concrete and minimal set of instructions that the Python interpreter really understands and can actually carry out.

This might feel a bit elaborate, but all high-level programming languages undergo a similar process. This essentially means all programming languages, except for very few and very low-level so-called assembly languages, which can run directly on a device.

You see, to get any device to do something using assembly language, we'd have to deal with very particular, and often rather quirky commands, designed to make sense to its particular (and often quirky) hardware. Any program we write this way wouldn't work on anydevice that doesn't adhere to the exact same standard.

This is why high-level programming languages were invented to begin with — to make programming across devices possible. The first (and relatively easy) challenge was to come up with a general-purpose syntax that would make sense for the various cases; the second, much more substantial challenge was (and is) to actually implement and maintain the mechanism by which this abstract representation is mapped to and executed on actual and different hardwares.

In compiled languages like C, Go and Rust, this is done by a compiler program, like gcc or msvc. Such a program takes text files with source code, and translates them into binary files with machine code, which a given device can actually execute. This way, the same source code can be reused across devices: the compiler has to rewrite it in assembly language anyway, and can do so for any of the supported devices.

Note that this happens well before the program is actually executed — which means the process can take all the time it needs, perform extra checks and figure out extra optimizations. Its results tend to be safer and faster, but they're still inherently limited; namely, they are tailored to the operating system and chip architecture on which they are intended to work. That's why you can't run a Windows program on a Linux system, or a 64-bit program on a 32-bit machine.

Also, it tends to be less flexible and dynamic, since it's so carefully packed in advance. That's why scanning and manipulating a program that's already running in memory is notoriously difficult; something only sophisticated and specialized software does, like malware or anti-virus systems.

That's what interpreted languages, like Python, Java and Javascript aim to address. Instead of turning the source code into a standalone binary program, they read the source code and operate the machine on its behalf. This way, the only tailored program we have to compile per device is python, the interpreter itself; once it's installed, any Python program should work across all of them.

This approach presents other drawbacks, however; chiefly performance. Compilers can take their time in analyzing the source code; but an interpreter may encounter a program for the first time as it begins executing it, so whatever guarantees it can make about its validity and coherence, it has to do in real time, and it better be quick.

First, it would have to comprehend the words, the syntax, by tokenization; then see how the tokens relate and give rise to meaning, the semantics, by parsing it into an AST. Finally, this has to be translated into practical, actionable instructions — a part that's confusingly called compilation as well. These instructions can then be executed by a component known as a the Python virtual machine, or VM, which emulates the actual machine on which we run. The Python interpreter, to be clear, is largely a Python VM, and sometimes the two terms are used interchangeably (although technically, the interpreter includes some other stuff, like the interactive sessions functionality).

Anyway, that's the reason for the elaborate process. Programming is sometimes misrepresented as "communicating with a computer", when in fact — at least, at this point — it's built on so many layers of abstraction that it isn't even close. If we really were communicating with the computer, running a program would be simple, if severely limited and particular; so in actuality, it requires sophisticated analysis and multiple steps of translation and retranslation before anything really happens.

As an aside, despite any reputation for being slow, Python goes out of its way to optimize this process. For example, it caches things aggressively — we've already seen that functions are only compiled once, which is why their default arguments shouldn't be mutable. Even between executions, Python attempts to keep what it had already figured out, like previously encountered code; it does so in .pyc files and __pycache__ directories you might have seen littering your codebase.

So the "safer and faster" argument for compiled languages is not as crucial as you might think: we can still establish a separate build step, during which we can do any checks and optimizations we want. A lot of time-critical systems are still written in compiled (or even assembly) languages, but we'll see that Python can achieve similar (and sometimes surprisingly better) speeds when we talk about performance in depth.

So yeah, what really sets interpreted languages apart is that they're not actually translated into machine code, but into generic bytecode: an intermediate representation that gets executed by some interpreter program, some VM. This VM is still compiled for the system, which is why we can't run a Python executable from Windows on Linux; but once a Python VM is properly installed on either, we can run any Python program on both — in other words, it makes the language cross-platform.

Emulating Interpretation

Interestingly, all the steps we discussed can be reproduced using the standard library.

To tokenize a string, we use the tokenize module; its generate_tokens function has a bit of a weird interface, in that it accepts not a string, but a function that reads it line by line. We can emulate this by wrapping our string in a file-like io.StringIO object, and passing its readline method:

>>> import io, tokenize
>>> text = 'x = 1 + 2'
>>> for token in tokenize.generate_tokens(io.StringIO(text).readline):
...     print(token)
TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1))
TokenInfo(type=54 (OP), string='=', start=(1, 2), end=(1, 3))
TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5))
TokenInfo(type=54 (OP), string='+', start=(1, 6), end=(1, 7))
TokenInfo(type=2 (NUMBER), string='2', start=(1, 8), end=(1, 9))
TokenInfo(type=4 (NEWLINE), string='', start=(1, 9), end=(1, 10))
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0))

So indeed, our tokens are x, =, 1, + and 2; and then a newline and an end of file marker, alright. To organize them in an AST, we can use the ast module; it actually does its own tokenization, so all it expects from us is the original string:

>>> import ast
>>> a = ast.parse('x = 1 + 2')
>>> a
<ast.Module object at 0x...>

This object can be quite complex, and not easy to inspect — so ast also provides the dump function, which elaborates on its structure:

>>> print(ast.dump(a, indent=4))
Module(
body=[
Assign(
    targets=[
        Name(id='x', ctx=Store())],
    value=BinOp(
        left=Constant(value=1),
        op=Add(),
        right=Constant(value=2)))],
type_ignores=[])

Again, quite similar to our initial guess: we have an assignment statement, operating on the target name x, with its value being the result of a binary operation, comprising of the addition operator and the constants 1 and 2.

This AST is then compiled — that's the builtin compile function, which we've already seen — producing a code object whose bytecode is stored in co_code:

>>> code = compile('x = 1 + 2', 'example.py', 'exec')
>>> code.co_code
b'd\x00Z\x00d\x01S\x00'

Python Bytecode

At this point, we're ready to decipher its meaning. It's not actually a string, but an array of bytes, with each character representing a value between 0 and 256, as converting them to a list of integers shows:

>>> list(code.co_code)
[100, 0, 90, 0, 100, 1, 83, 0]

In newer versions of Python, instructions always come in pairs: an opcode, designating the operation, and an argument, if any. To pair them up, we can turn the list into an iterater and zip it with itself:

>>> it = iter(code.co_code)
>>> opcodes = list(zip(it, it))
[(100, 0), (90, 0), (100, 1), (83, 0)]

And to convert them to their readable names, called mnemonics, we can use the opname dictionary in the opcode module:

>>> from opcode import opname
>>> for opcode, arg in opcodes:
...     print(opname[opcode], arg)
LOAD_CONST 0
STORE_NAME 0
LOAD_CONST 1
RETURN_VALUE 0

The various opcodes are described in the dis module documentation. The module itself provides helpful functions to do exactly what we've been doing manually; its name is a shorthand for disassembly, the process in which we take apart and understand this low-level, assembly (well, assembled) code:

>>> import dis
>>> dis.disassemble(code)
  1           0 LOAD_CONST               0 (3)
              2 STORE_NAME               0 (x)
              4 LOAD_CONST               1 (None)
              6 RETURN_VALUE

That's honestly be a bit different to what we expected. The first opcode is LOAD_CONST 0, which dis actually elaborates on, adding (3) next to it — but where are our actual values, 1 and 2?

What we're seeing is actually one of Python's optimizations in action: it figured that it can precalculate the addition, so it doesn't need to be recalculated every time, effectively changing our code to x = 3.

That's cool, but where does this 3 come from? The argument for the opcode itself is 0. However, if you think about it, the opcode's argument can't really hold the value, anyway — it's a number between 0 and 256, unfit for anything larger, let alone strings or sequences. In fact, it's an index into a separate structure — in this case, a tuple of all the constants used in the code (hence LOAD_CONST), found in co_consts:

>>> code.co_consts
(3, None)

So at index 0 we have 3 indeed. Next, we have STORE_NAME 0, which again refers to a separate tuple, time time co_names:

>>> code.co_names
('x',)

The Python Stack

The way these two opcodes — loading a constant, then storing it in a name — are linked together is through a stack. The bytecode is so limited, it doesn't even support variables; the way values are passed around is by essentially "leaving them" in a common area, in an order agreed upon with the instructions that follow. We can emulate this behavior with a Python list and its two basic methods, append (which adds an item at its end) and pop (which removes and returns its last item):

>>> stack = []
>>> stack.append(1)
>>> stack.pop()
1

Loading a constant, then, leaves something on (or appends it to) such a stack:

>>> stack.append(3) # LOAD_CONST 0 (3)

While storing a name pops what it needs off of it:

>>> x = stack.pop() # STORE_NAME 0 (x)
>>> x
3

If there was no optimization, and we'd actually have the full calculation as suggested in our initial guess, they way it looks would be:

LOAD_CONST 0 (1)
LOAD_CONST 1 (2)
BINARY_ADD
STORE_NAME 0 (x)

So first, loading the two constants:

>>> stack.append(1) # LOAD_CONST 0 (1)
>>> stack.append(2) # LOAD_CONST 1 (2)

Then having the addition, as a binary operation, pop both, do its thing, and push the result:

>>> stack.append(stack.pop() + stack.pop()) # BINARY_ADD

Which we can then assign like before:

>>> x = stack.pop() # STORE_NAME 0 (x)
>>> x
3

Note that the last operations, which we didn't really write ourselves, are LOAD_CONST 1 and RETURN_VALUE. In Python, every function and every code have return values — even if there's no explicit return-statement, in which case it's assumed to be None. That's why we saw it among the constants:

>>> code.co_consts
(3, None)

Specifically, at index 1 — so that's what we load and return at the end; our code, in full, would be:

x = 1 + 2
return None

Scopes and Calls

Let's take a slightly more complicated example:

>>> def f(x, y):
...     z = x + y
...     print(z)

First, we check the constants and names it uses:

>>> f.__code__.co_consts
(None,)
>>> f.__code__.co_names
('print',)

That there are no constants, only a None for the implicit return statement, is to be expected; but what's up with the names? It's a good point that print is actually a name, too — but where are x and y and z?

We've seen them already, of course, in co_varnames:

>>> f.__code__.co_varnames
('x', 'y', 'z')

That's where the local variable names are stored. In the previous example, when we compiled a raw code object directly, it had no notion of a local scope — so x was global by default, part of co_names. In bytecode terminology, then, names refer to global names, really; and of those, the only one we really use is indeed print. Looking at the actual bytecode:

>>> dis.dis(f) # shorthand for dis.disassemble(f.__code__)
  2           0 LOAD_FAST                0 (x)
              2 LOAD_FAST                1 (y)
              4 BINARY_ADD
              6 STORE_FAST               2 (z)

  3           8 LOAD_NAME                0 (print)
             10 LOAD_FAST                2 (z)
             12 CALL_FUNCTION            1
             14 POP_TOP
             16 LOAD_CONST               0 (None)
             18 RETURN_VALUE

We can wee that instead of LOAD_NAME 0/1, which would resolve a name in the global scope, we have LOAD_FAST 0/1, which resolves a (var)name in the local one; and notice how the index indeed refers to the co_varnames tuple.

So, like we've seen before, both x and y are pushed unto the stack; BINARY_ADD pops them, adds them, and pushes back their sum; and STORE_FAST 2 associates the value with the third local variable, z.

Then, to call a function using the stack — which involves communicating an unknown number of arguments to it — we do something curious: push the function first, then its argument, then do CALL_FUNCTION 1. CALL_FUNCTION's argument is not an index like the other arguments we've seen so far — it indicates how many arguments the function expects, and need to be popped off the stack. After this, one more argument is to be popped, to get the function itself; and after it returns, the return value is pushed unto the stack in its stead — something like:

>>> stack.append(print)
>>> stack.append(z)
>>> num_args = 1 # CALL_FUNCTION's argument
>>> args = [stack.pop() for i in range(num_args)]
>>> func = stack.pop()
>>> stack.append(func(*args))

That why, when a function's return value isn't used (like in our case), it's usually followed by an immediate POP_TOP to clear it up and keep the stack tidy:

>>> stack.pop()

Rewriting Bytecode

Knowing what we know, we can finally modify function code in a non-trivial way — lets change the addition into multiplication, like we failed to before. Looking through opname, we can see BINARY_MULTIPLY; to get the reverse relation of name-to-number, we can use opmap:

>>> from opcode import opmap
>>> opmap['BINARY_ADD']
23
>>> opmap['BINARY_MULTIPLY']
20

So we can take the function's opcodes as numbers:

>>> opcodes = list(f.__code__.co_code)
>>> opcodes
[124, 0, 124, 1, 23, 0, 125, 2, 116, 0, 124, 2, 131, 1, 1, 0, 100, 0, 83, 0]

And change 23 to 20:

>>> opcodes[opcodes.index(23)] = 20

Now, the code object itself is unfortunately immutable:

>>> f.__code__.co_code = bytes(opcodes)
AttributeError: readonly attribute

So in past versions of Python, we'd have to create a new code object, cloning over the many different fields it has, even if we only modified one. Luckily, it now has a replace method, which takes our changes and returns a similar code object with them applied. We only want to touch co_code, so:

>>> f.__code__ = f.__code__.replace(co_code=bytes(opcode))

And now, drumroll…

>>> f(2, 3)
6

Which is, indeed, their product; how about that.

The Disassembly Format

To finish up, I'd like to comment on two more things. First, we have all those strange numbers in our disassembly:

  2           0 LOAD_FAST                0 (x)
              2 LOAD_FAST                1 (y)
              4 BINARY_ADD
              6 STORE_FAST               2 (z)

  3           8 LOAD_NAME                0 (print)
             10 LOAD_FAST                2 (z)
             12 CALL_FUNCTION            1
             14 POP_TOP
             16 LOAD_CONST               0 (None)
             18 RETURN_VALUE

The inner ones, right next to the opcodes, are their offsets within the bytecode — you can see it growing, two at a time. The numbers on their right are actually their line numbers; since a standard Python statement (like z = x + y) can translate into multiple opcodes, there's a special line number table to keep track of it, co_lnotab:

>>> f.__code__.co_lnotab
b'\x00\x01\x08\x01'

Which encodes the information necessary to drive which opcodes belong to each line. Its once again a list of numbers:

>>> list(f.__code__.co_lnotab)
[0, 1, 8, 1]

Which, starting from the first line:

>>> f.__code__.co_firstlineno
1 # always 1 for functions defined in an interactive session

Tells us that at index 0, we need to break 1 line — moving us at line 2; and at index 8, we need to break again, moving us to line 3; so our conclusions are the same as dis's:

  2           0 LOAD_FAST                0 (x)
                ...

  3           8 LOAD_NAME                0 (print)

Closures and Cells

Lastly, a word about non-local variables. So far, we've seen global names and local (var)names; but if we have something like this:

>>> def f():
...     x = 1
...     def g():
...         print(x)
...     return g

Then we're dealing with a non-local scope. And indeed:

>>> dis.dis(f)
  2           0 LOAD_CONST               1 (1)
              2 STORE_DEREF              0 (x)
              ... # cropped

Disassembly of <code object g at...>:
  4           0 LOAD_GLOBAL              0 (print)
              2 LOAD_DEREF               0 (x)
              4 CALL_FUNCTION            1
              6 POP_TOP
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

In the disassembly of g, we see x loaded with LOAD_DEREF 0, the opcode used to dereference this strange pointer into an alien scope. The index refers to co_freevars; it's called this way to contrast the local variables, bound to the current scope, with variables that are unbound — that is, free:

>>> g = f()
>>> g.__code__.co_freevars
('x', 1)

The curious thing, though, is that x is treated like a special, non-local variable in f as well; it's stored with STORE_DEREF instead of STORE_FAST, even though it looks local. This is because the implementation of non-local name resolution is such that all the names involved are isolated into what's known as a closure, where they are kept in separate cells (so much for being free):

>>> g.__closure__
(<cell at 0x...>,)
>>> g.__closure__[0].cell_contents
1

That 1 is the value of x, and the way it can be accessed from g. On f's side, x is considered a cell variable:

>>> f.__code__.co_cellvars
('x',)

And it's made to reference (or rather, dereference) the same cell, thus guaranteeing synchrony between the various parties interested in the same non-local variables.

Rewiring Scopes

You might have noticed that the __closure__ attribute itself is part of the function object, not the code. The same is true for another scope-related attribute we need — __globals__, the dictionary through which we resolve global names like print. If we go back to a function that uses it, for a moment:

>>> def hello():
...     print('hello world')

>>> dis.dis(hello)
  2           0 LOAD_GLOBAL              0 (print)
              2 LOAD_CONST               1 ('hello world')
              4 CALL_FUNCTION            1
              6 POP_TOP
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

We see that it's accessed with LOAD_GLOBAL. In a previous disassembly it was LOAD_NAME, but only because I wanted to drive the point that names are global; it used to be the terminology (hence co_names), but has recently been refactored to the more specific LOAD_GLOBAL, LOAD_FAST (local), and LOAD_DEREF (non-local), with LOAD_NAME used in vague cases like a scopeless, raw code object. In any case, we see that index 0 reference the name print:

>>> hello.__code__.co_names
('print',)

And the name print, being global, needs to be resolved through __globals__:

>>> hello.__globals__['print']
KeyError: 'print'

Or at least, it used to be there; all the builtin functions used to be part of the global scope. Now they're tucked away separately, in __builtins__:

>>> hello.__builtins__['print']
<built-in function print>

And conveniently, that's the second place LOAD_GLOBAL's looks in. If we were to override it in hello's global scope, replacing it with a version that does nothing:

>>> hello.__globals__['print'] = lambda *args: None

Then that's what it would find, and the message would be suppressed:

>>> hello()
# nothing

So why are some of the attributes, from __defaults__ and __annotations__ to __closure__ and __globals__, and some are stored in the code object, in its various co_* fields?

Honestly, it's not entirely clear cut; it's just the way things are organized, and it's possible to concieve of organizing them differently. But very roughly, we can think of it this way: the code object is primarily about the bytecode, and the immediate fields necessary to make sense of it. This includes the various tuples needed to resolve opcode arguments, information necessary to derive the line numbers and other metadata — everything that makes sense for a raw, scopeless code object as well. A function object, then, places it within a scoped relationship with the rest of the code: what default values are available and how globals and non-locals names are resolved, as well as interface information that is entirely irrelevant to the code object, like docstrings and annotations.

Also, as you can see from this chapter, the details are minute, and keep changing often — it's probably the part I've rewritten most frequently over the years. Such internals are rarely touched by users, and are intended to make the rest of the language work well — so they're designed primarily with this goal in mind, and redesigned when the language changes in ways that warrant a new approach. Still, it's an incredibly powerful toolbox, and understanding it really clarifies what goes on behind the scenes, even when we type something as simple as x = 1 + 2.

Functions 3: Conclusion

So yeah, functions. We've talked about their interfaces, their capabilities, and finally their structure: from the function object to the code object and its bytecode, we saw how the different parts click together and why.

We also saw how this lets us bend the language in creative ways, like a dependency injection that infers what arguments a function has, generates them and calls it accordingly; or other metaprogramming scenarios, from practical testing and documentation libraries to playful rewriting-on-the-run (although this, too, is sometimes used by enterprise software, e.g. to seamlessly inject code that monitors the execution of a complex system for bugs, performance bottlenecks or security vulnerabilities). The understanding that code is not simply "talking to the computer", but actually a complicated process with layers upon layers of abstraction can seem daunting at first — but once we get used to it, we can start imagining new layers of our own; even dream up the next generation of code, a topic we'll get to in due time.