Functions 1: Signatures as Interfaces

The next few chapters are about functions.

In the first one, we'll consider them as interfaces, and focus on their signatures — from default arguments to variadic and something-only parameters, annotations and more. In the following chapters, we'll consider them as objects, its implications on various programming paradigms, and all the way down to their internal representation and bytecode.

Functions are arguably the most important idea in programming — or at least in software design.

While we can program without them, and people did that for a while, there is a limit to how big and complicated an idea we can hold in our head. Beyond this limit, we need to break it down and organize everything in smaller, simpler concepts, which interrelate and interact in coherent and coordinated ways.

In other words, we need good abstractions for our flows and data, and good interfaces between them. Each programming language already provides us with some of the most basic, useful and generic abstractions, from numbers and strings to conditions and loops, through the interface of syntax — but functions are effectively the first building block that we get to design ourselves. With functions, we get to decide how to augment and customize the language, so that it's easier and clearer for us to articulate our intent.

I find that a lot of people treat functions rather casually, when in fact these are the elementary particles that make up a codebase — but we'll talk about it more when we revisit functions from a perspective of software design.

The Function's Core

For now, the way it works should be familiar enough: we extract a piece of code and give it a name, after which we can call or invoke it to run this piece of code. In Python, we define a function with a def statement, and invoke it with parentheses:

>>> def hello():
...     print('hello world')
>>> hello()
hello world

We can also use return to communicate a result, called the return value, back to the caller:

>>> def three():
...     return 1 + 2
>>> x = three()
>>> x
3

If we don't, the function still returns at some point, if only when it ends; and it still has a return value, even if it's just None:

>>> def hello():
...     print('hello world')
...     # implicit return None
>>> x = hello()
hello world
>>> x
None

Incidentally, if all our function does is return a single expression, we can also define it as a lambda:

>>> lambda: 1 + 2
<function <lambda> at 0x...>

Although if we want to call it later, we should probably bind this lambda to a name:

>>> three = lambda: 1 + 2
>>> three()
3

In short, functions let us isolate some code and execute it later, on demand, or repeat it several times.

But what makes functions truly powerful, is that this code can be parametrized — it can take some input, and return some output accordingly. This is the function's signature, the most fundamental interface in software.

In the function definition, we can specify some parameters, making its code dependant on their values; and when the function is called, we provide actual values as arguments, to run its code within that particular context — either to customize the function's behavior, or so that a corresponding return value can be derived:

>>> def hello(name):
...     print('hello', name)
>>> hello('alice')
hello alice

>>> def add(x, y):
...     return x + y
>>> add(1, 2)
3

This input to output relationship, represented by the function's name, is our abstraction: it's the new, custom word we add to the language, so that we can organize our program and better integrate its parts. That's why understanding the protocol that governs its operation, its nuances and possibilities, is infinitely important — and why it'll take us this entire chapter to cover.

Default Arguments

Let's start with default arguments.

Suppose we have a function with an optional parameter; some value that, if provided, will augment its logic, but otherwise isn't strictly necessary. This is indicative of the function being somewhat generic — aiming to fulfil more than one case, and using this parameter not so much for input, as for configuration: a way to calibrate what is otherwise a reasonable execution with default values and behaviors.

For example, we can try saying hello to someone, but resort to saying "hello, stranger" if we don't know their name — that is, if name is None, an empty string, or otherwise false:

>>> def hello(name):
...     print('hello', name or 'stranger')

>>> hello('alice')
hello alice
>>> hello(None)
hello stranger

But if so — and here, imagine a function with many such optional parameters — we don't want to be passing a bunch of awkward Nones every time we want the default case.

Instead, we can associate optional parameters with default arguments, making it possible to omit them entirely from the call, in which case they — well, default to that argument, see?

>>> def hello(name='stranger'):
...     print('hello', name)

>>> hello('alice')
hello alice
>>> hello()
hello stranger

This can make our interfaces, our functions — especially the generic, reusable one — much cleaner, as it narrows the scope of immediate concerns; it's now only about the input, the mandatory parameters, with any other tweaking provided, or even mentioned, only on demand.

Mutable Default Arguments

But already, there are a few caveats.

The most famous one is that if our default argument is a mutable object — one that can change over time, like a list or a dictionary — then we might run into trouble.

Suppose we have a function that appends an item to a list, thus mutating or changing it; and if no list is provided, it defaults to an empty one:

>>> def append(item, items=[]):
...     items.append(item)
...     return items

It works well enough for an actual list:

>>> items = [1, 2, 3]
>>> append(4, items)
[1, 2, 3, 4]

And it seems to work without one as well:

>>> append(1)
[1]

But only once — calling it again keeps appending the values to the same list, which is probably not what we want:

>>> append(1)
[1, 1]
>>> append(1)
[1, 1, 1]

The key thing to understand here is that default arguments are evaluated once, when the function is defined — and then reused, whenever it's called and no actual argument is provided. Mutable objects like lists are unsuitable for this purpose, because they accumulate the changes from multiple (and probably unrelated) executions.

In other words, they make our interfaces stateful; our function is no longer an input-output relation, but one that depends on the history and state of the program, and this generally tends to make things less predictable and more complicated.

Testing for Absence

Luckily, it's easy enough to fix: we can allocate the list not in the function definition, but in the function execution. Instead of having a default argument with an actual value, we can make it None, to indicate the absence of an argument. This way, whenever it's absent, our first order of business would be to assign it with a fresh value — like a new, dedicated list:

>>> def append(item, items=None):
...     if not items:
...         items = []
...     items.append(item)
...     return items

It's a pretty comon idiom, and it works well:

>>> append(4, items)
[1, 2, 3, 4]
>>> append(1)
[1]
>>> append(1)
[1]

But it's still wrong, in a subtle way:

>>> items = []
>>> append(1, items)
[1] # seems good
>>> items
[]  # or not?

If we pass in an empty list, we want the item to be added to it, not to some other empty list. The problem is, if not items doesn't distinguish between None and empty lists, both of which are false. So, when we use None as a default argument to indicate absence, it's best to be explicit — if items is None makes sure that the absence of arguments is not confused with arguments that just happen to be false:

>>> def append(item, items=None):
...     if items is None:
...         items = []
...     items.append(item)
...     return items
>>> items = []
>>> append(1, items)
[1]
>>> items
[1]

Immutable Default Arguments

What about immutable default arguments — values that can't change, like strings, numbers or booleans? If we say "hello, stranger" with a default value:

>>> def hello(name='stranger'):
...     print('hello', name)

Is it in any way different than doing so through absence?

>>> def hello(name=None):
...     if name is None:
...         name = 'stranger'
...     print('hello', name)

Well, generally speaking, immutable default arguments are safe to use; at the very least, they're not going to mutate and introduce state into our functions.

But they do introduce coupling, which I'd like to illustrate by way of a story. Suppose we have a function, write, that writes some data to a path, and supports compression — if its compress parameter is True, that data is written using GzipFile. We set compress to False by default:

>>> import gzip
>>> def write(path, data, compress=False):
...     f = gzip.GzipFile(path, 'w') if compress else open(path, 'wb')
...     f.write(data)
...     f.close()

So we can use the function for plain writing:

>>> path = '/tmp/hello.txt'
>>> write(path, b'hello world')
>>> print(open(path).read())
hello world

And whenever we want to apply compression, we just pass in an additional True:

>>> write(path, b'hello world', True)
>>> print(open(path).read())
-‡¢fÿhello.txtËHÍÉÉW(Ï/ÊI…J           # data is garbled up
>>> print(gzip.GzipFile(path).read()) # until decompressed
b'hello world'

So far, so good; however, before we continue, take a look at this function again:

>>> write(path, b'hello world', True)

It's a pretty opaque invocation, isn't it? It would be pretty hard to tell what the True stands for if we weren't able to recall the exact signature of the function, which is usually the case — so let's take a quick detour to address this.

Keyword Arguments

When we call a function, we can pass it arguments by position, in which case they are matched to parameters in that order — path, then data, then compress. But we can also bind arguments to parameters by keyword, specifying the parameter name explicitly:

>>> write(path=path, data=b'hello world', compress=True)

When we pass arguments like that, their order doesn't matter — which is especially useful for functions that have many parameters, or whose parameter order is not obvious. You'll often see such invocations spread over multiple lines, with one parameter name, or keyword, per line:

>>> write(
...     path = path,
...     data = b'hello world'
...     compress = True,
... )

And it's also possible to mix the two invocation styles, avoiding obvious redundancies like path=path and getting the best of both worlds:

>>> write(path, b'hello world', compress=True)

The only caveat is that all the positional arguments have to come first:

>>> write(path=path, b'hello world', compress=True)
SyntaxError: positional argument follows keyword argument

Admittedly, it's a superficial restriction — it wouldn't be impossible to match the keyword arguments first, and distribute the positional arguments among the remaining parameters later. But doing so would make it needlessly difficult to figure out what goes where in an invocation, at least at a glance; and since it's such an essential and common occurance, it's a pretty reasonable convention.

That fact it's enforced on the syntax level is not so much an attempt to "protect us from ourselves", as a concession that makes the language simpler to implement. Notice how this is a very different kind of consideration than the ones we've discussed so far. It doesn't ask what would be better or worse for the user — but, assuming something wouldn't matter much to the user, remembers that there are other people involved; so, if some design decision would make the lives of the core developers who maintain the language easier, it might be very well worth it.

In any case, invocations by position and keyword behave almost exactly the same: it's mostly a matter of readability, of how concise or comprehensive we want our function call to look. When we're passing in opaque values, like booleans, it's usually a good idea to pass them by keyword, as it makes our code much more intelligible. The exception, then, is exactly those cases that are intelligible enough on their own, like set_enabled(True) — which is clear enough, and arguablly cleaner than set_enabled(enabled=True).

Parameter Coupling

Getting back to immutable default arguments, our write function has one, and it seems to be doing just fine. Right?

But suppose that there's another function, write_json, which similarly takes a path and some data, and writes it there in JSON format. It does so by converting the data to a JSON string, and delegating the actual writing to write.

But write also supports compression; so should we compress JSON data always, or never? I don't know — so to defer this decision, let's add a pass-through compress parameter with the same default argument to json_write. We end up with:

>>> import json
>>> def write_json(path, data, compress=False):
...     write(path, json.dumps(data).encode(), compress)

And it works — we can write a dictionary and read it back, either in plain or compressed form:

>>> write_json(path, {'x': 1})
>>> print(open(path).read())
{"x": 1}

>>> write_json(path, {'x': 1}, compress=True)
>>> print(open(path).read())
ž‡¢fÿhello.txt«VªP²R0¬ÄÝx             # data is garbled up
>>> print(gzip.GzipFile(path).read().decode()) # until decompressed
b'{"x": 1}'

The problem is, we have casually coupled the two interfaces; this example is actually based on a true story, so let me tell you what happened next. We were running out of storage space, so we decided to turn compression on by default:

>>> def write(path, data, compress=True):
...     # same as before

And this seemed to help — until we started running into weird bugs, where some files were indeed compressed, but others were not.

When I tell it like this, it's easier to spot the bug; but in a codebase with thousands of lines of code, it wasn't easy at all. When we changed compress to True, we did so in the write function only, assuming all other functions that delegate to it will work accordingly; except, of course they didn't. write_json still has its own compress, which still defaults to False:

>>> def write_json(path, data, compress=False):
...     write(path, json.dumps(data).encode(), compress)

So this is what it's passing to write — explicitly, mind you, so JSON files remained uncompressed by default. When we changed compress to True, we needed to update it here as well — and in any other function that similarly delegates works, either to write or to write_json.

Testing for Absence Still

That's what I mean by coupling interfaces: introducing an implicit interdependency between them, an arrangement that's fragile and error-prone to maintain.

What if instead, we rely on absence, and set any default arguments to None, regardless of what default value we want to use? If it's still None when we need it, we resolve it then, at the last possible moment:

>>> def write(path, data, compress=None):
...     if compress is None:
...         compress = True
...     # same as before

>>> def write_json(path, data, compress=None):
...     write(path, json.dumps(data).encode(), compress)

This way, our interfaces don't become coupled over default values; None is used everywhere to indicate absence, and it's only None that is passed through. We can even extract the default value out of the function, like this:

>>> default_compress = True
>>> def write_json(path, data, compress=None):
...     if compress is None:
...         compress = default_compress
...     # same as before

So it's not hardcoded in the function's body, and can be configured dynamically from outside.

TODO@image: coupled-arguments

Just one more thing before we move on: please don't think of this as "tips and tricks for writing better functions". The point I'm trying to drive is that functions are the most fundamental interfaces in our code, and we should take special care when designing them; the important question here is not "how do we add default arguments", but "why do we add default arguments to begin with". The How should come naturally from understanding their place in the interface, not from seeing a couple of bugs and saying "OK, so we shouldn't do that".

To recap this Why: a default argument indicates an optional parameter, which means our code is somehow configurable — so whoever uses it might come to rely on its default configuration. This effectively commits it to the function's interface, to the standard by which the function interacts with the rest of the codebase.

Publishing a mutable object as part of it is almost never a good idea, because the promise of an interface — a well-defined and predictable communication channel — suddenly becomes muddled by the state of a single, long-lived object.

And while publishing an immutable object, like True or False, is generally safe, it still makes it part of the interface — no longer an implementation detail, but something to commit to; something that if changed, affects not only the default behavior, but also the public contract, the function's relations with the rest of the code.

At this point, if we conclude that an optional parameter indicates that the absence of an argument is permissible, we can just use None to indicate this absence. This is essentially the bare minimum we can commit it — so it's the easiest promise to keep, and usually the best way to go.

The Variadic Positional Parameter

Moving on; Suppose we have a function to calculate an average of two numbers:

>>> def average(x, y):
...     return (x + y) / 2
>>> average(1, 2)
1.5

In some cases, it makes sense to calculate the average of three numbers, right?

>>> def average_3(x, y, z):
...     return (x + y + z) / 3
>>> average_3(1, 2, 3)
2.0

And to satisfy the general case, we'd have to make a function that receives a list of numbers:

>>> def average_n(xs):
...     return sum(xs) / len(xs)
>>> average_n([1, 2, 3, 4, 5])
3.0

It's a bit clunky, what with the extra brackets and all — but more importantly, it doesn't reflect our interface as well as it could. Instead of a function that takes "any number of arguments", we cheated by passing any number of arguments through a single list.

To get the actual "any number of" behavior, called variadic, we need to add the star character, or asterisk, right before the variadic parameter's name:

>>> def average(*xs):
...     return sum(xs) / len(xs)

This way, we can invoke our function with two, three, or any other number of arguments, and they will all be grouped into one tuple, assigned to xs:

>>> average(1, 2)
1.5
>>> average(1, 2, 3)
2.0
>>> average(1, 2, 3, 4, 5)
3.0

More generally, if we add some other parameters, the starred one will capture "the rest of" the arguments — or at least, the rest of the arguments passed by position, which is why it's called the variadic positional parameter:

>>> def f(x, y, *args):
...     print(x, y, args)
>>> f(1, 2)
1 2 ()
>>> f(1, 2, 3)
1 2 (3,)
>>> f(1, 2, 3, 4, 5)
1 2 (3, 4, 5)

This is very similar to star capturing, where we used the same operator to capture "the rest of" the values being assigned. Like assignment, we can only have one such parameter, otherwise it would be ambiguous which arguments go where; but unlike assignment, where we can put it at the beginning or in the middle, the variadic positional parameter must come after all the other positionl parameters — once again, not because it's impossible to support, but to keep function invocations simpler to understand and to maintain.

The Variadic Keyword Parameter

What about a variadic keyword parameter? Seeing as we can pass arguments by keyword, where we explicitly match them to a parameter name, wouldn't it be useful to capture "the rest of" these keyword arguments, the ones that don't match any particular parameter?

Consider this function, which creates an XML element — and don't worry if you don't know XML, it's just a particular format to write data in:

>>> def create_element(tag, text=None, attributes=None):
...     output = ['<', tag]
...     if attributes:
...         for key, value in attributes.items():
...             output.extend([' ', key, '="', value, '"'])
...     if text:
...         output.extend(['>', text, '</', tag, '>'])
...     else:
...         output.append(' />')
...     return ''.join(output)

An element starts with an angle bracket, followed by a tag; then, it can have zero or more key-value pairs, called attributes; and finally, it can nest some content — in this case, text — which is placed between it and a corresponding closing tag; or, it can have no content, in which case it's closed with a slash.

For example, we can put a string in a p tag, which stands for a paragraph:

>>> create_element('p', 'hello world')
'<p>hello world</p>'

Or we can have an image, which has no content, but uses attributes to define its source file and some alternative text in case it can't be loaded:

>>> create_element('img', None, {'src': 'cat.jpg', 'alt': 'a cat'})
'<img src="cat.jpg" alt="a cat" />'

Here, we're passing an awkward None just to get to the attributes parameter, but we can also access it directly by keyword:

>>> create_element('img', attributes={'src': 'cat.jpg', 'alt': 'a cat'})
'<img src="cat.jpg" alt="a cat" />'

It's still pretty clunky; not to mention that once again, it doesn't reflect that our interface can take "any keywords" to make XML attributes out of, we just sneak them in through a dictionary.

Instead, let's add two stars to our last parameter — this will make it capture "the rest of" the arguments passed by keyword, and expose those values, as well as their names, in a dictionary:

>>> def create_element(tag, text=None, attributes=None):
...     output = ['<', tag]
...     for key, value in attributes.items():
...         output.extend([' ', key, '="', value, '"'])
...     if text:
...         output.extend(['>', text, '</', tag, '>'])
...     else:
...         output.append(' />')
...     return ''.join(output)

The rest of the code remains the same (except that we can get rid of the if attributes check, since now, when no attributes are specified, it would result in an empty dictionary that can still be iterated over, unlike None). This works just like we wanted, with syntax that's about as clean and elegant as we could hope for:

>>> create_element('img', src='cat.jpg', alt='a cat')
'<img src="cat.jpg" alt="a cat" />'

Looking at the general case:

>>> def f(a, b, **kwargs):
...     print(a, b, kwargs)
>>> f(1, 2)
1 2 {}
>>> f(1, 2, c=3)
1 2 {'c': 3}
>>> f(1, 2, c=3, d=4, e=5)
1 2 {'c': 3, 'd': 4, 'e': 5}

We should note that only arbitrary or extra keywords are captured — keywords that match a parameter name are obviously assigned to it:

>>> f(a=1, b=2, c=3, d=4, e=5)
1 2 {'c': 3, 'd': 4, 'e': 5}

The Omnisignature

Putting together the two variadic parameters, we get something we might call the omnisignature:

>>> def f(*args, **kwargs):
...     print(args, kwargs)

The positional parameter is often called args, and the keyword parameter is either kwargs or kwds, but of course any name goes. The point is that this function has the most flexible signature imaginable — it can be called on any combination of arguments, passed by position or keyword or both:

>>> f()
() {}
>>> f(1, 2)
(1, 2) {}
>>> f(x=1, y=2)
() {'x': 1, 'y': 2}
>>> f(1, 2, x=1, y=2)
(1, 2) {'x': 1, 'y': 2}

It might seem like this defeats the purpose of having an interface — if there's no shape to the input, and we just pass in whatever, it's usually not a very good abstraction — but we'll see why this can be useful later.

TODO@image: omnisignature.svg

Exploding Arguments

For the moment, let's get back to the star operator. If we have a list of numbers that we want to average, passing it to our previous implementation, which expected a list, would be trivial:

>>> numbers = [1, 2, 3, 4, 5]
>>> average_n(numbers)
3.0

But with our newer, and supposedly superior version, we have to unpack everything into a ridiculous mess:

>>> average(numbers[0], numbers[1], numbers[2], numbers[3], numbers[4])
3.0

Luckily, much like with assignment, the star operators plays two different roles. On one "side" of a function call, in the signature, it helps capture "the rest of" the arguments; while on the other "side", in an invocation, it explodes them — spreading and flattening an iterable into separate positional arguments:

>>> average(*numbers) # numbers[0], numbers[1], ..., numbers[n]
3.0

Here, we can use it anywhere, multiple times, on any iterable:

>>> average(*[1, 2], 3, *[], 4, 5, *(6, 7, 8), *{9})
5.0

In fact, this syntax was even expanded to support lists and sets:

>>> [*[1, 2], 3, *[], 4, 5, *(6, 7, 8), *{9}]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> {*[1, 2], 3, *[], 4, 5, *(6, 7, 8), *{9}}
{1, 2, 3, 4, 5, 6, 7, 8, 9}

What about our keyword scenario? If we have a dictionary of attributes, we face a similar problem, where our supposedly elegant implementation requires some pretty tedious and redundant mapping:

>>> attributes = {'src': 'cat.jpg', 'alt': 'a cat'}
>>> create_element('img', src=attributes['src'], alt=attributes['alt'])
'<img src="cat.jpg" alt="a cat" />'

And just like we could capture a dictionary using two stars in the signature, we can use two stars in an invocation to explode it, as if all its key-value pairs are being passed in by explicit keyword arguments:

>>> create_element('img', **attributes) # <key>=<value>, <key>=<value>, ...
'<img src="cat.jpg" alt="a cat" />'

This, too, we can use anywhere and as many times as we like:

>>> create_element('img', **{'src': 'cat.jpg'}, alt='a cat',
                    **{}, **{'title': 'A cat!'})
'<img src="cat.jpg" alt="a cat" title="A cat!" />'

We can even use it in a dictionary definition, where it similarly unpacks as if the keys and values are flatly spelled out:

>>> {**{'a': 1, 'b': 2}, 'c': 3, **{}, 'd': 4, **{'e': 5}}
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}

As an aside, this is the quickest and simplest way to merge two dictionaries:

>>> {**{'x': 1}, **{'y': 2}}
{'x': 1, 'y': 2}

And if there are any duplicates, the later values overwrite the previous ones, so order does matter:

>>> d1 = {'x': 1, 'y': 2}
>>> d2 = {'y': 3, 'z': 4}
>>> {**d1, **d2}
{'x': 1, 'y': 3, 'z': 4}
>>> {**d2, **d1}
{'z': 4, 'x': 1, 'y': 2}

Perfect Forwarding

Putting this together with the omnisignature from before, we can achieve what's known as perfect forwarding: having one function invoke another in exactly the same way as it was invoked itself. For example, apply takes another function, as well as any number and style of arguments, and calls that function on them:

>>> def apply(f, *args, **kwargs):
...     return f(*args, **kwargs)

>>> def add(x, y):
...     return x + y

>>> apply(add, 1, 2)
3
>>> apply(add, x=1, y=2)
3

And in a more general case:

>>> def f(*args, **kwargs):
...     print(args, kwargs)

>>> apply(f, 1, 2)
(1, 2) {}
>>> apply(f, x=1, y=2)
() {'x': 1, 'y': 2}
>>> apply(f, 1, 2, x=1, y=2)
(1, 2) {'x': 1, 'y': 2}

It might look a bit odd — like, why would we do that, instead of just invoking the function directly? But suppose apply would register the function and its arguments to be executed later, or in some different context; or maybe do something interesting before or after the call, which we'll encounter when we talk about decorators.

In such cases, the omnisignature, combined with perfect forwarding, lets us have interfaces capable of taking any input in, without judgement or limitation, and passing it through, delegating to some other interface in exactly the same way as they were invoked.

Once again, just appreciate how clean and tight the syntax is, especially considering how much power and flexibility it packs.

Keyword-Only Parameters

But wait, there's more — some parameters can be keyword-only.

Suppose we have a function like print, which takes any number of arguments and prints them:

>>> print('hello')
hello
>>> print('hello', 'world')
hello world

If we were to implement it ourselves, this might be its signature, right?

>>> def print_(*args):
...     # do actual printing

And I've added an underscore to the name, just to avoid overwriting the builtin version. To do the actual printing, we can write directly to the standard output, available in the sys module as the file-like stdout:

>>> def print_(*args):
...     sys.stdout.write(' '.join(args) + '\n')
...     sys.stdout.flush() # make sure the change registers
>>> print_('hello')
hello
>>> print_('hello', 'world')
hello world

But it turns out that the original print is also configurable — we can change the word separator, which defaults to space, as well as the line ending, which defaults to a newline:

>>> print('Hello', 'world', sep=', ', end='!\n')
Hello, world!

But... how would that fit in a signature? We saw that the variadic positional parameter has to come after all the other positional parameters — but if we put sep and end before args, we end up with an incredibly clumsy interface:

>>> def print_(sep=None, end=None, *args):
...     if sep is None:
...         sep = ' '
...     if end is None:
...         end = '\n'
...     sys.stdout.write(sep.join(args) + end)
...     sys.stdout.flush()

Which is strange enough when we want to configure everything:

>>> print_(', ', '!\n', 'Hello', 'world')
Hello, world!

But otherwise is completely intolerable:

>>> print_(None, None, 'hello', 'world')
hello world

So instead, we put sep and end after args — and this creates an interesting dynamic. args captures any number of positional arguments, so there's effectively no way to "pass through" it and get to either sep or end by position. In other words, they become reachable by keyword only:

>>> def print_(*args, sep=' ', end='\n'):
...     sys.stdout.write(sep.join(args) + end)
...     sys.stdout.flush()
>>> print_('hello', 'world')
hello world
>>> print_('Hello', 'world', sep=', ', end='!\n')
Hello, world!

Enforcing Keyword Arguments

This can also be used to enforce clearer function invocations. Remember how calling write with a positional True was quite opaque?

>>> write(path, b'hello world', True)

Remember what it stands for? If we would pass it by keyword, we'd see that it actually maps to compress. Turns out we can disallow such opaque invocations altogether, even if our function doesn't have a variadic positional parameter, by adding a single star to its signature, making any parameters that come after it into keyword-only:

>>> def write(path, data, *, compress=None):
...     # same as before

Now, passing this boolean by position is simply impossible:

>>> write(path, b'hello world', True)
TypeError: write() takes 2 positional arguments but 3 were given

So we have to do it explicitly, by keyword:

>>> write(path, b'hello world', compress=True)

As an aside, keyword-only parameters are almost always optional, like sep, end and compress. But they don't have to be — we can, in principle, make compress mandatory, which would require every invocation to pass it, and to do so by name:

>>> def write(path, data, *, compress):
...     # same as before

>>> write(path, b'hello world')
TypeError: write() missing 1 required keyword-only argument: 'compress'

It might seem like a strange interface, but only because in our case, compress is so inherently optional. If it really was mandatory, and seeing as it's an opaque value, it would be an equally good idea to require its use by keyword only. In other words, being keyword-only is not about whether a parameter is optional or mandatory — it's about whether its value is representative of its essence, so the corresponding parameter can be inferred, or not.

Position-Only Parameters

Similarly, we have position-only parameters. Those always come first, and are separated from the rest with a slash:

>>> def add(x, /, y):
...     return x + y

In this example, we can call add with two positional parameters, or pass the second one by keyword:

>>> add(1, 2)
3
>>> add(1, y=2)
3

However, the point is that there's no way to pass the first one by keyword:

>>> add(x=1, y=2)
TypeError: add() got some positional-only arguments passed as keyword arguments: 'x'

The x parameter is effectively nameless, so it can be assigned by position only. Now, this signature is a bit odd on purpose, to clearly illustrate the difference; but we might actually see similar functions with signatures like this:

>>> def add(x, y, /):
...     return x + y

This indicates that none of the parameters should ever be called by keyword, at all. It makes sense — x and y are pretty arbitrary names, so we wouldn't want them mentioned explicitly as part of our interface. It's like the reverse of keyword-only parameters, which require mentioning a significant name — here, we prevent anyone from mentioning an insignificant one.

And while they're nice for readability, position-only parameters have a deeper purpose. Remember our create_element function from before? Suppose we wanted to create an element with the attribute text set to hello. Doing so would actually match it to the text parameter, and end up as nested content, not an attribute:

>>> create_element('element', text='hello') # <element text="hello" />
'<element>hello</element>'

Even worse, suppose we want a tag attribute; in this case it would simply crash, since the tag argument was already passed in:

>>> create_element('element', tag='greeting') # <element tag="greeting" />
TypeError: create_element() got multiple values for argument 'tag'

Unless, that is, we make the first two arguments position-only:

>>> def create_element(tag, text=None, /, **attributes):
...     # same as before

This way, they are nameless in our interface, so any name — including text and tag — can be scooped up by attributes. Our recent examples now work:

>>> create_element('element', text='hello')
'<element text="hello" />'
>>> create_element('element', tag='greeting') 
'<element tag="greeting" />'

And our previous use cases, like passing in some actual nested content, are still possible by position. If anything, this makes them less ambiguous, with the first argument always being the tag, and the second, if present, always being the content — no names, only positions.

Annotations

A few final things to mention about functions as interfaces, before we talk about them as objects, is docstrings and annotations.

As far as Python syntax goes, if a function begins with a single anonymous string (meaning it's just an expression, not assigned to any variable), then this text is retained as the function's documentation, and made available through the __doc__ attribute:

>>> def f():
...     'some string'
...     return 1
>>> f.__doc__
'some string'

This is usually a multiline string, so you'll often see it spaced out between triple quotes:

>>> def f():
...     """
...     some string.
...     """
...     return 1

It's intended to be the function's documentation:

>>> def add(x, y):
...     """
...     Returns the sum of x and y.
...     """
...     return x + y

But there's nothing preventing us from using it for something else. For example, the module of the standard library suggests adding interactive test cases to your docstrings:

>>> def div(x, y):
...     """
...     >>> div(5, 2)
...     2.5
...     >>> div(4, 2)
...     2
...     """
...     return x / y

Which it can then evaluate and run as actual tests:

>>> import doctest
>>> doctest.testmod()
Failed example:
div(4, 2)
Expected:
2
Got:
2.0

So we can actually validate our own understanding of what the function is supposed to do; not to mention that simple tests make for excellent documentation.

Similarly, a function can have annotations. Every parameter be associated with an extra value, after its name and before its default argument, if any; and the function itself can be associated with an extra value, placed after its parameter list:

>>> def f(x: 1, y: 2 = None) -> 3:
...     pass

This was added in Python 3.5, and intended primarily for type checking. Now, each parameter can indicate its expected argument type, and the last one, called the return annotation, can describe the type of the function's return value:

>>> def add(x: int, y: int) -> int:
...     return x + y

To be clear, this isn't enforced:

>>> add('hello ', 'world')
'hello world'

But you can use a 3rd party library, like mypy, to validate scripts that do use annotations this way; so if we'd have example.py like so:

def add(x: int, y: int) -> int:
    return x + y

add('hello', ' world')

We could run python -m mypy on it, and it would alert us to the discrepancy:

/home/user/$ python -m mypy example.py
/home/user/example.py:4: error: Argument 1 to "add" has incompatible type "str"; expected "int"
/home/user/example.py:4: error: Argument 2 to "add" has incompatible type "str"; expected "int"

However, that's not inherent to annotation — it's just one scenario built on top of them, and they can be anything else:

>>> def f(x: 'hello', y: [1, 2, 3]) -> ValueError:
...     pass

It's up to whoever's interested in them to interpret their meaning, as they are made available through the function's __annotations__ attribute:

>>> f.__annotations__
{'x': 'hello', 'y': [1, 2, 3], 'return': <class 'ValueError'>}

We'll cover how annotations are used for type checking in a later chapter, when we talk about type analysis; for now, I'd just like to underline this wonderful design decision, of docstrings and annotations both.

There aren't many languages that are as inviting as Python in letting us get under the hood, and this whole mindset is just so empowering. Instead of saying "we know what you need", they go for a much softer version: "we suspect what we need, but we don't know what we don't know".

So the language actually has syntax the does nothing; it's there completely and solely for us to extend. It was added for type checking, yes, but in such a respectful and humble way — decoupling the language, which is for everyone, from the particular paradigm that its developers were interested in. This paradigm was then implemented as a particular use case on top of this syntax; equally valid, but just one use case among many.

This allows, even encourages, a certain creativity, diversity and experimentation in the community. It lets us equip our interfaces with all this additional information, free of any commitments or side-effects, and use it in whatever way makes sense in our context — and there's something deeply humanistic about this attitude.

Functions 1: Conclusion

To sum up, we talked about default arguments and optional parameters, as well as variadic, keyword-only and positional-only. We also mentioned docstrings and annotations; but what I really tried to get across in this chapter is that functions are the foundational unit in software design, and their interfaces essentially define a codebase. In this regard, Python does an astounding job — function signatures pack a crazy amount of possibilities with syntax that is so clean that you can eat from it; and this, in turn, allows the interactions in a Python program to be both rich and precise.

So far, though, we've only talked about the function's interface, and haven't really addressed where this interface leads to — so next, we'll look into the function object, what it's made of and how it can be used.

TODO@image: signature.svg

Functions 1: Further Reading

TODO