Functions 2: Functional Programming

In the last chapter, we focused on the function's role as an interface, and discussed the nuances of its signature.

What's left is to understand is what a function actually is; but before we do that, I think it's worth taking a moment to reflect on what it does, or allows to do. We'll delve into its code object and bytecode soon enough; but all these minute details will make little sense before we understand the potential we're dealing with.

We'll consider functions as arguments, and see what this implies, from procedural subroutines to transformative functions; we'll talk about functions as return values, from factories to currying and modular composition; and we'll discover how it can be used to manage state and effectively implement an object-oriented paradigm, if it wasn't itself already implemented in this paradigm to begin with.

You see, in older languages, a function might be a simple label to the memory address of some code. When we call it, we simply jump there, execute the code, then jump back (that is, return) to our original flow.

In Python, however, it's clearly more than that — for starters, because it's an actual object, which can be printed and passed around:

>>> def f():
...     return 1
>>> f
<function f() at 0x...>
>>> g = f
>>> g()
1

This idiom, of treating functions as "first-class citizens" — native entities, not unlike numbers and strings — unlocks a lot of interesting possibilities, which would otherwise be considered advanced design patterns.

Functions as Arguments

For example, it means we can pass functions into other functions, as arguments; here's one that takes a function and calls it twice:

>>> def run_twice(f):
...     f()
...     f()

>>> def hello():
...     print('hello world')

>>> run_twice(hello)
hello world
hello world

A more practical version would retry the function again only if it fails; and here we can show off all the fancy syntax we learned last time:

>>> default_max_attempts = 5
>>> def retry(f, *args, max_attempts=None, **kwargs):
...     if max_attempts is None:
...         max_attempts = default_max_attempts
...     for _ in range(max_attempts):
...         try:
...             return f(*args, **kwargs)
...         except Exception:
...             pass

We accept a function and any other arguments, except the reserved max_attempts keyword; and then we pass them on with near-perfect forwarding, until the function either succeeds or fails too many times. This is specified by the optional parameter max_attempts, which defaults to a configurable value, decoupled from the interface.

But handy though it may be, it doesn't really showcase the power of functions as arguments.

The key insight here is that functions are not just some piece of code that can run once, twice, or until it succeeds. They actually encapsulate behaviors — even behaviors that can be parametrized and contextualized. By passing functions one into the other, and by coordinating their interfaces a little more closely, we can compose their different behaviors dynamically.

Injecting Behavior

To illustrate this, let's sort a list. In Python, it's easy enough: we call the list's sort method. But the overall process of sorting presupposes some logic of comparison — looking at two elements and deciding which one of them comes first. By default, sort knows how to compare numbers:

>>> numbers = [2, 3, 1]
>>> numbers.sort()
>>> numbers 
[1, 2, 3]

And it arranges strings alphabetically:

>>> names = ['charlie', 'alice', 'bob']
>>> names.sort()
>>> names
['alice', 'bob', 'charlie']

Which is a good default behavior — but what if we want to sort our items differently, like ordering strings by their length or numeric value?

>>> numbers = ['5', '100', '23']
>>> numbers.sort()
>>> numbers
['100', '23', '5'] # alphabetical order

Since we can pass functions as arguments, sort is actually written with that in mind. It accepts, as an optional argument, another function, then uses it to perform the comparisons — effectively letting us integrate custom logic into the broader sorting algorithm, without worrying about its details or implementation.

The only requirement is that our function accepts two items, a and b, and returns a positive number if a is larger, a negative number if it's smaller, and 0 if the items are equal. That's what I referred to as interface coordination: the assumptions sort makes when about foreign code when it injects it into its operation.

To order strings by size, then, we can have a function that subtracts their lengths:

>>> def by_length(a, b):
...     return len(a) - len(b)

And here it is in action:

>>> names.sort(by_length)
>>> names
['bob', 'alice', 'charlie']

In fact, the same can be done inline, using a lambda:

>>> names.sort(lambda a, b: len(a) - len(b))
>>> names
['bob', 'alice', 'charlie']

Similarly, to compare strings by numeric value, we can cast them into integers and subtract those:

>>> def numerically(a, b):
...     return int(a) - int(b)

>>> numbers.sort(numerically)
>>> numbers
['5', '23', '100']

>>> numbers.sort(lambda a, b: int(a) - int(b))
>>> numbers
['5', '23', '100']

And here, I'd like to make a subtle point.

Functional v. Procedural

I sometimes hear that because functions are objects that we can be passed around, it makes Python a functional language, as opposed to procedural languages like Java or C.

When you look it up, functional programming is defined as a paradigm where a program is constructed by "applying and composing functions", whereas in procedural programming, a program is "a series of steps that operate on a state".

TODO@image: procedures.svg

Personally, I find this confusing in both directions. First, if you look at Python code, it is very clearly a series of steps, with scope being the state; and second, if you look at Java or C, it's actually possible to pass around functions in both. Injecting custom behavior, like we did with sort, is just so incredibly useful, that people always seem to figure out a way to do it — in Java they wrap a function in a class, called a functor, and in C they pass the function's memory address, called a function pointer.

So to better understand the difference, let's go back to our example. While it's true that we're passing a function here, I would argue that our way of thinking is still procedural. We separate the broader sorting procedure from its comparison subprocedure, and swap it with custom logic; but we are still thinking in terms of steps — given a and b, what do we do to determine which one is larger?

Turns out, there's another approach. We can pass in a key function, which reduces every element in the list to a simpler value, and these values are then sorted with the default behavior. Passing in the len function will reduce the strings to their lengths, so that's what they'll be sorted by:

>>> names.sort(key=len)
>>> names
['bob', 'alice', 'charlie']

And passing in int will cast any strings into integers, so that's how they'll be compared:

>>> numbers.sort(key=int)
>>> numbers
['5', '23', '100']

Nevermind the programming language; this way of thinking is what I would call functional. It doesn't describe a series of changes to do, but declares what changes are to be done. Like I said, it's a subtle distinction — but an important one, particularly if we can distill its essence and apply it more broadly to software design.

Here, I think the terminology is somewhat to blame for the confusion: functional programming sounds like it has something to do with functions. On the surface of it, of course it does — it's all about defining, composing and applying functions, right?

The problem is, so is procedural programming. This one, we don't take so literaly; we don't expect it to be written in procedures, do we? And even if we did — how do you define a procedure in Python? Isn't it just a function as well?

To wit, both by_length and numerically are functions; and deep down, len and int also contain a series of steps, a "procedure", by which to change their input into output. So what gives? What exactly is the difference?

Transformation and Cooridnation

Let's try to refine this terminology, then. Think of the word "functional" as "instrumental" — an instrument, or tool, built to perform a particular task; to affect a particular transformation. The function of a knife, in this context, would be to cut, transforming a chunk into slices; the function of an oven would be to bake, transforming dough into bread.

Notice how len and int fit this description: they transform an iterable into its length, or some value into its corresponding number. by_length or numerically, in comparison, don't make a lot of sense on their own — it's not often that given two iterables or numerical values, we want to "get a positive number if the first one is larger, a negative one if it's smaller, or zero if they're equal". This is something that mostly makes sense in relation to sort, as a way to communicate our custom logic so that it can be reliably incorporated. A procedure, then, is not a tool, but a protocol; it's not about transformation — it's an agreement, a routine, a coordination. Fermenting pickles or making dough, to continue our previous analogy, would be examples of this method.

Generally, cooking strikes me as a good comparison, precisely because its tools and protocols are somewhat intertwined, just like functional and procedural programming are. Sometimes we transform food, like cut it or heat it up, and sometimes we combine different ingredients and carefully coordinate their ratios. Of course, it's entirely possible to use only one method or the other — microwave a precooked meal, or mix a cocktail — but most substantial recipes will require us to do both; and that's OK. It's not so important to "correctly" label each practice or each snippet of code, nor argue over the ambiguous cases that might fit either or neither. What we want to learn is, given some problem, how to conceive of solutions that might transform it, coordinate its part or both.

For instance, we can implement the len function like this:

>>> def proc_length(xs):
...     i = 0
...     for _ in xs:
...         i += 1
...     return i

And I'd say that this implementation is procedural, because it devises a solution by way of coordinating between the counter i, the list xs, and the loop that increments it. Naturally, this attitude of "who needs to do what" leads to a description of changes, the so-called "series of steps that operate on a state".

Incidentally, we can then use this procedural creature in an otherwise functional scenario:

>>> names.sort(key=proc_length)
>>> names
['bob', 'alice', 'charlie']

Or, we can do the opposite. Suppose we have a different implementation:

>>> def func_length(xs):
...     return 0 if not xs else 1 + func_length(xs[1:])

This one devises a solution through a transformation, in this case utilizing recursion. Empty lists are replaced with 0, and non-empty lists are reduced to 1 plus the length of the list's tail.

Naturally, the attitude of "who becomes what" leads to a declaration of changes.

Such a declaration, however, can only be expressed through the "application and composition of functions". Here, the recursive function calls itself — although on an increasingly simple case — until it hopefully converges, and the overall change coheres. So yeah, functions are definitely prominent in functional code; but like I said, I wouldn't think about it in these terms. According to this logic, "procedures" should be prominent in procedural code, and we don't even have such a construct.

And before anyone says that recursion is an inherently functional method, please consider that plenty of procedural code utilizes recursion, too. In fact, in the previous paragraph, rather than saying that "a recursive function calls itself", I would say that it "expresses its result in terms of itself, called on a simpler case". The emphasis is not on how functions are used, but on how problems are solved — in case of recursion, either with gradual coordination, or through compounding transformations.

Anyway, we can use our functional code in a procedural context:

>>> names.sort(lambda a, b: func_length(a) - func_length(b))
>>> names
['bob', 'alice', 'charlie']

So one last time: functional and procedural code is not about the challenge we solve, or the language constructs that we use to solve it — it's about how we concieve of the solution. One way is to coordinate a procedure, in which different values are stored in a state and interact with each other, step by step, to arrive at the result. Another way is to define a transformation, in which the concepts are composed and applied in such a way that simply resolving the expression would yield the desired outcome.

Functional v. Procedural: Looping

We actually encountered this way of thinking before, when we were talking about loops. In classical for-loops, we saw that the loop structure itself was only concerned with the index; to retrieve the relevant item in each iteration, we had to coordinate with this information, e.g. to print the sequence in reverse:

>>> for i in range(len(colors)): # i = 0; i < len(colors); i += 1
...     print(colors[len(colors) - i - 1])

For-each loops, in comparison, are meant to iterate directly over the sequence. In some sense, it's more limiting than a classic for-loop — we have no say on what the index is initialized with, how it's modified, or when it terminates; it just goes over the sequence, item by item.

But this inability to coordinate shifts our attention to what we can control — namely, the sequence itself, which can transform into a sequence that would adhere to the default behavior:

>>> for color in reversed(colors):
...     print(color)

Just like the key function transformed all the items into values that could be sorted with the standard algorithm, rather than letting us tap into it and coordinate custom comparisons from within.

We also saw this way of thinking with comprehensions, where a procedural code block that calculates some values over some range:

>>> squares = []
>>> for i in range(5):
...     squares.append(i**2)
>>> squares
[0, 1, 4, 9, 16]

Can be expressed as a transformation of that range in its entirety:

>>> [i**2 for i in range(5)]
[0, 1, 4, 9, 16]

Functional v. Procedural: Sorting

As a final example, consider how complex objects had to be sorted in older versions of Python, before we had the key keyword. Suppose we have a list of objects with dates and priorities:

>>> events = [
...     Event(priority=1, date=datetime(2000, 1, 1)),
...     Event(priority=2, date=datetime(2000, 1, 2)),
...     ...
... ]

To compare by priority and break ties with date, we either write a comparison function like before, which ends up quite tedious:

>>> def compare_events(event1, event2):
...     if event1.priority > event2.priority:
...         return +1
...     elif event1.priority < event2.priority:
...         return -1
...     if event1.date < event2.date:
...         return +1
...     elif event1.date > event2.date:
...         return -1
...     return 0

Or we do it quick and dirty, in a pattern called decorate-sort-undecorate, or DSU.

First, we decorate, turning our objects into tuples of whatever we want to sort by, like priority and date, followed by the object itself:

>>> dsu = [(event.priority, event.date, event) for event in events]

Then, we sort it. Since tuples are compared lexicographically, field by field, this means priorities are considered first, with dates used to break ties, just like we wanted. This part of the tuple, where we've effectively reduced our complex object to simpler values, is called the sort key; and while the tuples are being reorganized according to its logic, their last item, the complex object, conveniently tags along.

>>> dsu.sort()

So finally, we undecorate, peeling off the sort key but keeping the objects:

>>> events = [event for priority, date, event in dsu]

Except now they're in the right order, see?

Functional v. Procedural: Verdict

This works well, but it does copy the list twice, which is quite wasteful. Also, it's very procedural in its thinking, combining and recombining the data in separate steps (although, funnily enough, using functional comprehensions to do so elegantly). So in Python 2.4, they added the key parameter we saw before, with which we can declare the sort key directly and separately:

>>> events.sort(key=lambda event: (event.priority, event.date))

And sort knows what to do — essentially, what DSU does: associate each items with its derived key, sort the simple keys in a standard manner, and return the associated items in that order. What's crucial, though, is that sort does it for us; we just tell it how to do it.

In fact, this turned out to be so tremendously useful, that Python 3 removed support for the comparison function altogether; now, sorting can be customized by a sort key only. This brings us back to the question that started this side-quest: is Python a functional language? This change, for example, certainly follows a functional trend.

Hopefully, it should be clear by now that this question doesn't really make sense. Functional and procedural is a way of thinking; and while it's true that different languages can make more or less sense in one paradigm or another, most of them have a healthy mixture of both.

Python definitely does; although I'd say it leans more towards the procedural mindset. Functional concepts, from for-each loops to sort keys, are added where they appear to work best, and in a way that blends well with procedural code blocks.

The main takeaway from this is that one paradigm is not better than the other. Different problems lend themselves to different abstractions, and insisting to do everything in one purist way is not very wise; as we've seen in a previous chapter, when people argue too passionately for or against some pattern, they're invariably wrong. Instead of dividing things into black and white, we can appreciate both, and all the other colors, too — our insights will be all the more vivid for it.

Functions as Return Values

Getting back to functions — as objects, they can become return values as well. Suppose that in our code, we square numbers often, and decide to extract this logic into a dedicated function:

>>> def square(x):
...     return x**2
>>> square(2)
4

This is nice, but if we cube numbers as often, we'd need another one:

>>> def cube(x):
...     return x**3
>>> cube(2)
8

And so on. Instead, we can have a function that takes a number, and returns a new function — which, given a value, raises it to a fixed power:

>>> def create_power(n):
...     def power(x):
...         return x**n
...     return power

This can be used to square, cube, root, you name it:

>>> square = create_power(2)
>>> square(2)
4
>>> cube = create_power(3)
>>> cube(2)
8
>>> root = create_power(0.5)
>>> root(2)
1.414

In fact, we don't even have to bind the returned function to a name; we can use it immediately:

>>> create_power(2)(2)
4
>>> create_power(3)(2)
8
>>> create_power(0.5)(2)
1.414

We can think of create_power as a factory function, a way to construct a behavior dynamically — passing in the information necessary to prepare it, and getting back a callable result that can be applied just so.

TODO@image: function-creation.svg

To be fair, we didn't have to create a function on the fly. Starting with square and cube, then automating their creation through create_power, is just one way to go — we could have just as easily consolidated their behavior in one generic case:

>>> def power(x, n):
...     return x**n

So we have power(2, 3) instead of power(3)(2), which is much clearer.

Currying

Admittedly, this whole example is a bit silly, because raising a number to some power is so exceedingly easy in Python — we can just use x**n. But imagine it was some other, complicated calculation. What's the fundmanetal difference between create_power, which allows the creation of behaviors like square or cube, and power, which merges these behaviors in a generalized form?

Well, we can look at create_power differently — not as a factory that generates new behaviors, but as a gradually constructed function; a behavior in the making. create_power(n)(x) is the same as power(x, n), but it's somehow more modular — we can stop in the middle, and can concieve of create_power(n) on its own, while there's no way to divide power in two.

Of course, if we stop in the middle, create_power(n) on its own wouldn't return an actual number, but a function — some half-way behavior that, given x, will resolve it to the desired result. This ability, to create functions gradually, is well in line with functional programming; it even has a name of its own, currying, whereby any call like f(a, b, c) can be broken into an equivalent composition of f(c)(b)(a). Once again, this practice is functional in nature not because it has to do with functions, but because it helps us compose and apply them as transformations.

To illustrate this point further, suppose that we also need to iterate over a list, squaring its items. The procedural thing to do would be to square them in advance, or on demand:

>>> xs = [1, 2, 3]
>>> for x in xs:
...     print(x**2)
1
4
9

The functional thing to do would be to transform the list:

>>> for x in squared(xs):
...     print(x)
1
4
9

For which we'd have to define (a procedural) squared:

>>> def squared(xs):
...     result = []
...     for x in xs:
...         result.append(x**2)
...     return result

Or maybe a one-liner with a (functional) list comprehension:

>>> def squared(xs):
...     return [x**2 for x in xs]

However, this functionality is inherently limited to squares. If we wanted to iterate over a list cubed, we'd have to write a new function; let's skip this, and generalize it already:

>>> def raised(xs, power):
...     return [x**power for x in xs]

In which case we can support all cases with equal ease:

>>> raised(xs, power=2)
[1, 4, 9]
>>> raised(xs, power=3)
[1, 8, 27]

Like I said, it looks a bit silly with powers, but imagine a complicated calculation instead, yeah? Having already implemented it in square and cube, we'd want to reuse them, not inline them; to implement squared and cubed like so:

>>> def squared(xs):
...     return [square(x) for x in xs]
>>> def cubed(xs):
...     return [cube(x) for x in xs]

In which case, to support the general case in raised, we'd have to rely on the generalized behavior of power:

>>> def raised(xs, n):
...     return [power(x, n) for x in xs]

But here we're using the parametrized, indivisible version of power. Since we're exploring the functional paradigm, how would it look if we commit to this way of thinking and use the gradually constructed, modular create_power instead? Well, the obvious substitution would be:

>>> def raised(xs, n):
...     return [create_power(n)(x) for x in xs]

But here we don't gain anything in particular; just a different way to achieve the same thing. More interesting would be to pass the power function as an argument, instead of its n:

>>> def raised(xs, power):
...     return [power(x) for x in xs]

>>> raised(xs, create_power(2))
[1, 4, 9]
>>> raised(xs, create_power(3))
[1, 8, 27]

The interesting thing about it is that in this case, raised itself ceases to have anything to do with exponentiation; it becomes a generic way to apply some function — as in, transformation — to a list, and we can rewrite it accordingly:

>>> def transform(xs, f):
...     return [f(x) for x in xs]

>>> transform(xs, create_power(2))
[1, 4, 9]

Vectorization

And there's an even crazier way to go. Instead of transforming the list given a function, we can transform the function so it operates on a list. This is called vectorization:

>>> def vectorize(f):
...     def vectorized_f(xs):
...         return [f(x) for x in xs]
...     return vectorized_f

In which case we can create a square function, vectorize it, and then apply it to the list:

>>> vectorize(create_power(2))(xs)
[1, 4, 9]

The parentheses in functional programming sure do get confusing. If you've ever seen LISP, the paragon of functional languages, you know what I'm talking about:

(defun fib(n)
(if (or (= n 0) (= n 1))
    1
    (let ((a (fib(- n 1))) (b (fib(- n 2)))) (+ a b))))

But jokes aside, think of what we have accomplished so far. We created a function that represents a power of 2; vectorized it so it applies to lists; applied it to a list; and iterated over each of the results — all in a highly clear and declarative way, namely vectorize(create_power(2))(xs).

And it's pretty sophisticated code! create_power returns a function, which then goes into vectorize, which creates another function based on that function and returns it, so that it can transform a list so it adheres to iteration standards — that's funky. The result, though, is quite elegant and compact, especially if we compare it with a purely procedural attitude:

>>> result = []
>>> for i in range(len(xs)):
...     x = xs[i]
...     result.append(power(x, 2))
... result
[1, 4, 9]

Although, in this particular case, I would prefer a balanced mixture:

>>> [power(x, 2) for x in xs]
[1, 4, 9]

And Now in Pascal!

I bet you didn't see this one coming when you bought a Python programming book.

But seriously, bear with me for a moment — there's a moral to the story. The fact that Python doesn't limit us to either programming style, but allows us to combine both, is quite a big deal. I would even argue that it makes the sum greater than its part — that the two paradigms complement each other in ways that create synergy, and lead to opportunities much richer than either one can provide by itself.

TODO@image: concert.svg

That's what we should aspire to: not to make Python more functional, or more procedural, but to unite the best of both worlds, and make something even better out of it; a paradigm on its own merits.

So far, Python is doing brilliantly, at least in my opinion. It might be confusing at times — like when we first hear about functional vs. procedural programming, and don't quite understand if we're doing one or the other when writing a Python function. But this blurring of boundaries is actually a good sign — testimony to a harmonious synthesis.

Compare this to the Pascal programming language; it's not really fair, since it's quite old, but it did have a clear distinction between procedures and functions. Here's how we'd define a procedure:

procedure Hello;
begin
ShowMessage('hello world');
end;

While a function would look like this:

function Square(x: integer): integer;
begin
Square := x * x;
end;

So in effect, procedures do not have a return value — they just run a piece of code — while functions take some arguments and transform them into a result; the return-statement is expressed as an assignment to the function's name.

But then, remember our procedural comparison function, the one taking a and b and returning a number to indicate which one is larger? It's not really a transformation, but it does have a return value — so to implement it as a procedure, we would have reference parameters, that the function can write back into. It would look like this:

procedure CompareByLength(a: string, b: string, indicator: integer);
begin
if Length(a) > Length(b) then
    indicator := 1;
else begin
    if Length(a) < Length(b) then
        indicator := -1;
    else
        indicator := 0;
    end;
end;
end;

So during the execution, the indicator variable absorbs some negative or positive number, like before, to be read back by the caller.

Notice how this separation is actually limiting, causing complications — whereas Python merges both traditions, and ends up with an abstraction that is both simpler and more powerful.

It's a profound lesson in software design, or design in general; sticking for one tradition or another only makes sense until we can figure out to how to merge them into something that's better than both.

Looking at a question from different perspectives, with equal integrity and respect; addressing the good and the bad, honestly and thoroughly; and trying to salvage the best parts while letting go of the worst — this way of thinking is invaluable. Not just in software, but with any interpersonal relationship or political project; and if there is one idea I wish to impart throughout this book, this is it.

Functions as State

So far, we talked about functions that receive other functions as an argument, return them as return values, or both — collectively known as higher-order functions, to separate them from the first-order functions that operate on simple values.

We saw how this paradigm blends well with the more regular and straightforward procedural approach; but Python keeps pushing, and blends in a whole other attitude. Before we spell it out, though, I think it's interesting to see how it, too, grows out of functions — that most fundamental building block in software design.

So far, when we had functions as return values, it was so we can parametrize the created behavior. We relied on the fact that a dynamically defined function will retain access to the scope in which it was created, e.g. so we could fix the power it raises a number to:

>>> def create_power(n):
...     def power(x):
...         return x**n
...     return power

But the truth is, we can do more with a scope than just customize nested definitions. A scope is, after all, a sort of dictionary: we can add to it, remove from it, modify its values — and generally use it to manage state.

For example, let's encapsulate a multi-step process, like writing data to a file; and pretend for a moment that we don't have the builtin open function (I'm trying to keep the examples simple; but then, Python provides out-of-the-box solutions to many simple problems, so where's the fun in that?).

If we wante to do this in a procedural way, we'd have to develop several procedures to open a file, write to it and close it — all the while coordinating which file we're talking about across all calls, e.g. by passing its path everywhere:

>>> def open_(path):
...     ...
>>> def write(path, data):
...     ...
>>> def close(path):
...     ...

>>> open_('/tmp/file.txt')
>>> write('/tmp/file.txt', 'hello world')
>>> close('/tmp/file.txt')

The way this is done in C — and we can emulate this in Python with the os module — is slightly more efficient. We only pass the path once, to os.open, and get back a simple integer token called a file descriptor or file handle. We then use this number to direct os.write or os.close which file to work on without having to repeat ourselves:

>>> import os
>>> fd = os.open('/tmp/file.txt', os.O_CREAT | os.O_WRONLY)
>>> os.write(fd, b'hello world')
>>> os.close(fd)

But with higher-order functions, we can manage this messy implementation detail inside the function's belly. Suppose we define our own scope to keep this file descriptor, and return — not a token to make future operations possible, but the future operation themselves; write and close, hardwired to work on a fixed file:

>>> def open_(path):
...     fd = os.open(path, os.O_CREAT | os.O_WRONLY)
...     def write(data):
...         os.write(fd, data)
...     def close():
...         os.close(fd)
...     return write, close

Then we can use it like this:

>>> write, close = open_('/tmp/file.txt')
>>> write(b'hello world')
>>> close()

Functional Objects

This is a pretty cool idea; we can use it to abstract data in general, not just this particular flow. Suppose we have a social network, where each person has a name, and can subscribe to some other people. We could represent this with plain data — Alice and Bob would be dictionaries with a name, and a list of subscriptions:

>>> alice = {'name': 'alice', 'subscribed_to': []}
>>> bob = {'name': 'bob', 'subscribed_to': []}

And we could have functions like subscribe and unsubscribe, where we pass in a person and it's updated accordingly:

>>> def subscribe(subscriber, person):
...     subscriber['subscribed_to'].append(person['name'])
>>> def unsubscribe(subscriber, person):
...     subscriber['subscribed_to'].remove(person['name'])

>>> subscribe(alice, bob)
>>> alice
{'name': 'alice', 'subscribed_to': ['bob']}

But this feels very exposed, very fragile; so instead, let's encapsulate it in a constructor function. It takes a name, initializes an empty list of subscriptions, and returns custom subscribe and unsubscribe functions, like we did before. It should also return some accessors, like get_name and get_subscriptions, since those values are now hidden inside the function:

>>> def create_person(name):
...     subscribed_to = []
...     def get_name():
...         return name
...     def get_subscriptions():
...         return subscribed_to
...     def subscribe(person):
...         subscribed_to.append(person[0]()) # person.get_name()
...     def unsubscribe(person):
...         subscribed_to.remove(person[0]()) # person.get_name()
...     return get_name, get_subscriptions, subscribe, unsubscribe

This way, we can initialize Alice and Bob to be compact tuples of Alice- or Bob-related functions:

>>> alice = create_person('alice')
>>> bob = create_person('bob')
>>> alice[2](bob) # alice.subscribe(bob)
>>> alice[1]()    # alice.get_subscriptions()
['bob']

Of course, it's pretty horrible to access these functions by index, isn't it? You probably cringed already at the subscribe and unsubscribe implementations before.

So let's change this to a named tuple, which is a standard library extension that lets us assign an informative name to each index. We define it once, giving it a name and a space-delimited list of field names:

>>> from collections import namedtuple
>>> Person = namedtuple('Person', 'get_name get_subscriptions subscribe \
...                                                         unsubscribe')

And then we can populate it with person-specific functions:

>>> def create_person(name):
...     # same as before
...     return Person(get_name, get_subscriptions, subscribe, unsubscribe)

This way, we end up with what's effectively an object made up of plain functions:

>>> alice = create_person('alice')
>>> bob = create_person('bob')
>>> alice.subscribe(bob)
>>> alice.get_subscriptions()
['bob']

We can even do away with the accessors, and expose the name and the subscriptions list as plain attributes:

>>> Person = namedtuple('Person', 'name subscribed_to subscribe \
...                                                 unsubscribe')
>>> def create_person(name):
...     subscribed_to = []
...     def subscribe(person):
...         subscribed_to.append(person.name)
...     def unsubscribe(person):
...         subscribed_to.remove(person.name)
...     return Person(name, subscribed_to, subscribe, unsubscribe)

>>> alice = create_person('alice')
>>> bob = create_person('bob')
>>> alice.subscribe(bob)
>>> alice.subscribed_to
['bob']

This psychedelic functions-within-functions trip is used heavily in languages like Javascript; it can be extended even further, to support inheritence via the prototype pattern — but we'll talk about it when we get to objects proper.

@image: function-as-object.svg

… v. Object-Oriented

For now, let's admit that it's not easy — this trick needs to be carefully organized to remain usable, and for complex processes it can grow out of hand and become quite difficult to follow. Still, it's quite incredible that we can build such stateful and interconnected abstractions just by cleverly interlacing functions and scopes.

To achieve the same, but in a more orderly fashion, we have yet another paradigm: object-oriented programming. This practice suggests a more straightforward way of grouping attributes and operations — in this case, called methods — around some shared state.

Here, we define everything in a class, and then instantiate separate objects or instances of it, each with their own clean copy of a state:

>>> class A:
...     pass

>>> a1, a2 = A(), A()
>>> a1.x = 1
>>> a2.x = 2
>>> a1.x
1
>>> a2.x
2

The object's state is automatically passed to each method as the first argument, conventionally called self, when that method is called through a particular object:

>>> class A:
...     def f(self):
...         print(self.x)

>>> a1, a2 = A(), A()
>>> a1.x = 1
>>> a2.x = 2
>>> a1.f()
1
>>> a2.f()
2

And just like we had a constructor function to initialize our function-based object, so do classes support an initialization method with the special name __init__ (pronounced "dunder-init", as in "double-underscore"), which is called when the class is invoked to instantiate an object:

>>> class A:
...     def __init__(self, x):
...         self.x = x
...     def f(self):
...         print(self.x)

>>> a1 = A(1)
>>> a2 = A(2)
>>> a1.f()
1
>>> a2.f()
2

This lets us implement a person in a much more concise and cohesive way:

>>> class Person:
...     def __init__(self, name):
...         self.name = name
...         self.subscribed_to = []
...     def subscribe(self, person):
...         self.subscribed_to.append(person.name)
...     def unsubscribe(self, person):
...         self.subscribed_to.remove(person.name)

>>> alice = Person('alice')
>>> bob = Person('bob')
>>> alice.subscribe(bob)
>>> alice.subscribed_to
['bob']

Or a writable file object, for that matter:

>>> class File:
...     def __init__(self, path):
...         self.fd = os.open(path, os.O_WRONLY)
...     def write(self, data):
...         os.write(self.fd, data)
...     def close(self):
...         os.close(self.fd)

>>> f = File('/tmp/file.txt')
>>> f.write(b'hello world')
>>> f.close()

Python, as we keep seeing, is a profoundly object-oriented language; in fact (please act surprised), it has a builtin open function! This constructor instantiates a new file object — not descriptor, mind you, object — through which we can write, read, close and more, all while keeping the internal details hidden from sight:

>>> f = open('/tmp/file.txt', 'w')
>>> f.write('hello world')
>>> f.close()

So yeah, our own implementation is somewhat redundant. Even if we need the low-level details, like the file descriptor, they're still there — we just need to know where to find them:

>>> f.fileno()
3 # that's the file descriptor

Function State

So far, we saw how to use functions to manage state and implement an object; and we saw that Python already comes with native and optimized syntax for that purpose, which we'll cover in depth in the future. But if you remember, we actually started this chapter with the opposite idea — namely that in Python, functions are themselves objects. Seamlessly coming full circle is yet another example of how tightly and coherently the different concepts are integrated.

But if functions are objects, as I keep saying, and objects have attributes, methods, a shared state — is it true of functions? Can a function have a state, like an attribute that changes over time?

And of course it can; for example, this one knows how many times it has been invoked:

>>> def f():
...     f.runs += 1
...     print('run', f.runs)

Assuming, of course, that we've initialized the necessary attribute before the first invocation:

>>> f.runs = 0
>>> f()
run 1
>>> f()
run 2

We can even make it more self-contained, and initialize the value inside the function when it's missing, like on the first call — which we can inspect with hasattr, thanks again to it being an object:

>>> def f():
...     if not hasattr(f, 'runs'):
...         f.runs = 0
...     f.runs += 1
...     print('run', f.runs)

>>> f()
run 1
>>> f()
run 2

This feature can be useful to store default values, assuming we want to keep the default argument themselves None, to indicate absence. Before, we used to put them as a global variable, like in our `retry` function:

>>> default_max_attempts = 5
>>> def retry(f, *args, max_attempts=None, **kwargs):
...     if max_attempts is None:
...         max_attempts = default_max_attempts
...     # same as before

But honestly, it's not ideal — that global value, floating loosely next to the function. Better store it as part of the function itself, like in an attribute, so we can view and edit it just as easily, but within a clearer scope:

>>> def retry(f, *args, max_attempts=None, **kwargs):
...     if max_attempts is None:
...         max_attempts = retry.default_max_attempts
...     # same as before
>>> retry.default_max_attempts = 5

Other than that, though, stateful functions can be a bit messy. If you remember our deliberation about mutable default arguments, like the function append(item, items=[]) and its accumulating list — we did mention state as a major complexifier; something that makes invocations depend not just on the input, but on the execution history, which is less predictable and harder to work with. Still, it's possible; and like we've seen, in some cases it can even be useful.

Functions 2: Conclusion

This has been a bit of an odd chapter. We didn't learn any new syntax (unless you count LISP and Pascal); mostly, we were just playing with functions — seeing what happens when we pass them as arguments, return them as return values, use their scopes to manage state and so on. And while somewhat eclectic, this journey was intended to demonstrate just how much can be done with functions, and develop a more intuitive understanding of functional programming. We saw that it's not just about programming with functions, but using them to convey a functional purpose — so we can compose and apply these purposes in a modular, declarative way, compounding the transformations to express and adapt increasingly complex ideas.

We compared it to the more gradual and descriptive style of procedural programming, which seeks to coordinate different parts in a more flat and straightforward manner; and we learned how the two approaches can complement each other, which results in emergent qualities and harmonies that niether paradigm has on its own. Similarly, we saw how functional programming can be used to implement objects — and how object-oriented programming is actually used to implement functions. All that remains is to find out exactly how this is done — and that's precisely what we'll do in the next chapter.

TODO@image: different-story.svg

Functions 2: Further Reading

TODO