Writing/The Unbearable Lightness of Coding
Chapter 10 · 34 min read
Objects 1: IDs, Types and Values
In the next few episodes, we're going to talk about Python objects. As usual, we'll start with the motivation and the syntax, then talk about the object-oreinted design of Python itself and try to understand the implications. We'll cover magic methods that let us customize and tune up our codebase in some crazy ways, and see a real-world example that was a game-changer for doing data science in Python.
So what is an object? Most programming languages deal with more basic data types, like numbers and strings. Modern languages might include native ways to express more complex ideas, like lists and dictionaries — but eventually, there's always a limit to how much a generic syntax can accommodate our specific needs.
From Data Structures…
That's why, even in languages as old as C, there's usually a way to define custom data structures — to lay out information in a clear, informative manner, organizing data in explicit and convenient fields:
struct Person {
char* name; // string
int age; // number
};
It's a great way to encapsulate a concept; but most concepts also come with related behaviors, and we'd like to fit them in as well. Behaviors are generally best expressed as functions, so maybe let's add them as fields:
struct Person {
// same as before
void (*say_hello)(struct Person*); // function pointer
};
But then, we have to assign each such function explicitly, and pass the structure into it on every call, usually as the first argument, so it gets a chance to inspect and modify its data if necessary:
void say_hello(struct Person* this) {
printf("Hello, my name is %s and I'm %d years old.\n", this->name,
this->age);
}
void main() {
struct Person alice = {
.name = "Alice",
.age = 32,
.say_hello = say_hello,
};
alice.say_hello(&alice);
// prints "Hello, my name is Alice and I'm 32 yers old."
}
It's not ideal, but that's how we write "object-oriented" C; you can find it in some of the biggest and most serious codebases in the world, from Linux to CPython.
Still, like I said, it's not ideal; not just on the aesthetic level, but in terms of software design. The fields and the behaviors are very loosely coupled, and this affects how we perceive and reason about the system — whether our abstractions assume a flat model where we pass plain data to separate functions, or we can group them into a more robust and coherent taxonomy.
… to Classes
That's what object-oriented programming is about: the same data and the same functions, but organized in a different way. Related fields and behaviors are defined in a class, and this class can be instantiated to create a new, fresh unit of state called an object. Different objects can thus store different values, which are called attributes, and can be accessed, assigned and deleted using dot notation. So far, it's pretty similar to plain structures — for example, in C++:
class Person {
public:
char* name;
int age;
}
void main() {
Person alice;
alice.name = "Alice";
alice.age = 32;
}
However, with classes, we can also define and invoke related functionality with a similar syntax:
class Person {
// same as before
void sayHello() {
printf("Hello, my name is %s and I'm %d years old.\n", this->name,
this->age);
}
}
void main() {
Person alice = { .name = "Alice", .age = 32 };
alice.sayHello();
// prints "Hello, my name is Alice and I'm 32 yers old."
}
Specifically, note how we no longer have to assign the function, which is now part of the class, and how it can implicitly access the state of the object through which it was invoked using the this keyword.
Abstract Behaviors
Such functions are called methods, and they're a big deal. Their grouping means that different objects of the same class share the same behavior; but then, those methods are designed to receive the object from which their invocation emanates, and operator on its particular state:
void main() {
Person alice = { .name = "Alice", .age = 32 };
Person bob = { .name = "Bob", .age = 25 };
alice.sayHello();
bob.sayHello();
// prints:
// Hello, my name is Alice and I'm 32 yers old.
// Hello, my name is Bob and I'm 25 yers old.
}
So data is encapsulated and delineated much more neatly than in object-oriented C. Finally, there's a special function called the constructor, which is used to initialize an object when it's instantiated, and often bears the same name as the class:
class Person {
// same as before
Person(char* name, int age) {
this->name = name;
this->age = age;
}
}
void main() {
Person alice("Alice", 32);
alice.sayHello();
// prints "Hello, my name is Alice and I'm 32 yers old."
}
This way, we don't have to care about the object's attributes as much as about its behavior (and in fact, in C++, we would usually define them as private, and thus inaccessible from outside). This lets us deal with interfaces rather than with data: we don't care how the object organizes its information internally, just that it reacts in a predictable way. Once again, it's a big deal: it means that we can change an object's implementation entirely, as often happens when code evolves — but as long as we keep public-facing methods intact, any code that uses our objects will work as expected. For example, we can decide to store the year of birth rather than the age:
class Person {
public:
Person(char* name, int age) {
this->name = name;
this->birth_year = this->currentYear() - age;
}
void sayHello() {
int age = this->currentYear() - this->birth_year;
printf("Hello, my name is %s and I'm %d years old.\n", this->name,
age);
}
private:
char* name;
int birth_year;
int currentYear() {
time_t t = time(NULL);
tm* tm = localtime(&t);
return 1900 + tm->tm_year;
}
}
And while it's a bit more work — it's also better, because the person's age will keep updating as the years pass; in the previous implementation, it would retain its original value indefinitely. More to the point, we were able to carry out a pretty profound change without affecting the usage whatsoever:
void main() {
Person alice("Alice", 32);
alice.sayHello();
// still prints "Hello, my name is Alice and I'm 32 yers old."
}
Anyway, yeah; all that C and C++ is getting tedious; I just wanted to spice things up, so you can appreciate just what a revolution it was at the time. In Python, the syntax is much more economical — this is called self and the constructor is called __init__, but otherwise the gist is the same:
>>> class Person:
... def __init__(self, name, age):
... self.name = self.name
... self.age = self.age
... def say_hello(self):
... print('Hello, my name is %s and I'm %d years old.' % (self.name,
... self.age))
>>> alice = Person('Alice', 32)
>>> alice.say_hello()
Hello, my name is Alice and I'm 32 years old.
Like a breath of fresh air, isn't it? It's good to remind ourselves just how well we have it.
Python Objects
All of this is the usual stuff you'll encounter when reading about object-oriented programming: an introduction to core keywords and principles, followed by deliberations of how to organize your codebase in this new paradigm. And it's understandable — in classic, object-oriented programming languages like Java or C#, you might have some more advanced concepts like generics or reflection, but that's basically it. There's the procedural part of the language, where you deal with numbers and strings, code and algorithms — and there's the object-oriented part, where you define your data model.
But in Python, something else is going on. You see, as we've already mentioned, in Python everything is an object: numbers and strings have methods, functions can be passed around and have attributes — even their code is an object with its own attributes and methods. Even classes, which in most object-oriented languages is a purely organizational tool, the layout that group fields and behaviors — in Python, even classes are objects: we can print them, reassign them, pass them around.
This is a pretty profound difference; it means that Python is object-oriented in such a pervasive way, that those mechanisms govern not just user-defined classes but the entire language. This, in turn, means that if we understand how they work, we can have user-defined classes that fit right into the language, as if they were native types, taking full advantage of its syntax and semantic structures.
Before we can do that, however, we still need to articulate what exactly is an object; and in Python, where objects account for everything and cover so much ground, it's a pretty broad abstraction. Formally, an object has three parts: an identifier, a type, and a value.
Object IDs
Let's start with the identifier, or ID. Each object has its own unique token, which we can retrieve by passing it to the builtin id function:
>>> id(1)
140721419969320
>>> id(2)
140721419969352
>>> id('hello world')
2713657098352
As you can see, different numbers and different strings have different IDs; as do functions:
>>> def f():
... pass
>>> id(f)
2713655182496
And if we define our own class, different instances of it have different IDs, too:
>>> class A:
... pass
...
>>> a1 = A()
>>> a2 = A()
>>> id(a1)
2713657231248
>>> id(a2)
2713660620304
As does the class itself:
>>> id(A)
2713661166784
We also have the is and is not keywords, which let us compare the identity of two objects:
>>> a3 = a1
>>> id(a3)
2713657231248 # same as a1
>>> a1 is a3
True
>>> a2 is a3
False
Note how is returns true only if those two names actually refer to the same object: rather than comparing values, it effectively compares IDs. Even if the objects are completely equivalent, like in the case of two empty lists, they're still separate and distinct:
>>> x1 = []
>>> x2 = []
>>> x1 is x2
False
>>> id(x1)
2713660616320
>>> id(x2)
2713660616512
And it makes sense, doesn't it? One is not the other, and while they can look similar at a given moment, they can change independently of each other, so they are not the same.
Object Types
Next, we have the type. We can get this one by passing an object to the builtin type function, which returns the class where this object's behavior is defined. For custom objects, this is literally the class object we saw before:
>>> type(a1)
<class 'A'>
>>> type(a1) is A
True
But like we said, everything is an object, so there are builtin classes for things like numbers, strings and functions, too:
>>> type(1)
<class 'int'>
>>> type('hello world')
<class 'str'>
>>> type(f)
<class 'function'>
Even class objects have a type, which is always the same mysterious type class — and literally the same object as the builtin type function (well, maybe not "function", then):
>>> type(A)
<class 'type'>
>>> type(A) is type
True
But we'll talk about this weirdness when we get to metaclasses. For now, as far as our definition of objects goes, it's still pretty simple: an ID is a unique number, and the type is a reference to a class object.
Object Values (or Not)
Finally, we have the value: the essence that makes 1 a 1, or the string "hello world" what it is. It's what we see when we print the object — the configuration that determines how it it interacts with the world. It feels like the most important part, doesn't it? But at the same time, it's the most elusive: like, what makes 1 a 1? How is it actually encoded, and what's going on behind the scenes?
I think the key to this question is realizing that actually, the value is not the most important part; that in fact, it doesn't even make sense on its own. What's interesting about an object — and we've seen that this was the case from the very beginning of object-oriented programming — is not what it is, but what it does; how it behaves.
So in the case of 1, it's such an object that when we attempt to print it, we get the 1 digit; and if we add it to itself, we get a different object that prints the 2 digit:
>>> 1
1
>>> 1 + 1
2
At first, it's almost imperceptible that this is behavior, because Python's syntax makes it look so natural, like it's just basic math. But actually, these are object behaviors, defined in its class with special, "magic" methods that start and end with double underscores. When an object is printed by the interpreter, it's actually calling the __repr__ method, which we can call explicitly to get the same result:
>>> (1).__repr__()
'1'
And note that it actually returns a representation string, so to reproduce the exact behavior from before we'd also have to print it.
>>> print((1).__repr__())
1
Similarly, when two objects are added together, the __add__ method is called; but we can also call it directly, invoking it on the left operand and passing in the right:
>>> (1).__add__(1)
2
In fact, even when we call an object's method, like the bit_length of a number:
>>> (1).bit_length()
1
Think about what's actually going on: first we access an attribute by the name of bit_length, which uses __getattribute__; and then we invoke the result, which uses __call__:
>>> (1).__getattribute__('bit_length')
<built-in method bit_length of int object at 0x...>
>>> (1).__getattribute__('bit_length').__call__()
1
This might look like an odd technicality, since nobody in their right mind would use the language like that — so why does the implementation matter? The reason this is so interesting is that if we define our own classes, and implement __repr__ or __add__ (or even __getattribute__), we can make our own use cases look like basic math; we can augment the language organically to pretty unprecedented degrees.
Conceptual Values
But I'm getting ahead of myself: for now, I just want to reiterate that an object has a unique ID, a type, and a value — and that the most important part is the type, because it defines how the object behaves in different circumstances.
The value customizes this behavior, and the ID lets us distinguish between similar yet separate things — but really, it's the type that matters. Suppose we have a simple class that accepts some number, delta, and provides a method that increments another number by this delta:
>>> class A:
... def __init__(self, delta):
... self.delta = delta
... def inc(self, x):
... return x + self.delta
When we instantiate this class we get an object with a unique ID:
>>> a = A(1)
>>> id(a)
2713657105040
And when we call its inc method, we are invoking the very behavior that we defined in its type:
>>> a.inc(1)
2
As you can see, the bulk of what makes this object tick is part of its class. As for its value, we can think of it as 1, at least conceptually, because this is what calibrates its behavior to increment by specifically one. It's not the same as the number 1, of course — but it's an abstraction on top of it, which defines the behavior of "incrementing by" it. If we set delta to 2, the object changes; and if we set it back to 1, and the object behaves exactly like before.
So, going back to our question of what exactly is a value, we have at least part of the answer: for a user-defined object, the value is its state. This state is a series of references to other objects, whose values are series of references to yet other objects, and so on — until eventually we get references to basic, irreducable native types, like numbers and strings.
Mutability
The only question left to answer, then, is what exactly is the value of those basic objects — and to tackle it, we need to talk about mutability. It's a property that determines whether objects are mutable: that is, whether their values can change.
Most objects in Python are mutable: certainly all the instances of user-defined classes are, as we can always change their attributes, or even add and remove them. There are some ways for objects to resist this, but all of them can be "hacked"; at the end of the day, user-defined objects, by their very nature, represent state — and state is most useful when it's mutable. After all, when it's not, it's just data.
But a few types in Python are immutable: their values don't change. What's interesting is that those are exactly the basic objects from which the states of more complex objects are invariably composed. For example, numbers are immutable: if we have 1 and we add 1 to it, it doesn't grow into 2 — it returns a different object that is 2, with a different ID:
>>> x = 1
>>> id(x)
140721419969320
>>> x += 1
>>> id(x)
140721419969352
Strings are immutable, too. While we can access them by character, which might feel like we're dealing with some sort of list:
>>> s = 'hello world'
>>> s[0]
'h'
If we actually try to change anything, we get an error:
>>> s[0] = 'H'
TypeError: 'str' object does not support item assignment
Similarly, methods like upper don't modify the original text, but return a new, modified copy of it, with a new ID:
>>> u = s.upper()
>>> u
'HELLO WORLD'
>>> id(s)
2719693831792
>>> id(u)
2719692012016
Reusing Immutable Objects
Immutability presents an interesting opportunity that, while not terribly important to remember, might be confusing if you encounter it out of context, on your own — so let's address it. Because immutable objects can't change, there's no danger in reusing them; and since this can save some space and some time, it's exactly what Python does:
>>> n1 = 1
>>> n2 = 1
>>> id(n1)
140721419969320
>>> id(n1)
140721419969320
>>> n1 is n2
True
>>> s1 = 'hello'
>>> s2 = 'hello'
>>> id(s1)
2719693947120
>>> id(s2)
2719693947120
>>> s1 is s2
True
However, there are some caveats; they differ from implementation to implementation, but CPython, for one, doesn't cache all numbers, only those small enough that they're likely to be reused frequently:
>>> n1 = 256
>>> n2 = 256
>>> n1 is n2 # still the same object
True
>>> n1 = 257
>>> n2 = 257
>>> n1 is n2 # not the same object anymore
False
It doesn't mean that such numbers (in this case, anything larger than 256) are mutable; just that the optimization that reuses them ceases working at some limit — so unless you know exactly what you're doing, it's better to test for equality with ==, not is.
With strings, the logic is different: single characters are naturally shared, but then even very long literals can end up as the same object:
>>> s1 = 'x' * 1000
>>> s2 = 'x' * 1000
>>> s1 is s2
True
While relatively short ones don't:
>>> s1 = 'hello world'
>>> s2 = 'hello world'
>>> s1 is s2
False
The culprit, in this case, is the space between hello and world; for pretty arbitrary reasons, CPython only caches strings that could be valid variable names. It's possible to instruct it to reuse strings explicitly using sys.intern:
>>> import sys
>>> s1 = sys.intern('hello world')
>>> s2 = sys.intern('hello world')
>>> s1 is s2
True
But that's a pretty advanced optimization, really, so I'll leave it at that.
Singleton Objects
Besides numbers and strings, we also have booleans, None, and some other singleton objects like NotImplemented, Ellipsis and __debug__, which we'll talk about in due time. What this means is that these objects not so much have a value as a unique identity: True, False and None are special, builtin constants that are always themselves. That's why we often see conditions of the form if x is None — it is inherently equivalent to if x == None, just somewhat more readable.
And it's not hard to recreate, either. Suppose we have a function, f, that receives an argument or doesn't:
>>> def f(arg):
... if arg:
... print('received argument:', arg)
... else:
... print('no argument received')
This implementation is not ideal, in that f(0) and f([]) would claim to have received no argument just because their argument's truth value is false. So, like we had learned, we can test for absence rather than falsehood — even make the argument optional:
>>> def f(arg=None):
... if arg is None:
... print('no argument received')
... else:
... print('received argument:', arg)
>>> f(0)
received argument: 0
>>> f()
no argument received
This seems to work well, except in one particular case:
>>> f(None)
no argument received
That is, if we pass None explicitly, f doesn't recognize it for the argument that it is. In most cases it doesn't matter — that's what None is for, after all — but what do we do in the cases where it does? We can change the function's signature to *args and check its length, but that's clumsy; so instead, we can define a singleton object of our own:
>>> _undefined = object()
>>> def f(arg=_undefined):
... if arg is _undefined:
... print('no argument received')
... else:
... print('received argument:', arg)
>>> f(None)
received argument: None
>>> f()
no argument received
Here, we use the builtin object class to create… well, something, really — a basic object that can't do much, but we don't really care about. It's its identity that we're interested in, after all: nothing except _undefined is ever going to fulfil the condition of being _undefined, and it's safe to ignore the case where someone passes f(_undefined) explicitly, because, come on; the whole idea of supporting None was because it's a common Python builtin that recurs often.
The Mystery of the Tuple
At this point, the only immutable type left to cover is the tuple (there's also frozenset, which is an immutable set, but nobody cares about it. Seriously, I have never seen it used, ever). Tuples are like lists, but for some reason, we can't change them:
>>> x = (1, 2, 3)
>>> x[0]
1
>>> x[0] = 5
TypeError: 'tuple' object does not support item assignment
And there's something confusing about this redundancy: I mean, why are they such a central, builtin part of the language that e.g. packing or the variadic position parameter default to them? If lists are generally more versatile, why not just have them everywhere?
Some people argue that tuples are there for performance benefits, or for hashing purposes, but this is more of a beneficial side-effect than the real reason. Others say you're supposed to use tuples like this, and lists like that: maybe only put heterogenous or conceptually similar data in a list, while tuples can contain any assortment. Or maybe use tuples for const correctness, guaranteeing that certain values are not going to change when you reassign them or pass them into a function, simply because tuples are immutable and thus impossible to change.
But in my opinion, all these arguments are a bit moot, because the truth is that you can do with lists only, just like Javascript does. The way I see it, tuples are there because they fit so neatly into Python's abstraction of objects, and its overall design and implementation. To justify this view, let's recap what we know so far: everything is an object, has a unique ID, a type that determines its behavior, and a value that customizes it. An ID is a unique number, a type is a reference to another object, but what exactly is the value we haven't quite figured out yet.
In mutable types, we can imagine the value as the state of the object, which we can reduce to a series of numbers and strings. This object's value, we said, was essentially 1:
>>> a = A(1)
>>> a.inc(1)
2
And we can similarly claim that for:
>>> x = [1, 2, 3]
>>> d = {'x': 1, 'y': 2}
The value of the list is essentially 1, 2, 3, and the value of the dictionary can be reduced to the string x, the number 1, the string y and the number 2.
But what exactly is a tuple? What is its value, and what does it mean that it's immutable? It's a bit of a fine point — suppose that we have the following:
>>> t = (1, 2, [])
>>> t
(1, 2, [])
This tuple's value is 1, 2 and an empty list, right? We can see its representation, and we don't expect it to change. But here's a fun party trick for you:
>>> t[2].append(3)
>>> t
(1, 2, [3]) # whaaat
We can nonchalantly append a value to its list, and it sure seems like the tuple changed, doesn't it?
The point here is that it didn't: the tuple's value is not really 1, 2 and an empty list, but references to these objects — think of it as their IDs. And the IDs, the references, never changed: the last one still points to the same list, after all, and the fact that this list changed is really none of the tuple's business.
This also explains funny bugs like this one, when we try to create several lists with a seemingly clever shortcut:
>>> x = [[] * 5]
>>> x
[[], [], [], [], []]
But then when we append an item to one of them, it gets appended to all of them:
>>> x[0].append(1)
>>> x
[[1], [1], [1], [1], [1]]
What's actually happening is that we're creating one list inside another list, and then telling Python to replicate the reference — to the same list — multiple times, ending up with an outer list that contains the same inner list object several times over. When we append an item through any one of these references, then, the change is naturally reflected all across.
Numbers, Strings and Tuples
So anyway, tuples are effectively an ordering of other objects; and together with numbers and strings, this special subset of immutable types is enough to represent most everything else. Because these types are immutable, they're allowed to be opaque: their actual values, how they're implemented and encoded, isn't exposed — and doesn't really matter. As long as their behaviors are consistent and their state can't change, 1 will always be the object that behaves like 1, 2 will always be the object that behaves like 2, and adding the 1 object to itself would result in the 2 object, just like we'd expect.
Another way to think about it is that for immutable types, the value is entirely implied through the behavior — while for mutable types the value is the object's state, which can be represented as a composition of some more basic immutable types. If a string could change, its state would need to be mode of some more elementary particle, like a character. The same goes for a tuple: it's not a container like a list, because it doesn't have a state; it's an immovable series of references to other objects, which can be accessed in a fixed order. Even the object abstraction itself, in a way, relies on these basic, opaque concepts: it's a tuple of three references — a number, a class and a state, where the last two are further reducable to to tuples of what's eventually a specific order of number and strings.
To be fair, there are also a few other low-level constructs, like the code objects that make up functions or the traceback objects attached to exceptions, which are technically immutable; I say technically, though, because they could be implemented otherwise, and the way they are is mostly a matter of efficiency and practicality.
To bind everything together, I'd like to reiterate that it's the object's type that really matters; its value is basically a parameter to the behaviors this type defines. This parameter is, in turn, a bunch of references to other objects; and again, it's their types that matter, and their values are mere subparameters referencing other objects. And so it goes — until eventually we get to objects so basic, that their value is immaterial. In other languages, like Java or C#, not everything is an object — some data (namely, numbers, characters and arrays) are encoded in a more low-level way, and have to be treated accordingly. Python, on the other hand, found a way to fold the entire language into the object-oriented paradigm, by treating "primitive" data as objects that are immutable; and since their state can't change, its specifics are irrelevant, so this "primitive" functionality can be expressed solely through interfaces and behaviors. These interfaces and behaviors, in turn, can be extended to all other objects, so we end up with an exceptionally coherent, flexible and powerful language model — and I think that's a big part of Python's intuitive feel, popularity and success.
Implementing Behavior: Representation
Alright, that was all very minute and abstract: the key takeaway is that an object's essence is not in its value or state, but in its type and behavior — so let's talk about that.
A class is a user-defined type, and behaviors are methods with special name, starting and ending with two underscores. For example, when we print an arbitrary instance, we get a description of it, right? Capital A object at 0x-something:
>>> class A:
... pass
>>> a = A()
>>> a
<A object at 0x000002793A99FE90>
This, we saw, is coming from __repr__; and it's not hard to reconstruct. Objects also have a __class__ attribute, which references their class, and classes have a __name__ attribute, which stores their name. As for the 0x-something, it's actually the object's ID in hexadecimal notation, which we can double-check with the builtin hex function:
>>> hex(id(a))
0x2793a99fe90 # same as 0x000002793A99FE90
So effectively, we could reimplement the default behavior like so:
>>> class A:
... def __repr__(self):
... return '<%s object at %016X>' % (self.__class__.__name__,
... id(self))
>>> a = A()
>>> a
<A object at 0x000002793A99FE90>
Of course, it makes no sense to reimplement it to achieve exactly the same result — usually we'd override it to have a custom, nicer representation. For example, we can use it to reflect an object's value, or state, however we understand it, so it's easier to understand at a glance how the object is customized:
>>> class A:
... def __init__(self, x):
... self.x = x
... def __repr__(self):
... return '%s(%d)' % (self.__class__.__name__, self.x)
>>> a1 = A(1)
>>> a2 = A(2)
>>> a1
A(1)
>>> a2
A(2)
There are actually some nuances already on this level, of what makes a good representation string, but we'll cover those details and caveats in the next chapter. For now, let's check out another behavior.
Implementing Behavior: Equality
If a and b are conceptually the same, it doesn't mean they're considered equal, because Python's default equality behavior is to compare IDs, like the is keyword:
>>> a = A(1)
>>> b = A(1)
>>> a == b
False
So while both objects' x attribute is equal, the objects themselves are not; we need to add __eq__ to their class if we want to outline a different logic:
>>> class A:
... # same as before
... def __eq__(self, other):
... return self.x == other.x
>>> a = A(1)
>>> b = A(1)
>>> a == b
True
Notice how once again, we invariably delegate to more basic types — but we get to built new things on top of them, which adhere to a similar syntax. Maybe this example is a bit banal, but we could just as easily have arbitrarily complex schemes, like two teams being equal if they have the same amount of members, which in turn have the same amount of resources and skills.
Once we define equality, Python tries to generalize it, so inequality automatically works as its negation:
>>> c = A(2)
>>> a == c
False
>>> a != c
True
But if for some reason this standard logic doesn't suit us, we can always implement some different behavior in __ne__ (for "not equal"). Some things Python doesn't generalize, though: for example, we don't automatically get complete ordering between our objects, so we can't test if one is greater than the other:
>>> a > c
TypeError: '>' not supported between instances of 'A' and 'A'
To do that, and to support all its variations properly, we need to define __gt__ (for "greater than"), __lt__ (for "less than"), __ge__ (for "greater than or equal to") and __le__ (for "less than or equal to").
You might think that having __gt__ means that Python will figure out its negation, __le__, but it doesn't; and you might think that having __gt__ together with __eq__ means Python will figure their combination, __ge__, but it doesn't either. This might be a bit frustrating when our objects' values are so clearly orderable, but the aim is to support arbitrary behaviors: if some text object's __gt__, as in t > s, means t starts with s, it's not immediately clear that t <= s means the same (e.g., what if we want to use < for "ends with"?). Or, if a different text objects uses > and < to compare the text's length, we might still want to reserve == for comparing their content, in which case >= ("the string is longer or exactly the same content") makes little sense. So really, the automatic __ne__ we get from __eq__ is the exception rather than the rule.
Implementing Behavior: Containment
Let's check out one more behavior before we combine everything we learned into something that's actually useful and wrap up for the day. When I said before that immutable native types like numbers, strings and tuples are implemented in an opaque, low-level way, I skimmed over the fact that there are mutable native types, like lists and dictionaries, that are implemented a similar way. In their case, though, it's done mostly for performance reasons: we could, in principle, implement everything but their native syntax ourselves.
You see, our classes can pretend to be lists and dictionaries by implementing container behaviors: __getitem__ receives an index or a key passed into square brackets and returns the result; __setitem__ receives a key and a value, parsed from a square brackets assignment; and __delteim__ receives the key when we use a square brackets delete statement. Take this, for example:
>>> class A:
... def __getitem__(self, key):
... print('getting', key)
... return 1
... def __setitem__(self, key, value):
... print('setting', key, 'to', value)
... def __delitem__(self, key):
... print('deleting', key)
>>> a = A()
>>> a['x']
getting x
1
>>> a['x'] = 2
setting x to 2
>>> del a['x']
deleting x
This class is not particularly useful, in that its container methods don't actually do anything meaningful — but it illustrates how seamlessly we can tap into the syntax and integrate custom logics into the language. If we didn't have lists in Python, we could make them ourselves; for example, by hiding a series of linked node objects behind the same interface:
>>> class Node:
... def __init__(self, value):
... self.value = value
... self.next_node = None
... class List:
... def __init__(self, *values):
... self._first_node = cursor = None
... for value in values:
... node = Node(value)
... if cursor is None:
... self._first_node = cursor = node
... cursor.next_node = cursor = node
... def __getitem__(self, index):
... cursor = self._first_node
... while cursor and index > 0:
... cursor = cursor.next_node
... index -= 1
... if not cursor:
... raise IndexError('list index out of range')
... return cursor.value
>>> x = List(1, 2, 3)
>>> x[0]
1
>>> x[1]
2
>>> x[2]
3
>>> x[3]
IndexError: list index our of range
This is a rather partial implementation; we don't have __setitem__ and __delitem__, not to mention other behaviors (like calculating the length, checking if an element is in the list or iterating over it) that we'll cover later. We could also conceive of various optimizations, like making the nodes doubly-linked, which would make some operations more efficient — but you get the point: we can make something that looks and acts like a list, entirely by ourselves.
As an aside, note that unlike Java or C++, Python doesn't have a notion of private attributes: the object state is always public and accessible from outside. In case of internal implementation details, like the first node of the list, this is not ideal: we wouldn't want to allow code such as x.first_node, especially if we don't want to commit to this particular implementation just yet. So by convention, we prefix private attributes with an underscore: writing x._first_node is still perfectly doable, but at least the weird syntax draws attention to the fact this attribute isn't officially supported, so whoever is using it is doing so at their own risk — we reserve the right to change or remove it in the future.
This attitude is prevalent in Python, and was captured by Tim Peters in a perfect metaphor: "we are all consenting adults." — so we can touch each other's private parts, so to speak, as long as we do it in a responsible and agreed-upon manner. Or, as others have put it: "feel free to shoot yourself in the foot if you must." The idea that language features like private attributes or const correctness "protect programmers from themselves" is both limiting and patronizing; better to agree what not to do on an advisory basis, so we can do it anyway if the situation calls for it.
Also note that in Node (which, according to this logic, we could have called _Node), the attributes value and next_node are not prefixed, because the interface between it and List is such that they are "public", and accessed directly by List.__getitem__.
Implementing Pandas
And of course, we already have lists; so what can we do, with the limited set of behaviors we covered, that is new and exciting?
In case you haven't heard about pandas, it's a popular library for data science; and it provides a very special class called the data frame. Suppose we have some tabular data, like from a spreadsheet: we can organize it as a list of rows, with each row being a dictionary that maps the column names to the cell values:
>>> values = [{'x': n, 'y': n**2} for n in range(5)]
>>> values
[{'x': 0, 'y': 0}, {'x': 1, 'y': 1}, {'x': 2, 'y': 4}, {'x': 3, 'y': 9}, {'x': 4, 'y': 16}]
Now, let's say we want to filter out all the rows where y is greater than 5: how do we do this? Working with native objects, it's pretty cumbersome — we'd have to iterate through the data and store the rows that fit this criteria elsewhere. If you recall, we called this the procedural approach; and if you have any experience with data science, then you know how a lot of it is conceived in declarative (what we called functional) terms, like applying a filter in Excel.
So what pandas data frame does is effectively repurpose Python's syntax for such declarative data handling: if we just wrap our data in it, it suddenly lets us write:
>>> import pandas
>>> df = pandas.DataFrame(values)
>>> df[df['y'] > 5]
[{'x': 3, 'y': 9}, {'x': 4, 'y': 16}]
If you haven't seen such syntax before, take a moment; it's pretty bizarre. In some way, it doesn't even look like Python, since this isn't how Python dictionaries are supposed to work. But what's so fun about Python's design philosophy is that it's really egalitarian and empowering — I think Python's ecosystem is so diverse and so powerful as a direct result of this attitude. Instead of telling us "nope, this syntax is invalid", it says "why not" — as long as it can parse what's going on and we provide it with the right behaviors, Python doesn't mind running it on our behalf.
Still, it's a funky syntax, so let's see what it would mean to implement it ourselves. Again, I'm going to do it very partially, just to make this example work and get the point across — but here goes:
>>> class DataFrame:
... def __init__(self, values):
... self._values = values
First, we have a data frame class that receives our native list of dictionaries and stores it. Next, we need to address df['y']: this means we need to have __getitem__ that returns some special object which we can compare to 5 in a moment:
>>> class DataFrame:
... # same as before
... def __getitem__(self, column):
... return Filter(column, self._values)
>>> class Filter:
... def __init__(self, column, values):
... self.column = column
... self.values = values
In order to do the actual comparison, we need at least __gt__; and the trick here is that we're going to return a list of booleans, indicating whether this comparison holds true or not for each row:
>>> class Filter:
... # same as before
... def __gt__(self, other):
... return [value[self.column] > other for value in self.values]
At this point, we have:
>>> df = DataFrame(values)
>>> df['y']
<Filter object at 0x...>
>>> df['y'] > 5
[False, False, False, True, True]
This expression actually goes into df[...] again, so it seems like we have to refactor our __getitem__ — specifically so it behaves differently when it receives such a list:
>>> class DataFrame:
... # same as before
... def __getitem__(self, arg):
... if isinstance(arg, list):
... return [value for value, include in zip(self._values, arg)
... if include]
... return Filter(arg, self._values)
Et voilà:
>>> df[df['y'] > 5]
[{'x': 3, 'y': 9, {'x': 4, 'y': 16}]
There are still a lot of things to polish in this example — but hopefully it gives you a taste of just how much is possible, and how all these "magic" libraries like pandas work.
Conclusion: Objects 1
So yeah, objects. We saw that everything in Python is an object, meaning it's made up of an ID, a type and a value; and we saw how it's the type that holds the key to understanding an object. We talked about immutability, and how it lets us treat the "primitive" data that makes up objects as if it was an object itself. This way, the entire language is governed by the same interfaces — and we saw that tapping into them lets us augment the language and integrate with it organically. This can help with keeping our codebase clear and tidy, but it also lets us turbocharge it into a mini-language of our own: for example, pivoting Python's rich syntax for declarative data science. In the next videos, we'll take a deep dive into all the different magic methods and the protocols by which Python objects operate, so stay tuned.