Native Types: Numbers, Strings and Containers

We've covered so much of Python at this point, that it might seem weird to regress back and talk about its most basic types, like numbers and strings. It started out of two "miscellaneous" topics that I always tried to cover in my courses, because they confuse so many people: string encodings and datetime timezones. Even otherwise experienced programmers don't necessarily have a strong enough grip on how these work, so they might mishandle them — and these two topics are, strangely enough, harder to fix than to get right on the first attempt, so I felt it's important as "preventive treatment".

As I started writing about them, though, I realized there's a lot more to say about numbers, lists and dictionaries too, as well as path objects or regular expressions — which ended up ballooning into two entire chapters: this one from strings, the next one from datetimes. I considered putting them at the beginning, what with the topics being so seemingly basic — but in some sense, their intricacies are such that they are more advanced than loops or conditions, and more pernicious than metaclasses or packages; maybe because they go beyond Python altogether, so despite its best effort to encapsulate them, it can't fully account for the way computers were built to accommodate text and time.

Integers

Let's start with numbers, though: Python actually supports 7 types of them, believe it or not. First and foremost we have integers, of course — which, unlike other languages, can grow arbitrarily large (what's known as a BigInteger), at least until we run out of memory; and they come in four different flavors: decimal, binary, octal and hexadecimal:

>>> 250
250
>>> 0b11111010
250
>>> 0o372
250
>>> 0xFA
250

These four numeric basis are the most popular in computer science, so Python supports them right off the bat.

The decimal case, base 10, should be familiar enough: 250 means we sum .
0b11111010 means pretty much the same in binary, or base 2: .
0o372 uses octal numerals, or base 8: .
And 0xFA is hexadecimal, or base 16, where the additional numerals (after 0-9) are denoted with A-F: .

You might have noticed, when I was fixing permission errors with os.chmod, that octal is used by Linux to represent whether files are readable, writable, and/or executable for their owner, group and others. If we use a single bit to encode each of these nine combinations, then the standard configuration of a file being readable-writable by the owner and only-readable by everyone else would be 0b110100100, or in more condensed octal form (which matches the fact that we have three entities nicely), 0o644.

As for binary numbers — you might see them a lot when working with low-level data or code, since at the end of the day, everything on your computer is encoded like that, in ones and zeros. The bitwise operations Python supports, for example, are intended to manipulate exactly this representation; seeing that 2 | 5 is 7 might look arbitrary, but:

>>> 0b010 | 0b101 # merge the bits; keep 1 if either is 1
7 # 0b111
>>> 0b010 & 0b101 # intersect the bits; keep 1 if both are 1
0 # 0b000
>>> 0b010 ^ 0b101 # compare the bits; keep 1 if they differ
7 # 0b111
>>> ~0b101        # invert the bits; change 1 to 0 and 0 to 1
2 # 0b010
>>> 0b101 << 2    # shift the bits to the right, adding zeros
20 # 0b10100
>>> 0b101 >> 2    # shift the bits to the left, cropping them
1 # 0b001

As for hexadecimal, it's primarily used as a condensed way to represent binary: 0xFA is just much shorter to read than 0b11111010, and captures the entire range of a byte (0-255) in just two digits; so color codes, for example, like #FF0000, are a more concise way to describe three bytes for the values of red, green and blue: in this case, maximum red, no green, and no blue, resulting in ⯀.

Still, until we get good in converting them in your head, Python provides the builtin bin, oct and hex functions to do so:

>>> bin(250)
'0b11111010'
>>> oct(250)
'0o372'
>>> hex(250)
'0xfa'

And if we end up with such a string that we'd like to convert back to integer, int takes an optional second argument, denoting which base to use:

>>> int('11111010', 2)
250
>>> int('372', 8)
250
>>> int('FA', 16)
250

Another fun fact about integers, which applies to all their flavors, is that we can add _ to them, which Python is simply going to ignore. This feature is utterly genius, in my opinion; it was popularized by the Ruby programming language, by some views Python's main rival and competitor, back in 1995 — and took 20 years to make into into Python, with version 3.6. But it finally did, and now it lets us write things like:

>>> 1000000000    # is this a billion? a 100 million? 10 billion?
>>> 1_000_000_000 # ah

Which is considerably more readable; just like in text, we'd write 1,000,000,000, adding commas for a similar purpose. It also works marvelously well with hexadecimals, letting us separate bytes, for example:

>>> 0x_FF_00_00
16711680

And with binary, we can even separate half-bytes, brilliantly called nybbles:

>>> 0b_1111_1010
250

Fractions and More

Then, there are floats, able to represent not just whole numbers, but fractions. These are encoded in a bit of a quirky way, using 32 or 64 bits; the first version is usually called a "float", while the second one is called a "double", but in Python's float actually uses the second one — calling it a double (meaning, a double-float) would be strange without first supporting a single-float, but it was decided that having one type is simler, so… yeah.

Generally, naming is not the strongsuit of this format — even the name "float" itself is funky. It stands for floating-point precision, because of the technique it uses to utilize its fixed size to describe any number. The key insight behind it is that when dealing with small numbers, like 0.001, precision really matters; it's half of 0.002, after all, despite the small difference. With large numbers, like a billion, such a difference would be negligible — so for big enough values, this decimal point "floats" up, effectively cropping "insignificant" differences, while for small enough values it "floats" down, effectively adding more resolution.

This is a very smart way to encode something that's truly impossible to encode: given that there's an infinite number of fractions between 0 and 1, it's not at all clear how to use limited memory to describe them. It can, however, lead to precision loss if we pendulate this point to its extreme:

>>> x = 2**64 + 0.001
>>> x - 2**64
0.0

As you can see, such a large number can't maintain such fine precision; and even for smaller values, certain noise, known as floating-point inaccuracies, is introduced:

>>> x = 2**32 + 0.001
>>> x - 2**32
0.0010004043579101562

If you're working with data where greater precision is crucial, like financial calculations, beware of floats: Python provides the decimal.Decimal class instead, where we can set whatever precision is necessary ourself. By default, it lets us keep 28 digits after the dot, regardless of how big our numbers are:

>>> from decimal import Decimal
>>> x = 2**64 + Decimal(0.001)
>>> x - 2**64
Decimal('0.00100000')

And while it's a bit more cumbersome to work with, it provides a lot of functionality to work with very large, yet very precise numbers. And if this is not enough:

>>> print(1 + Decimal(1/10**28))
1.000000000000000000000000000

We can always extend the precision by setting setting prec on the global decimal context, or temporarily creating a local context instead of it:

>>> import decimal
>>> decimal.getcontext().prec = 32
>>> print(1 + Decimal(1/10**28))
1.0000000000000000000000000001000
>>> with decimal.localcontext() as ctx:
...     ctx.prec = 64
...     print(1 + Decimal(1/10**60))
1.000000000000000000000000000000000000000000000000000000000001000

Going back to floats, though, another interesting phenomena we might encounter (and occasionally even use) is what's known as scientific notation: rather than writing very large or very small numbers with lots and lots of zeroes, Python might do:

>>> 1/10**28
1e-28

Meaing, take 1.0, and then shift its floating decimal point 28 places to the left, resulting in 0.0000000000000000000000000001. Alternatively:

>>> 2.0**64
1.8446744073709552e+19

Means to take 1.8446744073709552, and shift its point 19 places to the right, resulting in 18446744073709552000.0, which is close enough to the real value of , that is 18446744073709551616 — you can literally see the rounding performed; this means that for such a big number, we wouldn't just lose a 0.001, but even 1000:

>>> 2.0**64 + 1000 - 2**64
0.0

Anyway, we can use scientific notation ourselves:

>>> 1e-3
0.001
>>> 1e+3
1000.0

And if this isn't weird enough for you, the floating point format also supports infinities: float('inf') stands for positive infinity and float('-inf') for negative one; we can even do funny maths with them:

>>> float('inf') + 1
inf
>>> 1 / float('inf')
0.0
>>> 1 / float('-inf')
-0.0
>>> float('inf') / float('-inf')
nan # not a number

But yeah, please don't do that.

The last type of number Python supports, but I'm not really going to elaborate on, is complex numbers: , where stands for . They have their own logic, like:

>>> 1+2j * 2+3j
(1+7j)
>>> abs(1+2j)
2.23606797749979

And they were added to Python very early on, presumably before anyone knew better. In retrospect, I really don't think they belong in the language: we could very easily have implemented a j = ComplexNumber() object to do all of the above with the slightly different notation of 1+2*j, but that's what we have. Unless you're doing complex analysis, though (in which case, you better know what they are — and are probably annoyed already that it's j instead of i), don't worry about them; they're one of very few corners of the language that Python would be better off without.

Encoding Integers

Like I said, everything in a computer is stored as zeros and ones; all the objects we see, all the interfaces we interact with, those are all abstractions upon abstractions upon abstractions, manipulating vast arrays of such bits with lightning speed. But it's all bits in the end, and truly internalizing it is fundamental to understanding computer science.

We've already seen how numbers can be encoded: we can write them in binary representation, and that's it, right? Well, almost: in all modern computers, the convention is to group bits in groups of 8, called bytes, and read these bytes sequentially. So the only numbers we can truly represent in a single unit are those that fit in these combinations: namely, 0 to 255, or sometimes -127 to 128. Any larger number would require multiple bytes, which begs the question: in what order should we read them? Do the earlier bytes represent the most significant bits, or MSB, so the number 65280 (11111111_00000000) is stored as the bytes 11111111, 00000000? This makes sense notation-wise, but if we were to actually implement it, wouldn't it make more sense to consume the least significant bits, or LSB, first — that is, store it as 00000000, 11111111?

There's no right answer: MSB, also known as big endian, is popular in network protocols, while hardware chips tend to prefer LSB, also known as little endian. Calling it endianess is particularly apt: the term is taken from Jonathan Swift's book about Gulliver, where he encounters two warring kingdoms, divided over whether a boiled egg's shell should be broken from the big end or the little one. When representing integers larger than a byte, we are faced with a similar dilemma:

>>> x = 65280
>>> x.to_bytes(2, 'big')
b'\xff\x00'
>>> x.to_bytes(2, 'little')
b'\x00\xff'

And here you can see that hexadecimal notation again: the b prefix means its a string of bytes, not characters, and each byte can be represented with an escape sequence, from \x00 to \xFF. You can always turn it into a list to get the numbers:

>>> list(b'\x00\xff')
[0, 255]

Or from a list of numbers into bytes:

>>> bytes([0, 255])
b'\x00\xff'

Which can be used to reconstruct the full integer:

>>> int.from_bytes(b'\x00\xff', 'little')
65280

And notice how confusing the endianess without adjusting the bytes accordingly will yield a completely different result:

>>> int.from_bytes(b'\x00\xff', 'big') # should have been b'\xff\x00'
255

Anyway, that's the way integers are represented in bytes: the process of convering them to such is called serialization, or encoding, and the process of translating them back into values is called deserialization, or decoding.

This might feel low-level to the degree of being irrelevant in Python — but when reading or writing a binary file, or sending data over a socket, there really is no other way; and once we get to strings it can become quite confusing, so it's important to understand.

Encoding Strings

Let's talk about strings, then. How would you represent them, using bits and bytes? The most straightforward solution would be to map each character to a number — and then, it's a problem we've already solved.

So we take the 23 letters of the English alphabet, double it to account for capitals, and throw in some punctuation symbols, whitespace and other special characters. The result is still less than 128, so we can fit the whole thing in a single byte, and still have one bit to spare. This is what's known as ASCII, or the American Standard Code for Information Interchange — and the map of such characters to integers is known as the ASCII table. We can transform them to and fro with the builtin ord and chr functions:

>>> ord('a')
97
>>> chr(97)
'a'

But here it can get confusing: the byte b'a' is not the the same as the character 'a'. You see, the character a is part of the abstract domain of text, not yet encoded as of yet; and only according to the ASCII table, and some other so called encodings that derive from it, does it map to the number 97.

The byte b'a', on the other hand, is… nonsensical, really. A byte can store a sequence of bits, nothing else; even saying that it can store a value between 0 and 255 presupposes that we have agreed upon a standard way to represent numbers through binary data — and like I said, even this isn't set in stone, since the same bits might very well represent a number between -127 and 128. Still, the serialization scheme we discussed is so ubiquitous, that for all intents and purposes we can assume a byte can store a number — with text encodings, this isn't the case at all.

How would you represent the latter 'ä'? Or the Hebrew 'א', or the Japanese 'あ', let alone the all-importat poop emoji, '💩'? All the different symbols in the world definitely do not fit within our limited range of 256 options — so while it's all good and well tht the American Standard Association found it appropriate to dedicate to map it first and foremost to English, other people had different ideas, and came up with different encodings.

So what does the byte b'a' mean? It's not the same as the letter 'a', not at all — it's actually the number 97, and a more "unbaised" way to represent it would be through its escaped hexadecimal sequence, \x61. In Python, however:

>>> b'\x61'
b'a'

It was decided that when a specific byte happens to map to some character according to the ASCII encoding, representing it as characters would be more intuitive and readable than as binary code. On one hand, when we do mean to encode strings into bytes, or decode bytes into strings, and we are using ascii — this all feels very natural and commnsensical:

>>> 'hello'.encode('ascii')
b'hello'
>>> b'hello'.decode('ascii')
'hello'

But when we don't, things can get pretty confusing. If we'd try to encode the number 65377, for example, in big endian:

>>> (65377).to_bytes(2, 'big')
b'\xffa'

It so happens that its first byte is 0b11111111, or \xFF — but its second byte is 0b1100001, or \x61, also known as 97, or simply a. And as we work with binary data with all sorts of values, we will encounter all sorts of characters that do not mean what we might think they mean. If we devise an encoding that maps b to 97, just to troll people, the byte b'a' might actually represent b.

This distinction between strings and bytes is particularly important, since the bytes '\xffa' is not even valid ASCII:

>>> b'\xffa'.decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

That is, the first byte, \xff, falls outside the valid ASCII range and has no corresponding character; the same would happen to any value above \x80. If we'd want to convert an arbitrary sequence of bytes to an arbitrary string, then, we might want to choose an encoding that actually covers this entire range — famously, Latin1:

>>> b'\xffa'.decode('latin1')
'ÿa'

Unicode

There is, however, a de facto standard used by most modern formats and languages to encode strings, known as unicode. It's not actually an encoding, but a charset, or character set: an abstract mapping between all the characters used around the world to so called code points — numbers that fit within 4 bytes, from 0 to 4294967296.

This standard was designed in such a way as to coincide with ASCII for its first 128 values (and even Latin1, for its first 256); so the builtin ord and chr functions actually have a broader purpose — namely converting between a character and its code point:

>>> ord('א')
1488
>>> ord('あ')
12354
>>> ord('💩')
128169
>>> chr(1488)
'א'
>>> chr(12354)
'あ'
>>> chr(128169)
'💩'

When it comes to encoding strings, however, unicode is more of a standard than an implementation, and there are several actual encodings to do this job. The first and most trivial one is UTF-32, so called because it uses the full 32 bits (or 4 bytes) necessary to describe each character for its Unicode Transformation Format:

>>> 'aאあ'.encode('utf-32')
b'\xff\xfe\x00\x00a\x00\x00\x00\xd0\x05\x00\x00B0\x00\x00'

We see the four bytes necessary to describe a (which really, fits in one byte, so the rest are zero); then four for א (that is, 0x5d0 with 2 zero bytes), and then four for あ (0x3042 with two zero bytes, where confusingly, 0x42 is represented as B and 0x30 as 0).

The four bytes at the beginning are called the BOM, or byte-order mark: they indicate that the serialization is done in little endian, which is the default used by my machine. If we wanted to set it explicitly, we could use utf-32le or utf-32be, in which case the BOM would be omitted on account of the endianess being well-defined.

As you can probably see, UTF-32 is pretty wasteful; unless we're using some very rare characters, the probability of us reaching its higher ranges is extremely low. All of the above would fit just fine with UTF-16, which uses 16 bits (or 2 bytes) instead:

>>> 'aאあ'.encode('utf-16')
b'\xff\xfea\x00\xd0\x05B0'

Here, we end up with just one byte wasted for zero padding, and even the BOM (since we still need to know about endianess) is shorter.

There is, however, an even shorter version: UTF-8, which attempts to use a single byte per character. That might sound weird, since the whole story started with the problem of fitting all these characters in a single byte — but UTF-8 actually comes up with an elegant solution: it's a variable-length encoding. Basically, it reserved one bit to indicate whether another byte is needed: so if a character fits within 7 bits, as ASCII does, one byte is enough — and if not, another one is read, which similarly contributes 7 bits of range, and 1 to indicate if yet another byte should be read:

>>> 'aאあ'.encode('utf-8')
b'a\xd7\x90\xe3\x81\x82'

Unsurprisingly, a fits within a single byte, א within two, and あ within three, although the non-trivial use of bits makes it a bit harder to decipher the exact code points. Also, since endianess doesn't matter here, we don't have a BOM — although you might sometimes encounter them with the prefix \xEF\xBB\xBF, used to indicate it's a UTF-8 encoding and not something different.

Interestingly, UTF-8 gives us a tradeoff: we saved a byte for a, but had to use an extra byte for あ, as opposed to UTF-16, which managed to describe it in 2 rather than 3 bytes; so if we were dealing with lots of Japanese text and wanted to save space, going with UTF-16 would actually work to our advantage. For English, though, and Latin languages in general, UTF-8 provides the most compressed option — so this is what Python uses by default, e.g. when we call str.encode or str.decode with no argument, or open a file for writing without explicitly specifying the encoding.

Strings

We've talked a bit about string encoding, but we glimpsed over the string object itself, so let's remedy this oversight. In Python, strings can be enclosed in either single or double quotes — I prefer single ones (as does Python itself), since they avoid visual clutter, although some people swear by double quotes (like the black formatting library), claiming using them avoids the confusion over whether it's one " or two ''. In any case, having this option to choose from lets us include one in the other, when we need to:

>>> print("Scarlett O'Hara")
Scarlett O'Hara
>>> print('"Tomorrow is another day," she said.')
"Tomorrow is another day," she said.

If we need both types of quotes, however, we have no choice but to escape them, which is done with a backslash:

>>> print('"Tomorrow is another day," said Scarlett O\'Hara.')
"Tomorrow is another day," said Scarlett O'Hara.

We've seen similar escaping, followed by x, to describe a hexadecimal value in a byte, as in \x61; strings support that, too, as well as the more versatile \u, followed by 4 hexadecimals (allowing the UTF-16 range), and \U, followed by 8 (for full UTF-32).

We don't usually need this, seeing as we can type or paste any such character directly into our string (Python source code is itself UTF-8, so it supports them all); what we might need, though, is a way to type otherwise tricky characters: \t for tab (since an actual tab character might be mistaken for a sequence of spaces) and \n for newlines, that is line breaks.

On Windows, a proper line break is actually \r\n, where the \r character is called a carriage return. It's a strange story, but back when people were using typewriters, pulling the newline lever (called the line feed) merely shifted us one line down — not to its start, but as far into it as we were — so an additional button was pressed that returned the carriage to its beginning; so oddly enough, computer pioneers decided to emulate this behavior, and now if we forget to add \r we get:

line 1
      line 2
            line 3

Whereas if we add \r without \n, we go back to the beginning of our current line and overwrite it. When working with text, Python does a pretty good job of handling this anomaly from us, automatically condensing \r\n into \n when we read files on Windows, and expanding it back when we write them — but just in case you encounter this strange behavior, I thought I'd mention that it exists.

As an aside, if we're writing a string with multiple newlines, rather than putting it all in one line with a bunch of \ns we can opt for a multiline string with triple quotes:

>>> print('''line 1
... line 2
... line 3''')
line 1
line 2
line 3

In which case a backslash at the end of the line disables the linebreak; you'll often see those at the first and/or last line of a multiline string, so that we can the triple string on a separate line without this resulting in leading or trailing newlies:

>>> print('''\
... line 1
... line 2
... line 3\
... ''')
line 1
line 2
line 3

Now, since backslashes are used so extensively for escaping, we can't really use them in strings as-is; so if we need to we'd have to escape it as \\. This can get a bit tedious when we have a lot of them (for example, when working with paths on Windows, or with regular expression, which we'll cover later) — so Python also provides us with the concept of a raw string, or r-string, in which escaping is disabled and backslashes present as-is; all we have to do is add r before the first quote:

>>> print(r'C:\Users\User\file.txt')
C:\Users\User\file.txt

A similar convention is used for format strings, or f-strings, this time by adding f. This is an incredibly useful tool that I've been deliberately avoiding until now, just so I can also introduce you to old and new-style formatting as well (with % and format) — because the truth is, f-strings render both of those pretty irrelevant, so from version 3.6 on, they are probably all that you will see.

Essentially, f-strings work like that cool frame-inspecting log function from the last chapter: they let us interpolate values, and even expressions, right into our string:

>>> x = 1
>>> print(f'{x} + {x} = {x + x}')
1 + 1 = 2

Which is both more readable than '%s' % x, less repetitive than '{x}'.format(x=x), and generally more powerful than either — since, once again, it lets us embed not just any name available in our scope, but almost any code utilizing these names. We could raise a value error like so:

ValueError(f'invalid value: {value} (expected: {", ".join(options)})')

Or a representation method like this:

def __repr__(self):
    return f'{self.__class__.__name__}({self.x!r})'

So as you can see, it also accepts formatting directives like !r; accordingly, we can print the current date and time in any format we'd like as concisely as:

>>> print(f'{dt.datetime.now():%d/%m/%Y %H:%M:%s}')
01/01/2024 12:00:00

And if we're dumping a bunch of values and would like to include their names, too, we can do so by adding = before the closing brace:

>>> x = 1
>>> y = 2
>>> print(f'{x=} {y=}')
x=1 y=2

All in all, the best of all worlds when it comes to string interpolation. f-strings did use to have some unfortunate limitations, like the fact we couldn't use a backslash inside their clause, nor the same quote (notice how I used ", " in this first example), nor a # if we had a multiline expression and wanted to add a comment — but in Python 3.12, their implementation was finally tidied up to address these cases, and now we can do:

>>> print(f'{'\n'.join(
...     ['line 1', 'line 2', 'line 3'] # this is a comment
... )}')
line 1
line 2
line 3

No problem. In fact, f-strings are so great that I'm only going to use them from now on; once you get used to them, all the other formatting schemes look ridiculously clumsy by comparison, and it took me enormous willpower to refrain from it until now.

String Methods

Beside all these syntactic superpowers, Python's string objects come equipped with almost 50 methods for us to make use of — and while I'm not going to cover all of them, I highly recommend going over and studying them, seeing as strings are such a basic concept that we deal with all the time.

For example, we've seen upper and lower; but there's also swapcase, and capitalize, and title (capitalizing every word). Or, if we'd like to check rather than change, there's isupper, islower and istitle, as well as isalpha (letters only), isnumeric (numbers only), isalphanum (letters and numbers only), isidentifier (letters, numbers and underscores only, not starting with a number) and so on.

We've also seen index and find — but did you know that there's also rindex and rfind, which search the string from the right? Or that all of them receive optional start and end parameters, to search the string only from and until a certain offset?

This whole arsenal of utility is not without its warts, however, so let me focus on those — not (only) because I'm a bitter person, but since they provide cute and illuminating lessons on software design, and I trust you to go over str's documentation yourself. Take startswith and endswith, for example: they test if a string has a certain prefix or suffix, which can be extremely useful. Still, sometimes we'd like to check if a string has one of several affixes, as in:

>>> if asset.endswith('.jpg') or asset.endswith('.png'):
...     print('asset is an image')

So it was decided these methods should support multiple arguments. How would you implement it? My inclination would be asset.endswith('.jpg', '.png'), using *args — but unfortunately, this turned out to be impossible, so we ended up with:

>>> if asset.endswith(('.jpg', '.png')):
...     print('asset is an image')

That is, passing in a tuple; better than nothing, but a bit awkward. Why couldn't they use *args? Because startswith and endswith already had some optional parameters, which would clash with *args, and removing them would break existing code. What parameters? start and end, which limit their search to particular offsets.

Except, if you think about it, it's quite bizarre. Checking if a string starts with some prefix from its middle, or ends with some suffix at its th character is not only useless, but extremely confusing — it doesn't really start or end with either, then, does it? Unfortunately, when designing these signatures, the core developers decided to match them to those of other functions, like index and find — a decision they later came to regret, when a much better use case (namely, supporting multiple affixes) presented itself. The moral of the story here, beside the fact that even seasoned programmers can make silly mistakes, is to beware of overdesigning our signatures: don't add parameters to your functions unless you're absolutely sure you need them, and definitely not for purely aesthetic purposes like making them symmetrical to other parts of the interface.

Another example comes from the method strip, which trims the string's leading and trailing whitespace, if any. This use case is actually well thought through: if we only want to trim leading whitespace, we have lstrip, and for trailing whitespace, rstrip; and if we want to trim something other than than whitepace, like dots, we can pass it in as an argument:

>>> print('A long time ago in a galaxy far, far away....'.strip('.'))
A long time ago in a galaxy far, far away

What if we want to strip a variety of characters, like all punctuation? We should have been able to pass in a collection thereof, like (',', '.', '!'), as *args or at least a tuple — but instead it was decided that since a string is an iterable, it will be more elegant to simply pass all of them in as a string: strip(',.!').

And it's shorter, sure; but if you didn't know any better, what would you think this call does? Or to put it differently, what would you think rstrip('.png') does? The answer: removes dots, js, ps and gs from the string's end indiscriminately, so you can expect wedding.png to end up as weddi. To ameliorate this situation, Python 3.9 introduced removeprefix and removesuffix — but the original point still stands: a quirky, unconventional idea ("let's pass a collection of unrelated characters as a string") is probably not the right choice for a core interface.

Let's do one more: replace lets us substitute one substring for another, and even accepts an optional parameter to limit the maximum number of substitutions. But if we want to replace multiple chunks — for example, remove all the whitespace — there's no way to do it except with repeated calls:

>>> cram = text.replace(' ', '').replace('\t', '').replace('\n', '')

And if we want to escale a string for HTML, we have to be extra careful with the order:

>>> html = text.replace('<', '&lt;')
...            .replace('>', '&gt;')
...            .replace('&', '&amp;') # actually, this should come first

So we don't end up replacing & (used by HTML as the primary escape character, like \ in Python) with its own escaped version. Wouldn't it be nice if we could do:

>>> cram = text.replace({' ': '', '\t': '', '\n': ''})
>>> html = text.replace({'<': '&lt;', '>': '&gt;', '&': '&amp;'})

Unfortunately, we can't. Luckily, there's another string method that looks promising: translate only works for replacing single characters, but this should suffice for our purposes:

>>> 'hello world\n'.translate({' ': '', '\t': '', '\n': ''})
'hello world\n'
>>> '<p>'.translate({'<': '&lt;', '>': '&gt;', '&': '&amp;'})
'<p>'

Only it doesn't work, because for some reason, translate expects the keys of its mapping to be code points, and can't be bothered to convert them for us:

>>> 'hello world\n'.translate({ord(' '): '', ord('\t'): '', ord('\n'): ''})
'helloworld'
>>> '<p>'.translate({ord('<'): '&lt;', ord('>'): '&gt;', ord('&'): '&amp;'})
'&lt;p&gt;'

If we only seek to replace one set of characters by another, the maketrans method can be of help in creating such a mapping: the first string is matched to the second, with an optional third one designated for removal:

>>> table = str.maketrans('abc', 'xyz', '.,!')
>>> table
{97: 120, 98: 121, 99: 122, 46: None, 44: None, 33: None}
>>> 'abc!'.translate(table)
'xyyz'

That's… something, I guess — but it's weird. We can't use it for HTML escaping, since it only allows one-to-one matching (although translate does work with one-to-many); and for whitespace removal, we'd have to invoke it like this:

>>> str.maketrans('', '', ' \t\n') # even Nones wouldn't work!
{32: None, 9: None, 10: None}

In other words, we have two whole methods as part of one of the most basic and widely used types (of which one is static, so there's really no reason to make it part of str); they only work for the specific use case of replacing single characters; and even then, we have to convert everything to integers first. If in our first example, the problem was that startswith and endswith did too much, and in the second that what strip did, it did too weirdly — then here, translate (or arguably, replace) does too little, and as a result we end up with an interface that's at once bloated and useless.

In the next chapter we'll see how these cases can be salvaged with regular expressions; for now, I thought they make for an interesting discussion. I've sung Python's praises for most of this book, so it's only fair to talk about its wrinkles, too — and besides, there's a lot we can learn from bad design: what not to do, and also, that even core developers can be wrong (and that's OK; maybe they'll remove it in Python 4).

Containers

Next, let's talk about container types: the sequences tuple, list, and set, and the mapping dict. Once again, I won't bore you with the details — you can read their full documentation yourself. Instead, I'll compare each to a lesser-known type that complements it — and while admittedly, those other types are not native per say, they're part of the standard library, and well worth knowing as such.

First, uples: there isn't much to say about them, especially when we use them in passing, like to auto-pack values as part of an assignment or a return-statement. But when we find ourselves using them more consistently, they're not ideal, because their representation and access pattern are quite opaque:

>>> user = 'alice', 25, True
>>> user
('alice', 25, True)
>>> if user[2]:
...     print(user[0], 'is an admin')
alice is an admin

So when we need to store a series of values in a predictable structure, it might be a good idea to use a more dedicated container. We can write a class for it, of course, but unless it has some additional methods, there's no need to burden our codebase with such tedious boilerplate — especially when we have namedtuples instead:

>>> import collections
>>> User = collections.namedtuple('User', 'name age is_admin')
>>> user = User('alice', 25, True)
>>> user
User(name='alice', age=25, is_admin=True)
>>> if user.is_admin:
...     print(user.name, 'is an admin')
alice is an admin

That's much better, isn't it? It's still a tuple, which we can iterate over or unpack — but where tuples come short, it provides a friendlier interface. As a tuple, though, we can't assign values to its fields — so if that's what we need:

>>> from dataclasses import dataclass

>>> @dataclass
... class User:
...     name: str
...     age: int
...     is_admin: bool

>>> user = User('alice', 25, True)
>>> user
User(name='alice', age=25, is_admin=True)
>>> if user.is_admin:
...     print(user.name, 'is an admin')
alice is an admin
>>> user.is_admin = False

Then we might want to look at dataclasses; yes, they require a class definition, but it's about as short and efficient as one can imagine, and we can toggle a bunch of its standard behaviors on or off (e.g. supporting comparison operators, or forcing its constructor to accept keyword arguments only).

What about lists? Internally, they're implemented like stacks, in what's known as the last-in, first-out paradigm, or LIFO: that's why its two most prominent methods are append and pop: the last element to be added is the first one to be removed. We can still override this behavior, injecting a value at its beginning with insert, and removing it by providing pop with an index:

>>> x = [2, 3]
>>> x.insert(0, 1)
>>> x
[1, 2, 3]
>>> x.pop(0)
1
>>> x
[2, 3]

But besides being inelegant, it's also quite inefficient, especially for large lists. So instead, the standard library provides as with a double-ended queue, or deque:

>>> x = collections.deque([2, 3])
>>> x.appendleft(1)
>>> x
deque([1, 2, 3])
>>> x.popleft()
1
>>> x
deque([2, 3])

It even has an optional parameter to limit its size, effectively cropping elements when it's exceeded:

>>> x = collections.deque([1, 2, 3], maxlen=3)
>>> x.append(4)
>>> x
deque([2, 3, 4], maxlen=3)
>>> x.appendleft(1)
>>> x
deque([1, 2, 3], maxlen=3)

And comes with handy methods like rotate:

>>> x.rotate(1)
>>> x
deque([3, 1, 2], maxlen=3)
>>> x.rotate(-1)
>>> x
deque([1, 2, 3], maxlen=3)

We could do all of the above with a regular list, of course:

>>> x, maxlen = [1, 2, 3], 3
>>> if len(x) > maxlen:
...     x.pop(0)
...     x.append(4)
>>> x
[2, 3, 4]
>>> x.insert(0, x.pop()) # rotate
>>> x
[4, 2, 3]

But deque is better equipped and specifically optimized for such scenarios — so if we find ourselves in need of something like this, it's good to know Python has our back.

Now, let's do sets. We've already seen how those can be merged, intersected and so on — but because their elements are unique, one piece of information they don't retain is how many times each of their elements was encountered. This is called a multiset, and in cases where such information is import, Python provides it via the Counter class:

>>> x = collections.Counter('abbccc')
>>> x
Counter({'c': 3, 'b': 2, 'a': 1})
>>> y = collections.Counter('bccddd')
>>> y
Counter({'d': 3, 'c': 2, 'b': 1})
>>> x + y
Counter({'c': 5, 'b': 3, 'd': 3, 'a': 1})
>>> x - y
Counter({'a': 1, 'b': 1, 'c': 1})

Finally, a fun fact about dictionaries. Normally, we have to be careful not to access keys that don't exist, lest they throw a KeyError; so if Counter doesn't suit our needs and we want to count occurrences ourselves, or maybe accumulate multiple elements under a single key, we'd have to do:

>>> d1 = {}
>>> # d1['x'] += 1 would throw an error, so we have to do:
>>> if 'x' not in d1:
...     d1['x'] = 0
... d1['x'] += 1

>>> d2 = {}
>>> # d2['x'].append(1) would throw an error, so we have to do:
>>> if 'x' not in d2:
...     d2['x'] = []
>>> d2['x'].append(1)

Which is hardly elegant. Maybe, if we know all our keys in advance, we might do:

>>> d1 = dict.fromkeys(['x', 'y', 'z'], value=0)
>>> d1['x'] += 1
>>> d1
{'x': 1, 'y': 0, 'z': 0}

But this wouldn't work for a list; adding value=[] would map every key to the same list object, which is probably not what we want. Interestingly, dictionaries have a setdefault method, which sets the key to a default value if it's missing and immediately returns it, letting us have:

>>> d2 = {}
>>> d2.setdefault('x', []).append(1)
>>> d2
{'x': [1]}
>>> d2.setdefault('x', []).append(2)
>>> d2
{'x': [1, 2]}

But then we just end up creating empty lists in vain whenever the key does exist. For these reasons and more, Python provides us with a defaultdict: a dictionary accepting a factory function, which it invokes whenever a missing key is accessed, prepopulating it with its result rather than throwing an error. And since int can be invoked to create the "default" integer 0, and list can be called to create a new, empty list, we can do:

>>> d1 = collections.defaultdict(int)
>>> d1['x'] += 1
>>> d1
{'x': 1}

>>> d2 = collections.defaultdict(list)
>>> d2['x'].append(1)
>>> d2
{'x': [1]}

That's why it's sometimes said that Python is a language that comes with "batteries included": the shear amount of functionality we get out of the box, whether through its syntax, native types or standard library, is just mind boggling. This is by no means an exhaustive overview (if anything, it feels kind of rushed) — but rather than compiling long lists of dry details, I wanted to whet your appetite, show you just how much there is to learn, and encourage you to go ahead and investigate it: it's what sets apart someone who just "knows Python" from someone who's truly mastered the language. Spoon feeding all this mundane information ends up rather boring, so I find it's better to leave you with an itch; or in the words of Antoine de Saint Exupéry, if you pardon the hubris: "if you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea."

Encoding Structures

One thing we are yet to cover, having discussed the binary representation of integers and strings, is how to larger, more sophisticated structures are actually encoded. What are the bits the make up a list, or a dictionary? Or even simpler, consider:

>>> x = 1
>>> y = 2
>>> s = 'hello world'

If we want to send this data over the wire, we have to put it together:

>>> data = x.to_bytes(1, 'big') + y.to_bytes(1, 'big') + s.encode()
>>> data
b'\x01\x02hello world'

In which case we can parse it on the other side:

>>> data = b'\x01\x02hello world'
>>> x = int.from_bytes(data[0:1], 'big')
>>> y = int.from_bytes(data[1:2], 'big')
>>> s = data[2:].decode()
>>> x, y, s
(1, 2, 'hello world')

But what if instead, we'd have one integer and two strings? How can we tell where one string begins, and another one ends?

It's the same problem we might have with lists and dictionaries: their items and keys we know how to encode, but it's the separation between them, especially given their variable size, that is the problem. To wit, Python provides us with the standard array module: if all our elements are of the same type, we can store them in an efficient manner and easily encode them to and from their binary representation like so:

>>> from array import array
>>> xs = array('b', [1, 2, 3])
>>> data = xs.tobytes()
>>> data
'\x01\x02\x03'
>>> # and later on:
>>> xs = array('b')
>>> xs.frombytes(data)
>>> xs
array('b', [1, 2, 3])

The b here stands for signed bytes, capable of holding an integer between -127 and 128. If we'd need a different range, we could use h for 2 bytes (also known as half a word), l for 4 bytes (a long, or double-word), and q for 8 bytes (quad-word). What happened to the regular word? Good question — it differs between different architectures: so on a 16-bit machine, 2 bytes would be a word, 4 a long word and 8 a quadword, while on a 32-bit machine, 2 would be half a word, 4 a word, and 8 a long word; Python went with the somewhat incoherent but definitely unambiguous mix.

Anyway, if we don't need to support negative integers, we give up the extra bit, otherwise dedicated to store the sign, and thus bump our range further: B will carry us from 0 to 255 (instead of 128), H up to 65535 (instead of 32768) and so on. These conventions let us serialize entire series at once, without having to specify the size for each element.

We also have f for 32-bit floats, and d for 64-bit double floats; note that here, because of the way floating-point representation works, it doesn't limit our range so much as our precision within it.

What about endianess? array just uses the one used by the machine that runs it; it's not actually intended for serialization, but rather for storing numbers in as tight a manner as possible. Besides, the fact all of them are of the same size is yet another constraint, not to mention it does nothing to solve the list's variable size problem. So Python provides us with another module, struct, that's more oriented towards serialization:

>>> import struct
>>> xs = [1, 2, 3]
>>> data = struct.pack('bbb', *xs)
>>> data
'\x01\x02\x03'
>>> # and later on:
>>> xs = struct.unpack('bbb', data)
>>> xs
[1, 2, 3]

Here, we can mix and match different sizes and types of values in the so called format argument, describing the pattern of our data; as well as set the endianess explicitly by prepending it with < (for little endian) or > (for big endian):

>>> struct.pack('>bhi', *xs)
b'\x01\x00\x02\x00\x00\x00\x03'

Encoding Variable Sizes

It also provides us with a straightforward way to solve our variable size problem: we can simply fix it as part of the format, adding this number before the formatting character to indicate how many times we want it to repeat:

>>> fmt = f'{len(xs)}b'
>>> data = struct.pack(fmt, *xs)
>>> data
'\x01\x02\x03'
>>> struct.unpack(fmt, data)
[1, 2, 3]

For strings, if we know that all of them will fit within, say, a 16-characters limit, we can use s, and it will pad them accordingly for us, using the null byte (\x00, or \0), which isn't a valid character anyway, and we can then strip away:

>>> x, s1, s2 = 1, 'one', 'two'
>>> fmt = 'b16s16s'
>>> data = struct.pack(fmt, x, s1.encode(), s2.encode())
>>> data
b'\x01one\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00two\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
>>> # and later on:
>>> x, b1, b2 = struct.unpack(fmt, data)
>>> s1 = b1.strip(b'\0').decode()
>>> s2 = b2.strip(b'\0').decode()
>>> x, s1, s2
(1, 'one', 'two')

As you can see, it's very wasteful, and doesn't really work for strings longer than 16 characters (unless we make it even more wasteful), but it's something. A more interesting approach, coming from C, would be to notice that if the nil byte can be used for padding, then it can be used as a terminator, effectively playing the role of a stop sign that tells us "your string ends here". It isn't supported by struct, but we could do:

>>> data = x.to_bytes(1, 'big') + s1.encode() + b'\0' + s2.encode() + b'\0'
>>> data
b'\x01one\x00two\x00'

But then, decoding it with slicing becomes a pain, since we have to repeatedly find the next null byte and crop our data until then. So instead, we can use Python's io.BytesIO class to consume the bytestream sequentially, much like we read a file. We just need:

>>> def read_string(stream, terminator=b'\0'):
...     output = []
...     while True:
...         byte = stream.read(1)
...         if not byte or byte == terminator:
...             break
...         output.append(byte)
...     return b''.join(output).decode()

And we can do:

>>> import io
>>> stream = io.BytesIO(data)
>>> x = int.from_bytes(stream.read(1), 'big')
>>> s1 = read_string(stream)
>>> s2 = read_string(stream)
>>> x, s1, s2
(1, 'one', 'two')

Still, it's pretty ineffective, reading one character at a time; and it doesn't solve our problem for lists, for which there is no terminator. If our list contains structures of a specified size, we can envision an all-zero sequence of the appropriate length to mean the end — and this is sometimes done, like in the C code we saw when we implemented an .so module:

static PyMethodDef ExampleMethods[] = {
    {"hello", hello, METH_VARARGS, ""},
    {NULL, NULL, 0, NULL} // right here
};

But the whole story is pretty messy at this point. So instead, we can borrow an idea from Pascal strings: prepending variable-sized data with its size. That is:

>>> fmt = f'bb{len(s1)}sb{len(s2)}s'
>>> data = struct.pack(fmt, x, len(s1), s1.encode(), len(s2), s2.encode())
>>> data
b'\x01\x03one\x03two'

And then:

>>> stream = io.BytesIO(data)
>>> x = int.from_bytes(stream.read(1), 'big')
>>> s1_size = int.from_bytes(stream.read(1), 'big')
>>> s1 = stream.read(s1_size).decode()
>>> s2_size = int.from_bytes(stream.read(1), 'big')
>>> s2 = stream.read(s2_size).decode()
>>> x, s1, s2
(1, 'one', 'two')

Still a bit clunky, but at least this is efficient (and if our strings exceed 255 characters, we can use h or l for their size). The same principle can be applied to lists, where we prepend their content with their length. For dictionaries, we'd have to prepend the whole dictionary with its length, and (say, if its keys are strings and its values are integers) each of its items with the key length, then the key, and then the value. It's a mess, as binary representations often are, but at least it's a coherent mess, and we're developing an intuition on how to think in binary — which I find to be sorely lacking, especially among people who jumped straight into high-level languages like Python or Javascript, without ever cutting their teeth on something as low-level as C.

Complex Serialization

The good news are that this logic can be neatly encapsulated in a dedicated library — and indeed, serialization frameworks that work just like that abound, providing a tidy interface to dump and load our data to and from bytes. Take BSON, for example: a binary version of JSON, courtesy of MongoDB:

>>> import bson # 3rd party library available with pip install bson
>>> bson.dumps({'x': 1, 'y': 'hello', 'z': [1, 2, 3]})
b'6\x00\x00\x00\x10x\x00\x01\x00\x00\x00\x02y\x00\x06\x00\x00\x00hello\x00
\x04z\x00\x1a\x00\x00\x00\x100\x00\x01\x00\x00\x00\x101\x00\x02\x00\x00\x00
\x102\x00\x03\x00\x00\x00\x00\x00'

Let me help you decipher this one:

6\x00\x00\x00 # dict size: ord('6') = 0x36 (54 bytes)
  \x10  # type: 0x10 stands for int32 (l)
  x\x00 # key:  "x" (null-terminated)
    \x01\x00\x00\x00 # value: 1 (4 bytes in little endian)
  \x02  # type: 0x02 stands for string
  y\x00 # key:  "y" (null-terminated)
    \x06\x00\x00\x00 # string length: 0x06 (including nil byte)
    hello\x00        # string value: hello (null-terminated)
  \x04  # type: 0x04 stands for list (which is a dict with numeric keys)
  z\x00 # key:  "z"
    \x1a\x00\x00\x00 # dict/list size: 0x1a (26 bytes)
    \x10  # type: 0x10 stands for int32 (l)
    0\x00 # key:  "0" (null-terminated)
      \x01\x00\x00\x00 # value: 1 (4 bytes in little endian)
    \x10  # type: 0x10 stands for int32 (l)
    1\x00 # key:  "1" (null-terminated)
      \x02\x00\x00\x00 # value: 2 (4 bytes in little endian)
    \x10  # type: 0x10 stands for int32 (l)
    2\x00 # key:  "2" (null-terminated)
      \x03\x00\x00\x00 # value: 3 (4 bytes in little endian)
    \x00 # null byte terminates our dict/list
\x00 # null byte terminates our dict

As you can see, it's much more sophisticated than anything we've seen so far: using a combination of prefixed sizes with bull terminators, and adding bytes to indicate which type of data is to follow. It's not that you're expected to remember this format by heart (I certainly don't) — but by reverse engineering it together I wanted to show how arbitrarily complex data is actually stored on the disk, or sent over the wire.

To give one last example, consider images: how is it even possible that a sequence of bits is translated into a two-dimensional, colorful picture? There are many formats (that's what the extensions .bmp, .jpg or .png stand for) — but essentially, here's a sketch: first we store two (say, 4-byte) integers, w and h, donating that are image is w pixels by h pixels in size; and that we follow by w*h triplets of bytes, (r, g, b), donating the red, green and blue value combination of each pixel. A program that displays images, then, knows to decode this format, laying out the pixels and coloring them on our screen accordingly — and suddenly, something that might've seemed quite impossible without our binary intuition turns out to be quite straightforward.

The same holds true for audio, video, and other media: they all have their own organizing logic, abstractions and formats, but in the end — they're all just elaborate streams of bits and bytes. In fact, even code itself — that is, the instructions that describe how such bits should be manipulated — is itself serialized to and stored as bits; computer programs are, after all, also files to the filesystem.

As meta as this sounds, this concept is as old as computers themselves: soon after Alan Turing developed the fundamental abstraction of computer science, the Turing Machine on which all our computational models are based, he came up with the Universal Turing Machine: a computer capable of emulating other computers — effectively "running programs". This concept was then solidified in what's known as von Neumann's architecture, where input and output devices (like keyboards and screens) are mediated by a central processing unit and a memory unit, capable of storing and executing code as regular data. In Python, we've seen this concept in action when we discussed the Python's code object, and specifically its co_code attribute:

>>> def add(x, y):
...     return x + y
>>> add.__code__.co_code
b'\x97\x00|\x00|\x01z\x00\x00\x00S\x00'

That is, a bytestream encapsulating the idea of addition itself. And in fact, Python provides us with the pickle module, capable of encoding and decoding almost any Python object without us having to do anything at all:

>>> import pickle
>>> data = pickle.dumps({'x': 1, 'y': 'hello', 'z': [1, 2, 3]})
>>> data
b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.'
>>> pickle.loads(data)
{'x': 1, 'y': 'hello', 'z': [1, 2, 3]}

>>> class A:
...     def f(self):
...         return 1
>>> a = A()
>>> data = pickle.dumps(a)
>>> data
b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.'
>>> a = pickle.loads(data)
>>> a.f()
1

This format is highly specific to Python, though; so if we want to share data among programs that aren't necessarily written in Python themselves, more agnostic forms of encoding, like BSON, are still needed. What's more, pickle is famously insecure, so you should never load data from an untrusted source: as part of its deserialization, in order to reconstruct our objects fully, it indiscriminately runs some of their code, so:

>>> class A:
...     def __new__(cls):
...         print('hacked!')
...         return super().__new__(cls)
>>> a = A()
hacked!
>>> data = pickle.dumps(a)
>>> a = pickle.loads(data)
hacked!