The Standard Library: Datetimes, Paths and Regular Expressions

We've talked about numbers, string and bytes, lists and dictionaries; but another absolutely fundamental data type in software is the datetime, denoting a particular — well, date and time.

Epoch Time

This wasn't always considered important; in a lot of older programming languages, and operating systems in general, time is simply stored as a number; and in Python we can glimpse the same functionality here:

>>> import time
>>> time.time()
1704110400.0

This value is called Epoch time, or sometimes Unix timestamp, since this is where the idea originated: it represents the number of seconds elapsed from the Unix Epoch, a somewhat dramatic nickname given to midnight of January 1st, 1970, when the Unix operating system was first released (it actually happened in Novermber of 1971, but who'd want to count seconds from that date).

To convert this to an actually sensible value, we could count how many days, hours and minutes have passed since then, accounting for leap years, etc. — but seeing as it's such a common need, there are quite a few libraries to do it for us. The standard one is:

>>> t = time.gmtime(1704110400)
>>> t
time.struct_time(tm_year=2024, tm_mon=1, tm_mday=1, tm_hour=12, tm_min=0, tm_sec=0, tm_wday=0, tm_yday=1, tm_isdst=0)

Which is just a wrapper around similar functions in C. It, too, is not very readable, so a further conversion usually follows:

>>> time.strftime('%H:%M:%S %d/%m/%Y ', t)
12:00:00 01/01/2024

Which are all standard C formatting codes, worth familiarizing yourself with. For example, %y gives you the short version of the year (24), %a and %A the short or full day name (it was Monday), %b and %B the short or full month name (January) — and if you don't care about specifics, %c for a locale appropriate representation (e.g. the first of May would be 01/05 in most of the world, but 05/01 in the US). On top of that, Python also supports %f (sorely lacking in C), to add resolution in microseconds.

Datetime

In a modern language like Python, however, we can do better than that; enter the datetime module. We've seen it before, but there's plenty more to say about it, so let's dive in. Naturally, we can create specific dates:

>>> import datetime as dt
>>> # accepts (year, month, day, [hour, [minute, [second, [microsecond]]]])
>>> d = dt.datetime(2000, 5, 1, 15, 30)
>>> d
datetime.datetime(2000, 5, 1, 15, 30)

Or, more commonly, we use the factory method now to get the current time:

>>> d = dt.datetime.now()
>>> d
datetime.datetime(2024, 1, 1, 12, 0) # pretend this is today

Once we've got it, it has all the attributes we might expect: year, month, day, hour, minute, second and microsecond. Other values that can be inferred from that, like what weekday this happens to be, are available via additional methods:

>>> d.weekday()
0 # Monday

Besides being a much more sensible and convenient structure than a number, this object also alleviates a significant constraint of unix timestamps: since they were conceived as 32-bit integers counting seconds from Januarty 1st, 1970, not only are they unable to describe dates prior to that, but they will actually max out and overflow on Januarty 19th, 2038 (it's known as the 2038 problem, and expected to cause much more trouble than Y2K). With datetimes, the range is much less limited: any date and time from 1 AD and until 9999 are supported, which should be enough for our purposes.

Datetimes do have a couple of caveats, though. The first one is that they're immutable: contrary to what you might expect given such a friendly structure, once created, we can't set or edit their attributes. This is done to ensure datetimes are hashable, so we can use them as keys in a dictionary; if they were to change, their hash values would change as well, and using them for mapping wouldn't be consistent. So, it was decided to make them unchangeable — but we can always derive a different date from an existing one using the replace method:

>>> d.replace(month=5, hour=15, minute=30)
datetime.datetime(2024, 5, 1, 15, 30)

The second issue is that while "the number of seconds" is easy enough to do math with, it's not nearly as straightforward to handle a structure. If we'd want to jump forwards or backwards by one day with a Unix timestamp, we can simply add or subtract 60*60*24, right? With datetimes, while we can replace the day to an its next or previous value, we'd have to handle overflows (i.e. the day going below 0 or above 28-31, depending on the month), which can also propagate (i.e. the month going below 0 or above 12).

Date Math

Luckily, Python comes equipped with a way to handle such calculations — and it turns out to be more elegant and robust than counting seconds, albeit introducing a slightly more confusing concept: the time delta.

You see, dates don't actually adhere to "normal" math: it doesn't really make sense to add two dates together, despite the fact that technically, with Unix timestamps, we could do that. What is January 1st plus January 2nd, though? Even if we had some notion of what this might mean, different years would produce different results, because the accumulating seconds pay no heed to our grouping them in days and months. What we can add to a date, though, is a certain number of days, hours, minutes or seconds — and the timedelta class captures exactly this concept, handling all the irregularities, leap years and other nuances for us:

>>> d + dt.timedelta(days=121, hours=3.5)
datetime.datetime(2024, 5, 1, 15, 30)

Note that one operation that does make sense for datetimes is subtraction: we can ask what is the difference between January 2nd and January 1st, and the result should be "one day" regardless of the year. In Python, this is results in a timedelta:

>>> td = dt.datetime(2024, 5, 1, 15, 30) - dt.datetime(2024, 1, 1, 12, 0)
>>> td
datetime.timedelta(days=121, seconds=12600)

And notice how its internal representation actually deals with seconds, even though we described it as 3.5 hours, and the same could be described as 3 hours and 30 minutes. Internally, timedeltas only reasons in days, seconds, and microseconds, which strikes a good balance between precision and usability; still, they accept hours, minutes and milliseconds to them, normalizing those values accordingly. If we ever want to merge it all into a single number, however, we can do:

>>> td.total_seconds()
10467000.0

As opposed to datetimes, timedeltas can be added and subtracted from each other, multiplied and divided by scalars (that is, integers or floats) — and even divided and modded (as in, modulo) by each other, in which case a quotient or remainder of their total number of seconds is computed. Lastly, they can also be negative, which might be confusing at times:

>>> dt.datetime(2024, 1, 1, 0, 0) + dt.timedelta(seconds=-1)
datetime.datetime(2023, 12, 31, 23, 59, 59)

But if you think about it, that's exactly the date and time one second before.

Date Conversions

In Python, working with datetimes is definitely the way to go — but when interacting with other programs or users, we have to convert them to and from more standard or readable formats. To deal turn a datetime into a timestamp, for example, we do:

>>> d.timestamp()
1704106800.0

And to turn a timestamp into a datetime, this factory method comes in handy:

>>> dt.datetime.fromtimestamp(1704106800)
datetime.datetime(2024, 1, 1, 12, 0)

To a convert a datetime into a string, we have the strftime method ("f" stands for "format"):

>>> s = d.strftime('%H:%M:%S %d/%m/%Y')
>>> s
'12:00:00 01/01/2024'

And to convert it back, the factory method strptime (where "p" is for "parse") is used:

>>> dt.datetime.strptime(s, '%H:%M:%S %d/%m/%Y')
datetime.datetime(2024, 1, 1, 12, 0)

If we don't care for the exact layout of this string, and only want to convert datetimes to and from strings consistently, we can use the ISO 8601 standard for date strings:

>>> s = d.isoformat()
>>> s
'2024-01-01T12:00:00'
>>> dt.datetime.fromisoformat(s)
datetime.datetime(2024, 1, 1, 12, 0)

And there are some other formats that are less frequently encountered, but just so you know:

>>> cal = d.isocalendar() # counts dates in weeks
>>> cal
datetime.IsoCalendarDate(year=2024, week=1, weekday=1)
>>> dt.datetime.fromisocalendar(*cal)
datetime.datetime(2024, 1, 1, 0, 0) # note the time resolution is lost

>>> o = d.toordinal() # counts dates in days since January 1st, 1 AD
>>> o
738886
>>> dt.datetime.fromordinal(o)
datetime.datetime(2024, 1, 1, 0, 0) # note the time resolution is lost

And some standard C versions:

>>> d.timetuple() # the same as what we got from gmtime()
time.struct_time(tm_year=2024, tm_mon=1, tm_mday=1, tm_hour=12, tm_min=0, tm_sec=0, tm_wday=0, tm_yday=1, tm_isdst=-1)
>>> d.ctime()     # the date string format we see on Unix
Mon Jan  1 12:00:00 2024

As an aside, we can also break datetimes down into separate date and time objects (which we can also create separately, if we want), or merge such objects together:

>>> d.date()
datetime.date(2024, 1, 1)
>>> d.time()
datetime.time(12, 0)

>>> d = dt.date.today()
>>> t = dt.time(12)
>>> dt.datetime.combine(d, t)
datetime.datetime(2024, 1, 1, 12, 0)

Timezones and Daylight Saving

If this was all there is to say about datetimes, I don't imagine there'd be so many problems with them. Sure, some months have 30 days and some 31, and there's an occasional leap year to account for — but honestly, all these problems are solved.

Unfortunately, there's a much bigger problem: none of the datetimes we've seen so far describe an actual moment in time in any meaningful or consistent way. I'm writing this in Barcelona, and pretending its noon at January 1st — but this means it's 6 AM in New York and 8 PM in Tokyo: in most parts of the word, my timestamp is either in the future or in the past.

So, if I want to coordinating a call at noon tomorrow, I can't just send a datetime over: one part will attend it 6 hours late, the other 8 hours early. The reason for that, of course, is that the earth is round — so the position of the sun in the sky, the so called solar time, is different in different parts of the planet.

In the past, this wasn't much of a problem: different places simply observed different solar times (if they observed time at all), and anyone passing between them would do it at such a speed that those differences were immaterial: New York and Tokyo are half a world apart, so 14 hours are quite negligible when it would take a year to sail from one to another.

As the world became more globalized, however, this started becoming a problem. In 1675, the Royal Observatory in Greenwich, England, established what was effectively the first timezone: Greenwich Mean Time, or GMT, aimed at helping mariners navigate and coordinate at sea. Global adoption, however, was still slow: throughout the 19th century, even with the advent of trains, which could cross distances significantly faster, different railroads had their own standard time, so coordination was challenging. By the middle of the 20th century, however, most countries have aligned to a global standard, centered around the same, initial GMT, now known as Universal Coordinated Time, or UTC (the acronym itself is in French).

Everybody didn't just sync their clocks, though: it would be silly for Americans to wake up after noon, or for Japanese to have dinner at that hour. Rather, the way this was done is by shifting the local time slightly, to match fixed offsets from UTC. Initial suggestions were to do so by longitudinal lines, but it turned out to be more practical to skew this logic to account for borders, trying to keep the same time across the country (to the degree possible: Russia is so huge it spans 11 timezones). Barcelona, in this case, is aligned to the Madrid's timezone, UTC+1, meaning the "abstract", universal time on Planet Earth at noon my time, is actually 11 AM. New York, to the west of of the imaginary reference line known as the Greenwhich Meridian, is UTC-5, and Tokyo, to the East, is UTC+9. Generally, the more west you go, the more negative the offset, and the more east, the more positive it is, capping at -12 and +14, where the zones meet in the middle of the Pacific Ocean (although, like I said, it's not exactly longitude-based: in fact, Madrid is 3 degrees west of Greenwhich; days in Spain just start and end later).

To make it even more confusing, there's also daylight saving time, or DST. This practice is actually much older than timezones: even ancient societies would adjust their timekeeping practices, like the waterclocks used in Egypt or Rome, to account for the varying day length in different seasons, due to the Earth's axial tilt. They would do so largely for religious purposes: the winter and summer Solstice happen at the shortest and longest days of the year, for example, and various rituals needed to be scheduled accordingly. Today, it's mostly done to save energy: adjusting the clock to have more daylight means less artificial lighting is necessary, but it also makes our time calculations trickier, since different countries move their clocks at different times.

The mnemonic here, to help you remember when it happens, is "spring forward and fall back": in Spain, for example, we move the clock 1 hour forward at 2 AM on the last Sunday of March (that is, in the spring), and 1 hour back at 3 AM on the last Sunday of October (that is, fall, or autumn). It's a more minute difference than timezones, but it still matters: it effectively means that spain is sometimes UTC+1, but sometimes UTC+2, and this shift is not coordinated globally. In New York, for example, they move their clocks on the second Sunday of March and the first Sunday of November, and in Japan they don't move their clocks at all — so even after we synchronize our timezones, people might still be an hour early or an hour late to our international call.

TZInfo

Having set the motivation, I have some good news: all these quirky anomalies are managed automatically by the operating system — it's just that for some reason, in Python, they're not included by default. When creating a datetime, we might assume, quite reasonably, that it's aligned to our local time — but in fact, it's what known as a naive datetime: one without timezone and DST information at all. To turn it into an aware timezone, we need to pass in a TZinfo object. Prior to Python 3.9, this required constructing such an object ourself, which was a serious headache:

>>> # Madrid's timezone is also known as Central European Time, or CET
>>> class CET(dt.tzinfo):
...     def __init__(self):
...         self.std_offset = dt.timedelta(hours=1)
...         self.dst_offset = dt.timedelta(hours=2)
...         self._dst_range = {}
... 
...     def utcoffset(self, d):
...         return self.dst_offset if self._is_dst(dt) else self.std_offset
... 
...     def dst(self, dt):
...         if not self._is_dst(dt):
...             return dt.timedelta(0)
...         return self.dst_offset - self.std_offset
... 
...     def tzname(self, dt):
...         return 'CEST' if self._is_dst(dt) else 'CET'
... 
...     def _is_dst(self, d):
...         if d.year not in self._dst_range:
...             self._dst_range[dt.year] = list(self._last_sundays(year))
...         dst_start, dst_end = self._dst_range[d.year]
...         return dst_start <= d.replace(tzinfo=None) < dst_end
... 
...     def _last_sundays(self, year):
...         for month in [3, 10]: # find last sundays of March and October
...             last_day = dt.datetime(year, month + 1, 1) - \
...                        dt.timedelta(days=1)
...             yield last_day - dt.timedelta(days=last_day.weekday() + 1)

>>> d = dt.datetime(2024, 1, 1, 12, 0, tzinfo=CET)

And prior to 3.2, there was no way to apply it to the current time, so we'd have to do:

>>> d = dt.datetime.now().replace(tzinfo=CET)

Thankfully, they added the optional tz argument, so we can do dt.datetime.now(CET); and there were 3rd party libraries like pytz that predefined all these cumbersome structures:

>>> import pytz
>>> dt.datetime.now(pytz.timezone('Europe/Madrid'))
datetime.datetime(2024, 1, 1, 12, 0,
                  tzinfo=<DstTzInfo 'Europe/Madrid' LMT+0:09:00 STD>)

Even more thankfully, this is now part of the standard library via zoneinfo, which acquires all the necessary information from the operating system:

>>> import zoneinfo
>>> dt.datetime.now(zoneinfo.ZoneInfo('Europe/Madrid'))
datetime.datetime(2024, 1, 1, 12, 0,
                  tzinfo=zoneinfo.ZoneInfo(key='Europe/Madrid')>

And in Python 3.3, they made it possible to equip datetimes with whatever the local timezone is much more easily, by calling astimezone with no argument:

>>> d = dt.datetime.now().astimezone()
>>> d
datetime.datetime(2024, 1, 1, 12, 0,
         tzinfo=datetime.timezone(datetime.timedelta(seconds=7200), 'CEST')>

These tzinfo objects (whether DstTzInfo, ZoneInfo or timezone) all implement the same interface: namely, tzname to get the timezone name and utcoffset to get the time delta of its offset from UTC (as well as dst, to gets its DST offset, but usually we don't need it, since it's already included in utcoffset). Thus we can do:

>>> (d + d.tzinfo.utcoffset(d)).replace(tzinfo=dt.UTC)
datetime.datetime(2024, 1, 1, 11, 0, tzinfo=datetime.timezone.utc>)

To get the abstract UTC time and work from there, or further adjust it to some other timezone. Incidentally, we don't need to, since this was the original purpose of astimezone (like it name implies):

>>> d.astimezone(dt.UTC)
datetime.datetime(2024, 1, 1, 11, 0, tzinfo=datetime.timezone.utc>)

UTC Confusion

I imagine it's a bit mind numbing to read some ten pages about datetimes — but trust me, it's important: they're extremely common, and extremely useful, and yet inherently complicated.

The way people used to handle this with Unix timestamps is to just work in UTC, always. In fact, it's part of the Epoch time definition: saying "the number of seconds since midnight of January 1st, 1970" would not be enough without specifying in what timezone this midnight was observed. And since a single number can't carry this additional information, it's should be assumed to be in UTC. When I called it before, and got 1704110400 — that's actually the number of seconds that passed until noon of January 1st, 2024 if I was standing in Greenwich observatory. In Barcelona, it's already 1 PM by this time. So when I was talking about conversions, you might've noticed:

>>> d = dt.datetime(2024, 1, 1, 12, 0)
>>> d.timestamp()
1704106800.0 # not 1704110400
>>> dt.datetime.fromtimestamp(1704106800)
datetime.datetime(2024, 1, 1, 12, 0)
>>> dt.datetime.fromtimestamp(1704110400)
datetime.datetime(2024, 1, 1, 13, 0)

And herein lies the problem. On one hand, Python datetimes are definitely more powerful than Unix timestamps, and come with a lot of convenient functionality that makes our life easier. On the other hand, it also means we have to take care of timezones properly, because for some unfortunate reason, this is not the default behavior in Python.

What's worse, it took a while to iron out how exactly to do it, and you will find remnants of this process in the standard library, which only compounds the confusion about UTC. In the previous example, we created a naive datetime object, and when we asked for its timestamp, Python assumed it's in UTC, even though we were most likely reasoning in our local time. This is even worse when we use now, because then the result is definitely in local time, and yet its timestamp conversion doesn't take this into account — so it ends up producing the wrong result. If we really meant UTC, what we should have done was:

>>> d = dt.datetime(2024, 1, 1, 12, 0, tzinfo=dt.UTC)
>>> d.timestamp()
1704110400.0 # correct timestamp

And then, to convert it back:

>>> d = dt.datetime.utcfromtimestamp(1704110400)
>>> d
datetime.datetime(2024, 1, 1, 12, 0) # correct, but naive, so:
>>> d = d.replace(tzinfo=dt.UTC)
>>> d
datetime.datetime(2024, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)

If we'd use fromtimestamp, Python would adjust it to our local time, in which case we'd have to add our local timezone and then shift the result:

>>> d = dt.datetime.fromtimestamp(1704110400)
>>> d
datetime.datetime(2024, 1, 1, 13, 0) # local time, and naive, so:
>>> d = d.astimezone().astimezone(dt.UTC)
>>> d
datetime.datetime(2024, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)

And if we were using now (which, remember, at 12 is actually 11), we would need to do:

>>> d = dt.datetime.utcnow()
>>> d
datetime.datetime(2024, 1, 1, 11, 0) # correct, but naive, so:
>>> d = d.replace(tzinfo=dt.UTC)

Or, ever since we got the tz argument:

>>> d = dt.datetime.now(dt.UTC)
>>> d
datetime.datetime(2024, 1, 1, 11, 0, tzinfo=datetime.timezone.utc)

A Simpler Approach

I don't know about you, but I find it pretty confusing: sometimes we need to replace the timestamp, sometimes to shift it with astimezone, sometimes to call astimezone twice; and besides, if we're always working with UTC, displaying the result will show a time different to our local time, which can be counterintuitive, unless we're careful to localize it every time we dump it into a log or the interpreter's output.

This method, of converting everything to UTC as soon as possible, and converting it back to local time at the last moment, is the safest route with Unix timestamps; but in Python, we needn't limit ourself to this approach. When we do get one, I recommend converting it like this:

>>> d = dt.datetime.fromtimestamp(1704110400).astimezone()
>>> d
datetime.datetime(2024, 1, 1, 13, 0, 
         tzinfo=datetime.timezone(datetime.timedelta(seconds=3600), 'CET'))

Because from our point of view, it actually does describe 1 PM; and when we want to generate one ourselves, I recommend doing:

>>> d = dt.datetime(2024, 1, 1, 12, 0).astimezone()
>>> d.astimezone(dt.UTC).timestamp()
1704106800

For everything else, I recommend using the local time, properly timezoned:

>>> d = dt.datetime.now().astimezone()
>>> d
datetime.datetime(2024, 1, 1, 12, 0, 
         tzinfo=datetime.timezone(datetime.timedelta(seconds=3600), 'CET'))

This way, all you need to remember is adding astimezone whenever you create a datetime, and using it once more, to shift to UTC, before converting it into a timestamp. And unless you absolutely have too, don't use a timestamp at all — datetimes can be serialized just fine into strings instead of numbers:

>>> d.isoformat()
'2024-01-01T12:00:00+01:00'

And as you can see, these strings can include the datetime information: +01:00, available as %z if we're formatting them ourselves. When we read it back:

>>> d = dt.datetime.fromisoformat('2024-01-01T12:00:00+01:00')
datetime.datetime(2024, 1, 1, 12, 0, 
         tzinfo=datetime.timezone(datetime.timedelta(seconds=3600)))

We get everything exactly as it should be, maybe except the timezone name — but that's usually quite immaterial anyway.

The rest of the time, timezone-aware datetimes work marvelously: quick, what's later, 9 AM in New York or 9 PM in Tokyo?

>>> from zoneinfo import ZoneInfo
>>> d1 = dt.datetime(2024, 1, 1, 9,  0, tzinfo=ZoneInfo('America/New_York'))
>>> d2 = dt.datetime(2024, 1, 1, 21, 0, tzinfo=ZoneInfo('Asia/Tokyo'))
>>> d1 < d2
False
>>> d1 > d2
True

And as a bonus, they refuse to work with naive datetimes, so we're quite safe there:

>>> d1 > dt.datetime(2024, 1, 1, 12, 0)
TypeError: can't compare offset-naive and offset-aware datetimes

Path Objects

Another welcome addition to the standard library, ever since version 3.4, is the pathlib library and its Path object. Until then, working with filesystem paths meant working with string, although we had the os.path module to manipulate them more consistently, like calling os.path.join rather than adding slashes (on Unix) and backslashes (on Windows) ourselves.

It worked well enough, and I've used it throughout the book until now, just like with f-strings, so that you recognize it if you see it in somebody else's code. For our own code, though, there's no reason to use anything but: especially in a language as object-oriented as Python, suddenly switching to such procedural shenanigans as os.path.dirname, os.path.basename or os.path.splitext looks rather jarring; especially when we can do:

>>> import pathlib
>>> p = pathlib.Path('/home/user/file.txt')
>>> p.parent
pathlib.Path('/home/user')
>>> p.basename()
'file.txt'
>>> p.stem()
'file'
>>> p.suffix
'.txt'

We've already seen how these path objects cleverly overload the true division operator to allow for proper concatenation:

>>> p = pathlib.Path('/home')
>>> p / 'user'
pathlib.Path('/home/user')
>>> p / 'user' / 'file.txt'
pathlib.Path('/home/user/file.txt')

And if you recall our recursive directory scans, we can now implement is as easily as:

>>> def scan(directory):
...     directory = pathlib.Path(directory)
...     for path in directory.iterdir():
...         yield path
...         if path.is_dir():
...             yield from scan(path)

>>> for path in scan('dir'):
...     print(path)
dir/subdir1
dir/subdir1/file1
dir/subdir2
dir/subdir2/file2

With the added benefit that the results yielded are path objects themselves (it's just that when we print them, they look like strings).

Path Tips

As usually, I won't go over the documentation in detail, but rather draw attention to some of its lesser known features, or use cases you might encounter often.

First of all, notice how the scan function immediately wrapped its argument in a path object: this is done to support strings (so we can pass in 'dir'), and doesn't come at the expense of paths, as those simply stay the same in this situation (so we could do scan(pathlib.Path('dir')) as well, or scan(directory) where directory is a path object). This idiom, of either-string-or-path, spans across the whole standard library, and it's well supported: we can pass path objects to functions that previously accepts strings only, like open(path), os.path.join(path1, path2) and so on. This is done mostly to ease the transition and allow mixing the two if necessary; on our own, using path.open() or path1 / path2 instead is more elegant.

To that end, if we got used to working with strings, there are quite a few habits to relearn. Instead of reading a file like this:

>>> data = open(path).read()
>>> # or, more responsibly:
>>> with open(path) as f:
...     data = f.read()

We can do this:

>>> data = path.read_text()  # opens the file with 'r'
>>> data = path.read_bytes() # opens the file with 'rb'

And instead of writing a file like this:

>>> open(path, 'w').write(data)
>>> # or, more responsibly:
>>> with open(path, 'w') as f:
...     f.write(data)

We have:

>>> path.write_text(data)  # opens the file with 'w'
>>> path.write_bytes(data) # opens the file with 'wb'

As for path changes, its worth knowing that besides the more obvious parent, basename (the name with the extension), stem (the name only) and suffix, we also have parents (for progressively higher directory), suffixes (if there is a compound extension, like archive.tar.gz) and even parts, if we want to split it by the delimiter. However, if what we're trying to do is change the file's basename, we'll have better luck with:

>>> p = pathlib.Path('/home/user/file.txt')
>>> p.with_stem('file2')
pathlib.Path('/home/user/file2.txt')
>>> p.with_suffix('.json')
pathlib.Path('/home/user/file.json')
>>> p.with_name('file2.json')
pathlib.Path('/home/user/file2.json')

As for directories, it's worth knowing that:

>>> directory = pathlib.Path('/home/user/dir')
>>> directory.mkdir()

Might raise an error: FileExistsError if the directory already exists, and FileNotFoundError if its parent directory (in this case, /home/user) doesn't. To handle such concerns with grace, we can do:

>>> directory.mkdir(exists_ok=True, parents=True)

Where the first parameter tells Python to ignore the case where it's already present, and the second to create all the parents necessary to make it happen. As for scanning — beside iterdir (similar to os.listdir), we also have glob, which lets us use Unix shell patterns to quickly hone in on the right files: ? lets us match any (one) character, [] matches a character from a specific range (like [0-9]), and * is a wildcard that matches any number of any characters, letting us have something like *.txt to match all the text files, or *_backup.* to match all the entries whose names end with _backup. We can even make this search recursive if we use rglob:

>>> for path in directory.rglob('file*'):
...     print(path)
dir/subdir/file1
dir/subdir/file2

Or see if a given path matches some pattern with path.match('*.txt').

Accessing Nearbly Files

One thing that scripts often need to do, and packages with assets might need as well, is to access some files located next to the Python file being executed (at least, in some predictable, relative fashion). Suppose, for example, that there are some data files in JSON format that are sitting in a sibling subdirectory, like so:

script.py
data/
    data1.json
    data2.json

To open then, script.py could use the full path (e.g. /home/user/data) — but that would break if we ever place it elsewhere; a better idiom would be to start with its own __file__ (guaranteed to point to wherever it is), and work from there:

# script.py

import json

data_directory = pathlib.Path(__file__).parent() / 'data'

# and later...

data_path = data_directory / 'data1.json'
data = json.loads(data_path.read_text())

# or if we want to read all of them:

datas = [json.loads(path.read_text()) for path in data_directory.iterdir()]

Alternative, if we'd like to store something like the user preferences or configurations, we can decide to do so in their home directory (in my case, /home/user), under a special subdirectory like .config, in a file with the same name like our module (in this case, the rather generic script). Again, rather than hardcoding it, we can let the pathlib library figure out this particular user's home directory:

# script.py

config_path = pathlib.Path.home() / '.config' / f'{__name__}.json'

# and later...

config_path.parent.mkdir(exist_ok=True, parents=True)
config_path.write_text(json.dumps(config))

Regular Expressions

Another extremely important construct provided by the standard library is regular expressions. They are a powerful way of defining patterns of text, which can then be matched against actual strings to see if they fit, or even extract and substitute parts of them if they do. Their syntax, a small programming language of its own, is more elaborate than the simple glob patterns we've seen before, but much more versatile — and while I won't cover all of it, I'd like to give you the highlights.

The brackets, [], work much like in a glob: they match exactly one character of the specified collection or range, called a character class. [abcdef], for example, matches one of the specified letters, as does [a-f]; [0-9a-f] is effectively a hexadecimal digit:

>>> import re

>>> def is_hex(s):
...     return bool(re.match('[0-9a-f]', s))

>>> is_hex('1')
True
>>> is_hex('a')
True
>>> is_hex('q')
False

Hexadecimal digits, however, should be case insensitive; A is just as valid as a. We could accommodate this by expanding our pattern to [0-9a-fA-F], but we can also make the matching case-insensitive by passing the re.IGNORECASE flag (or its shorter version, re.I) as an extra argument:

>>> def is_hex(s):
...     return bool(re.match('[0-9a-f]', s, re.I))

>>> is_hex('A')
True

Checking for one digit, though, is not very interesting — so regular expressions are famous for their repetition abilities. Adding + after some token (like a character class) means it can appear more than one time:

>>> def is_hex(s):
...     return bool(re.match('[0-9a-f]+', s, re.I))

>>> is_hex('ab12')
True
>>> is_hex('qb12')
False

Characters like [, ] and + are called meta characters, because of the special meaning they take in a pattern (unlike regular characters like a, b or 1). If we would like to describe an addition of two hexadecimal numbers, then, we can't simply add + and expect it to match — it'll just try to repeat whatever comes before it. That's why we need to escape it, using the familiar backslash: \+; but because it's also used as an escape character in Python strings, we have to escape it twice, as in \\+ (so after the string is escaped, there's still a backslash left for the pattern). When writing elaborate regular expressions, this gets pretty tedious, which is why Python's raw strings are particularly useful in this case, disabling string escaping so we can focus on the pattern alone:

>>> def is_hex_sum(s):
...     return bool(re.match(r'[0-9a-f]+ \+ [0-9a-f]+', s, re.I))

>>> is_hex_sum('ab + 12')
True
>>> is_hex_sum('qb + 12') # not a hex number
False
>>> is_hex_sum('ab - 12') # not addition
False

Regular Minutiae

There's a problem with our pattern, though: it's rather fragile, because it assumes there'd be exactly one space before and after the +: 'ab+12' or 'ab + 12' would not match it, even though we'd probably like to support them, too. We can try adding another + after each of these spaces, meaning "match one space or more", as in:

>>> def is_hex_sum(s):
...     return bool(re.match(r'[0-9a-f]+ +\+ +[0-9a-f]+', s, re.I))

And I know what you're thinking — a lot of people find this dense, quirky syntax bizarre and hard to follow. At the same time, parsing this in any other way (let alone, so concisely) is inherently nontrivial; and once we get used to this notation, it does make its own kind of perverse sense.

So let's plow on: this is still not exactly what we had in mind, because it expects at least one space, whereas we want to support no spaces as well. Another meta character, *, lets us do exactly that: it means a token can repeat zero or more times, matching even if it's missing entirely:

>>> def is_hex_sum(s):
...     return bool(re.match('[0-9a-f]+ *\+ *[0-9a-f]+', s, re.I))

>>> is_hex_sum('ab+12')
True
>>> is_hex_sum('ab  + 12')
True

So to recap on a simpler example: a+ would match one a or more, whereas a* would match zero as or more. There's also a?, which would match zero or one as, a{5} (or any other number) which would match exactly five as, and a{2,8} which would match anywhere between two and eight as; pretty cool, no?

And we can do even better. Right now, since we only expect spaces, a different whitespace like a tab or a newline can throw us off course. We could use something like [ \t\r\n]* instead, but regular expressions provide us with shorthand character classes, encapsulating standard groups of related symbols for us. We're interested in \s, which means whitespace characters, but there's also \d for digits, and \w for word (i.e. alphanumeric) characters, and a few more; and if we want the opposite, as in all the characters except the ones in this group, we simply capitalize the letter: \S for all non-whitespace, \D for all non-digit, \W for all non-word, etc. So:

>>> def is_hex_sum(s):
...     return bool(re.match('[\da-f]+\s*\+\s*[\da-f]+', s, re.I))

>>> is_hex_sum('ab\t+\t12')
True

Extracting Patterns

So far, I was wrapping re.match in bool to get True or False, because actually, it returns a Match object (or None, if not match was found). In a more realistic example, we probably wouldn't just check if some string is a hexadecimal sum, but would also want to compute it — like, maybe we're implementing a hexadecimal calculator — in which case, we'd be interested in said match object. If we add groups to our pattern, we'll be able to capture any parts of the pattern we put in parentheses, and extract them later:

>>> def hex_sum(s):
...     match = re.match(r'([\da-f]+)\s*\+\s*([\da-f]+)', s, re.I)
...     if not match:
...         raise ValueError(f'not a hexadecimal sum: {s}')
...     num1, num2 = match.groups()
...     return int(num1, 16) + int(num2, 16)

>>> hex_sum('ab + 12')
189

We can also access specific groups as match.group(1), or simply match[1]; and note that index 0 in both holds the entire matched expression (for reasons we'll cover later). Now, if we really are implementing a calculator, we'd like to support other basic operators as well — and as a bonus, allow our hexadecimal numbers to start with an optional 0x, like they do in Python.

Let's start with the first challenge: it's quite simple, to put all of the operators in brackets, especially since we don't need to escape them inside. The only thing we need to pay attention to is putting - (as in, minus) either first or last, so the character class doesn't consider it as a range separator.

Adding an optional 0x is a bit trickier, because 0x? is not good enough: the ? will only affect the x, so 0 will be required; and 0?x? will accept either one of them (e.g. 012 or x12). To achieve what we want, we have to group them in parenthesis: (0x)? — even though we don't really care to capture this value. Since we already have quite a few groups, we might want to consider giving them names, which has the rather weird syntax of (?P<name>...) — and this already makes the regular expression so long and complicated, that I would break it over several lines and add comments to its parts:

>>> pattern = r'''
...     (0x)?                  # optional prefix
...     (?P<operand1>[\da-f]+) # hexadecimal number as operand1
...     \s*                    # whitespace
...     (?P<operator>[+*/-])   # mathematical operator
...     \s*                    # whitesapce
...     (0x)?                  # optional prefix
...     (?P<operand2>[\da-f]+) # hexadecimal number as operand2
... '''

Which requires adding the re.VERBOSE (or re.X) flag; and since flags are actually expressed as numbers (using their bits), the way to combine them is with bitwise-or, |:

>>> def hex_sum(s):
...     match = re.match(r'([\da-f]+)\s*\+\s*([\da-f]+)', s, re.I)
...     if not match:
...         raise ValueError(f'not a hexadecimal sum: {s}')
...     num1, num2 = match.groups()
...     return int(num1, 16) + int(num2, 16)

>>> hex_sum('ab + 12')
189

>>> def hex_sum(s):
...     match = re.match(pattern, s, re.I | re.X)
...     if not match:
...         raise ValueError(f'not a hexadecimal sum: {s}')
...     args = match.groupdict()
...     operator = {
...         '+': lambda num1, num2: num1 + num2,
...         '-': lambda num1, num2: num1 - num2,
...         '*': lambda num1, num2: num1 * num2,
...         '/': lambda num1, num2: num1 / num2,
...     }[args['operator']]
...     operand1 = int(args['operand1'], 16)
...     operand2 = int(args['operand2'], 16)
...     return operator(operand1, operand2)

>>> hex_sum('ab * 0x12')
3078

Alternatively, we can use (?:...) to group 0x without capturing it, in which case our scenario is probably simple enough for unnamed groups after all:

>>> pattern = r'(?:0x)?([\da-f]+)\s*([+*/-])\s*(?:0x)?([\da-f]+)'

>>> def hex_sum(s):
...     match = re.match(pattern, s, re.I)
...     if not match:
...         raise ValueError(f'not a hexadecimal sum: {s}')
...     operand1, operator, operand2 = match.groups()
...     # similar to before

Matching Lines

To give another example, suppose we have a list of people's names and phone numbers in the following format:

>>> data = '''
... Alice Pleasance Lidell: 1234567;
... Bob Baratheon:          2345678;
... Charlie Windsor:        3456789;
... '''

And we want to extract it into (name, phone_number) pairs for further processing. Figuring out the character class for a valid name can be tricky (for example, Scarlet O'Hara has an apostrophe; Jean-Paul Sartre has a dash; René Descartes has an acute accent), and the same goes for phone numbers. Luckily, regular expressions provide the meta character . (dot), which matches anything at all (except newlines), so we can simply do:

>>> re.findall(r'(.+):\s*(.+);', data)
[('Alice Pleasance Lidell', '1234567'),
 ('Bob Baratheon', '2345678'),
 ('Charlie Windsor', '3456789')]

And while re.findall returns a list of the matched groups (or strings, if there are no groups in the pattern), if we need proper match objects, re.finditer gives us an iterator thereof. But now, suppose the data might include additional details:

>>> data = '''
... Alice Pleasance Lidell: 1234567;
...     Date of birth: 04/05/1852;
... # and so on
... '''

Suddenly, we get (' Date of birth', '04/05/1852') in our list, as if it was a name and a phone number, because technically, it matches our pattern. We can try to exclude it by making the phone number more specific: \d*, or even \d{7}; but then we miss on valid records like +1 (123) 456-7890, the whole reason we used . to begin with, and it still doesn't guarantee we won't have some unrelated field (like an employee ID) that maps to 7 digits. instead, let's make sure the line starts with a non-space character: \S.* — but this isn't enough either, because as re.findall scans the string, 'Date of birth' qualifies as a valid name all the same, albeit without the leading whitespace.

What we need is a way to mark the beginning of a line: ^ (there's also an end of line, $). Unfortunately, the default assumption is that our string is one line long (so effectively, those meta characters stand for the beginning and end of the string, as a whole), so adding it will only match the very first record:

>>> re.findall(r'^(\S.*):\s*(.+);', data)
[('Alice Pleasance Lidell', '1234567')]

That is, unless we add the re.MULTILINE (or re.M) flag:

>>> re.findall(r'^(\S.*):\s*(.+);', data, re.M)
[('Alice Pleasance Lidell', '1234567'),
 ('Bob Baratheon', '2345678'),
 ('Charlie Windsor', '3456789')]

As an aside, adding ^ in re.match is redundant: it expects the string to start with the provided pattern anyway. And if we want the string to end with the pattern as well, rather than adding $ we can call re.fullmatch. If, on the other hand, we want to search for the pattern anywhere inside the string, re.search is our function; that is to say, re.match(r'pattern') is equivalent to re.search(r'^pattern'), and re.fullmatch(r'pattern') is equivalent to re.search(r'^pattern$').

Non-Greedy Matching

Let's make the problem even messier: suppose that sometimes, linebreak are missing, and we might end up with:

>>> data = '''
... Alice Pleasance Lidell: 1234567;Bob Baratheon: 2345678;
... # and so on
... '''

Watch what happens with our pattern as it is right now:

>>> re.findall(r'^(\S.*):\s*(.+);', data)
[('Alice Pleasance Lidell: 1234567;Bob Baratheon', '2345678'), ...]

In other words, * (as well as +) is greedy: it tries to consume as many characters as it can, and since technically the line does end with a colon and a phone number, it gobbles up any preceding data and calls it a name.

The trick to avoid this is, rather than using ., opt for a negated character class: if the first character after the bracket is ^, regular expressions will match any character that is not part of this collection. Since colon is our delimiter, and we don't expect it to appear in names, we can say that a name is anything that starts with a non-space character and contains any number of non-colon characters afterwards, effectively capping it at the first colon:

>>> re.findall(r'^(\S[^:]*):\s*(.+);', data)
[('Alice Pleasance Lidell', '1234567'), ('Bob Baratheon', '2345678'), ...]

This doesn't always work, though: if our delimiter was a double dash, for example, we'd have no (easy) way to express this, especially since single dashes are allows in names. So regular expressions provide us with non-greedy alternatives, *? (and +?):

>>> re.findall(r'^(\S.*?):\s*(.+);', data)
[('Alice Pleasance Lidell', '1234567'), ('Bob Baratheon', '2345678'), ...]

Which try to match as little as possible, but also "stretch" on demand, if the rest of the pattern requires it; supporting a double dash separator this way would be no problem, since the pattern would hesitate at the first dash it encounters, but unless it's followed by another one, it would be forced to keep going.

Matching Blocks

Now, suppose our scenario is a bit different: we're not trying to tease out a few specific values like the name and phone number, but rather separate the records; in other words, match each line and all its nested lines together. Our data:

>>> data = '''
... Alice Pleasance Lidell: 1234567;
...     Date of birth: 04/05/1852
... Bob Baratheon:          2345678;
...     Date of birth: 262 AC
... Charlie Windsor:        3456789;
...     Date of birth: 14/11/1948
... '''

The first problem we encounter is that even if we use ., I said it doesn't match newlines, so even in the best case scenario, .* will only capture one line at a time. To alleviate this concern, we can add the re.DOTALL (or re.S) flag, but then .* will match everything — so we need to figure out a different stop condition.

Ideally, we'd like something like ^(\S.*?)^\S — that is, for any line that doesn't start with a space, consume it and all the following lines until another such line is encountered. The problem is, this will also consume, and effectively "steal", that character from the next line. To avoid this, we need to use a lookahead assertion: matching a pattern without actually including some part of it in the result, using (?=...):

>>> re.findall(r'^(\S.*?)(?=^\S)', data, re.M | re.S)
['Alice Pleasance Lidell: 1234567;\n    Date of birth: 04/05/1852;\n',
 'Bob Baratheon:          2345678;\n    Date of birth: 262 AC;\n']

This seems to work, except it misses out on the last line; and no wonder, since it's not actually followed by another non-empty line, which was our stop condition. Here, we'll need another feature: alternation, as in (?=^\S|$) — either another line starting with a non-space character, or an end-of-string. Unfortunately:

>>> re.findall(r'^(\S.*?)(?=^\S|$)', data, re.M | re.S)
['Alice Pleasance Lidell: 1234567;',
 'Bob Baratheon:          2345678;',
 'Charlie Windsor:        3456789;']

Since we're in MULTILINE mode, $ also means end of line; and you have to admit, every line ends with an end-of-line, which satisfies the pattern. So when these symbols are "busy", we can use their equivalent shorthands, \A (for start of string) and \Z (for end):

>>> re.findall(r'^(\S.*?)(?=^\S|\Z)', data, re.M | re.S)
['Alice Pleasance Lidell: 1234567;\n    Date of birth: 04/05/1852;\n',
 'Bob Baratheon:          2345678;\n    Date of birth: 262 AC;\n',
 'Charlie Windsor:        3456789;\n    Date of birth: 14/11/1948;\n']

There was actually another thing we could do: re.split, which accepts a pattern by which to break the string up — in our case, simply any newline (\n) followed by a lookahead assertion of a non-space character:

>>> re.split(r'(\n)(?=\S)', data)
['Alice Pleasance Lidell: 1234567;\n    Date of birth: 04/05/1852;',
 'Bob Baratheon:          2345678;\n    Date of birth: 262 AC;',
 'Charlie Windsor:        3456789;\n    Date of birth: 14/11/1948;\n']

Backreferences

One last thing I'd like to show is backreferences: the ability to reference back to an already captured group. In matching, it can help us do something like have an HTML element that starts and ends with the same tag, like p in <p>...</p>:

>>> pattern = r'<(\w+)>(.*?)</\1>'
>>> re.match(pattern, '<x>hello world</y>')
None
>>> re.match(pattern, '<p>hello world</p>')
<re.Match object; span=(0, 18), match='<p>hello world</p>'>

A backslash followed by a number means "whatever got into that group" (and for named groups, we can use (?P=name)). This becomes particularly useful, however, when we deal with substitution. Suppose we want to convert markdown syntax, which is easier for users to write, into HTML, which is easier for browsers to display; in the first one, links are written as [text](url), and in the second, <a href="url">text</a>. Using re.sub, we can do this like so:

>>> markdown = 'click [here](www.google.com)'
>>> re.sub(r'\[(.+?)\]\((.+?)\)', r'<a href="\2\">\1</a>', markdown)
'click <a href="www.google.com\\">here</a>'

If you've worked with Unix tools like sed or vim, you might be familiar with search and replace functionality expressed like so:

/\[(.+?)\]\((.+?)\/<a href="\2\">\1</a>/g

/ is used as a delimiter: the first part is the pattern to match, the second is the pattern to replace it with (often using backreferences, if we want to preserve some groups), and the third one can be empty, but if it isn't, should contain flags (like i for case insensitivity or s for dot-all). A particularly common flag in this case is g, or GLOBAL, which means "replace all such occurrences", because the default behavior is to replace just one such occurrence per line; in Python, though, this is the default behavior, which is why this flag is missing and we didn't need to use it in any of our examples.

And anyway, Python's re.sub is much more powerful than that. Suppose we want to convert the same markdown into a printable version of our website: so, HTML is still used to define the layout, but putting in links is immaterial, since nobody will be able to actually click them. Rather, we'd like to replace them with incremental footnote markers, and append an index of such footnotes (with explicit URLs that can be copied manually at the end). This is not a simple search and replace: it searches, sure, and replaces what it finds with something else, but in so doing also captures parts of it in a data structure that we can then format and include at the bottom — which means we need to run some code, and Python lets us do just that by passing a callable (rather than a replacement string) as the second argument:

>>> footnotes = {}
>>> def replace(match):
...     text, url = match.groups()
...     index = len(footnotes) + 1
...     footnotes[index] = url
...     return f'{text}<sup>{index}</sup>' # add index in superscript
>>> re.sub(r'\[(.+?)\]\((.+?)\)', replace, markdown)
'click here<sup>1</sup>' # click here1
>>> # and at the end:
>>> print('\n'.join(f'{index}. {url}' for index, url in footnotes.items()))
1. www.example.com

In the previous chapter, I also promised that I'd tell you how to get rid of whitespaces in an elegant fashion using regular expressions. Well, at this point, it should be a piece of cake:

>>> re.sub(r'\s+', '', 'hello world\n')
'helloworld'

More interestingly, we can also implement that multi-replace we dreamed up, where we pass in a dictionary with all sorts of strings to change at once. See if you can follow:

>>> def multireplace(string, changes):
...     pattern = '|'.join(re.escape(change) for change in changes)
...     def replace_change(match):
...         return changes[match[0]]
...     return re.sub(pattern, replace_change, string)
>>> '<p>'.translate()
>>> multireplace('<p>', {'<': '&lt;', '>': '&gt;', '&': '&amp;'})
&lt;p&gt;

First, we create a pattern of all the changes separated by |, and escape them in case they contain meta characters (we wouldn't want . in e.g. .txt to match any character, so that would make it \.txt); and then, whenever one of those options is matched, we index the dictionary to fetch its replacement. Cute, no?

The Standard Library: Conclusions

I'm actually a bit sorry they're not a native type (and complex numbers are? Seriously?) — in Javascript, one can write something like /[0-9]+/g to end up.

The Standard Library: Exercises

The Standard Library: Further Reading

• https://www.regular-expressions.info/tutorial.html