Shell Autocompletion

This idea of multilingualism — programming in multiple languages at once, so as to leverage the benefits of each language — is actually quite a fascinating idea to explore, so let's take it for a spin.

I'm guessing that if you're reading a book on Python this advanced, you probably worked with Linux at some point in your life; and if you did, you probably used its shell, bash. It's somewhat more cumbersome and verbose than a graphical UI, no doubt, but it lets us write scripts, automate stuff, and generally feel like cool, hardcore programmers. As for its shortcomings, there are plenty of features to ameliorate them, from history to aliases — and chief amongst them is auto-completion.

The premise is simple: we type cd, hit tab, and see the directories into which we can change; or if we type cd proj, we see those starting with proj; or if there's just the one, it auto-completes to cd project, and in we go. If we type ls -- and hit tab, we see all the available options, and ls --a narrows it down to all those starting with a; and if we type git push origin and hit tab, we see all the local branches that we can push. In other words, it's context-aware, filling in not just command names, options and paths, but any argument we might be considering at the moment.

That is, if it's implemented. And while the idea is simple, the implementation is anything but. If you search the internet on how to do it, you'll find a whole three pages of documentation on the subject; and if you look for a practical example, you'll find git's autocompletion script, which turns out to be more than three thousand lines of agony. At this point, if you're looking to upgrade your own command-line interface in this way, you might very well give up. Not necessarily because Bash is a bad programming language; it's not — it has a lot of neat utilities and one-liners that would take many lines of Python code to reproduce. But when it comes to string manipulations, like splitting them and looping over them and filtering them out — which is exactly what we have to do in our scenario — it just isn't the best choice. You know what is, though? Python.

In Bash

So let's see if we can combine the two: one for its convenient control over the Linux environment, the other for deciding which auto-completion options to suggest. Luckily, the basics are straightforward, and that's all we need, Bash-side: we call complete -F function command, and it binds said function to said command, so whenever the command is typed and tab is hit, the function is called with some environment variables, defines the COMPREPLY array, and those values are used for auto-completion. For example, if our command is called prog, we might write prog-completion.sh like so:

_complete_prog()
{
    local OPTIONS="foo bar foobar"
    COMPREPLY=( $OPTIONS )
}
complete -F _complete_prog prog

In which case we can do:

$ source prog-completion.sh # applies auto-completion
$ prog <TAB>
foo  bar  foobar
$ prog f<TAB>    # becomes foo and shows:
foo  foobar
$ prog foob<TAB> # becomes foobar

This is actually a lie — I just wanted to make sure we understand how this example's auto-completion is supposed to work. In reality, because it doesn't do any filtering and always generates the same three options, typing f<TAB> would still offer bar as one if the suggested completions. So really, we have to do this:

_complete_prog()
{
    local OPTIONS="foo bar foobar"
    local CURRENT="${COMP_WORDS[$COMP_CWORD]}"
    COMPREPLY=( $(compgen -W "$OPTIONS" -- "$CURRENT") )
}
complete -F _complete_prog prog

That is, take the words typed so far (that's COMP_WORDS), focus on the one being typed at the moment (that's COMP_CWORD), and use compgen to filter the options that start with it. It's already full of funny characters and escaping considerations; just imagine how quickly it grows in complexity when we want to account for some actual context (or don't — you can always look at git's extension to appreciate the madness).

In Python

So instead, let's do the bare minimum, grab only what we need, and escape into Python as soon as possible:

_complete_prog()
{
    local CURRENT="${COMP_WORDS[$COMP_CWORD]}"
    COMPREPLY=( $(python autocomplete.py "$CURRENT" ) )
}
complete -F _complete_prog prog

As for autocomplete.py:

import sys

def main():
    options = "foo", "bar", "foobar"
    cword = sys.argv[1]
    match = [word for word in options if word.startswith(cword)]
    print("\n".join(match))
    
if __name__ == "__main__":
    main()

At first sight, it looks like more work than what we did in Bash (I told you it's full of funky one-liners) — but that's only because compgen did all the heavy lifting for us, and the moment we'd want to do something more complicated, it might very well fall short. With Python, we're in a different world when it comes to string manipulation — or context management, for that matter, like connecting to a database to generate our options from there.

The only nuisance left is the setup: with pure Bash, all we needed to do was source the script and everything worked. Here, we also need to install the Python script, figure out where it's stored, and make sure the Bash script references it correctly; what's more, we have to do this for every program we'd want to autocomplete.

In Both — At Once

So instead, let me show you something quite perverse:

""":"
echo "hello from bash"
exit
"""
print("hello from python")

Let's call it script.sh.py; I'm not even sure how to syntax-highlight this fellow, because it's what's known as a polyglot script: code that is completely valid in more than one language — in this case, in both Python and Bash. In Python, it's this:

""":"
echo "hello from bash"
exit
"""
print("hello from python")

That is, a docstring that is simply ignored, followed by a print statement, so:

$ python script.sh.py
hello from python

In Bash, however, it's this:

""":"
echo "hello from bash"
exit
"""
print("hello from python")

That is, an empty string ("") which is ignored; a quoted command (":") which results in the no-op : operation — so effectively, the first line does nothing, and is only there to get us out of the woods — and then an echo and an exit, bailing us out and discarding the rest of the (admittedly, Python-only) code. The result:

$ bash script.sh.py
hello from bash

This nifty trick will let us implement our entire auto-completion framework in a single, self-contained file. It will be Bash when we source it, setting up all the necessary shell configurations — and then Python, when it re-invokes itself, letting us switch languages like we wanted without worrying which part is stored where. In other words:

""":"
autocomplete()
{
    local COMMAND="$1"
    eval "
    _complete_$COMMAND()
    {
        local THIS_FILE=\"\$(readlink -f \"\${BASH_SOURCE[0]}\")\"
        local CURRENT=\"\${COMP_WORDS[\$COMP_CWORD]}\"
        COMPREPLY=( \$(python \"\$THIS_FILE\" \"\$CURRENT\") )
    }
    complete -F _complete_$COMMAND $COMMAND
    "
}
return # exit would kill our shell when we source this, so we use return
"""

import sys

def main():
    options = "foo", "bar", "foobar"
    cword = sys.argv[1]
    match = [word for word in options if word.startswith(cword)]
    print("\n".join(match))
    
if __name__ == "__main__":
    main()

All we need to do is source this once (e.g. in our .bashrc or .profile), and we will gain the autocomplete utility, which accepts some command (like our prog), creates an auto-completion function for it (_complete_prog) and binds the two. Then, when we type prog and hit tab, the same script is run — this time, as Python — to generate the options like we did before.

A Language for Auto-Completion

Of course, having the same auto-completion options for every command is hardly useful; so when we specify that we want to autocomplete prog, we should pass in some auto-completion specification as well, to account for the particular options of the command in question. Let's start with the following format (say, in prog.completions):

my-command --log --level=debug|info|warning|error

And augment our polyglot script to pass this configuration on. Since we want context-awareness (i.e. --log if it hasn't appeared yet, or --level= and then one of its options), we need not only the current word, but the entire line:

''':'
autocomplete()
{
    local COMMAND="$1"
    local CONFIG_FILE="$(readlink -f "$2")"
    eval "
    _complete_$COMMAND()
    {
        local THIS_FILE=\"\$(readlink -f \"\${BASH_SOURCE[0]}\")\"
        COMPREPLY=( \$(python \"\$THIS_FILE\" \"$CONFIG_FILE\" \
                              \"\$COMP_LINE\" \"\$COMP_CWORD\") )
    }
    complete -F _complete_$COMMAND $COMMAND
    "
}
return
# … same as before

As an aside, from Python 3.12 on, this throws a SyntaxWarning that \$ is not a valid escape sequence. Since we have quite a few of those in our supposed docstring, we should suppress it by changing our invocation to python -W ignore.

Anyway, let's think about the Python part. It now receives the configuration file, which it needs to parse into a mapping of commands to their expected options and arguments. Then, depending on the command we're dealing with (although for now we have just the one, my-command), it needs to suggest a completion: either --log or --level=, accounting for the prefix typed so far, but also for whether this option already appears in the line or not, since we don't want duplicates. We also want --log to be auto-completed with a space afterwards, letting us work on the rest of the command if necessary, while --level= should stay put after the =, helping us auto-complete its value as well. This will introduce an interesting inter-language complication, but first things first:

# … same as before
import sys

def main():
    commands = parse_commands(sys.argv[1])
    words, index = parse_words(sys.argv[2], sys.argv[3])
    if index == 0:
        completions = complete_command_name(commands, words)
    else:
        command_name = words.pop(0)
        if command_name not in commands:
            raise ValueError(f"invalid command {command_name!r} "
                             f"(expected one of: {", ".join(commands)})")
        index -= 1
        options, arguments = commands[command_name]
        completions = complete_command(options, arguments, words, index)
    complete(completions)
    
# to be continued…

if __name__ == "__main__":
    main()

We start by parsing all the available commands from the configuration file, as well as the words typed so far and the index of the word that we're at. If we're at the very first word, it means we need to complete the command name; otherwise, we need to fetch that command's expected options and arguments, and complete those. The parsing of the configuration is quite trivial:

def parse_commands(path):
    commands = {}
    for line in open(path):
        command_name, *parameters = line.split()
        options, arguments = {}, []
        for parameter in parameters:
            if parameter.startswith("--"):
                if "=" in parameter:
                    option, values = parameter.split("=", 1)
                    options[option] = values.split("|")
                else:
                    options[parameter] = None
            else:
                arguments.append(parameter.split("|"))
        commands[command_name] = options, arguments
    return commands

That is, we go over the file line by line, mapping the command name to its options and arguments. Options start with --, and can be either flags (in which case we map them to None) or valued options (in which case we map them to a list of values); arguments are anything that's not an option. Then:

import shlex

def parse_words(line, index):
    words = shlex.split(line)[1:]
    index = int(index) - 1
    return words, index

We split the line into words using the standard shlex module, removing the program name itself, and convert the current word index to an integer, adjusting for that removal. To complete the first bit — that is, the command name:

def complete_command_name(commands, words):
    prefix = words[0] if len(words) > 0 else ""
    return [name for name in commands if name.startswith(prefix)]
    
def complete(completions):
    print("\n".join(completions))

We use the current word (if any of it was typed, even) to filter out the appropriate command names, and simply print all available suggestions. Once the command is determined, filling in its options is a bit more sophisticated:

def complete_command(options, arguments, words, index):
    for word in words:
        if word.endswith("="):
            index -= 1
        elif "=" in word:
            index -= 2
    prefix = words[index] if index < len(words) else ""
    completions = complete_options(options, prefix, words)
    completions.extend(complete_arguments(arguments, prefix, words, index))
    return completions

Basically, we collect all the options that match the current word (if any), as well as all the arguments. Because of a particular peculiarity of Bash auto-completion, valued options (e.g. --level=debug) are not considered as a single word; the = effectively splits them into two, so to catch the correct prefix we need to account for that, decreasing the index by 1 when we see a word ending in =, and by 2 (for both the = and its followup) when it's in the middle. This gives us the actual index of the current word, which is then used as a prefix to figure out which options and arguments should be completed. Bear with me — we're almost there:

nospace = False

def complete_options(options, prefix, words):
    global nospace
    if "=" in prefix:
        name, prefix = prefix.split("=", 1)
        values = options.get(name, [])
        return complete_values(values, prefix)
    completions = []
    for option, values in options.items():
        if values is None: # flag
            included = option in words
            completion = option
        else: # valued option
            included = any(word.startswith(f"{option}=") for word in words)
            completion = f"{option}="
        if option.startswith(prefix) and not included:
            nospace = completion.endswith("=")
            completions.append(completion)
    return completions
    
def complete_values(values, prefix):
    completions = []
    for value in values:
        if value.startswith(prefix):
            completions.append(value)
    return completions

If an option contains =, we complete its values based on the prefix that comes after it; and if not, we add any option that matches our prefix and hasn't been included yet, distinguishing between flags and valued options. This has been tedious enough as it is, so handling arguments is left as an exercise to the reader; the interesting question left to answer is what exactly is that global nospace for?

Code-Switching

Excellent question. Before we answer it, though, a disclaimer: I realize all this parsing is hardly exciting, but I've decided to include it for a reason. Besides developing a full-blown, multi-lingual auto-completion framework in the span of a single chapter, which is cool, the truth is that metaprogramming — that is, code that works on other code — deals a lot with string manipulations, seeing as code is, at the end of the day, a string. I'm trying to keep the examples clean and versatile enough as to be educational, and to build towards interesting complications with creative solutions – but sometimes this takes a bit of effort. Luckily, we've arrived at such a complication.

Like I said before, we want flag options to have a space after them (which is the default in Bash), but valued ones to stay put, so another tab hit would show the values of that option rather than additional options and arguments of the overall command. It might seem a bit minute, but it's a precursor for a similar situation we'll encounter when dealing with paths, so I thought it's best to start with the simplest case.

The way such scenarios are normally handled is with the compopt command: before returning COMPREPLY, we configure how we want our auto-completion to behave. To avoid the extra space, for example, we use compopt -o nospace — but how can we do so from Python?

Honestly, we can't: each language has its own advantages, and while Python lets us parse with ease, the actual intricacies of auto-completion are inherently Bash's domain. What this means is that we have to zig-zag between the two languages, finding a way to coordinate their strengths so as to get the best of both worlds.

To that extent, consider the following convention: so far we've been emitting all the completions to the standard output, separated by a newline, which our Bash script collected into the COMPREPLY array. However, we can agree that the first of those lines will always be a compopt configuration, rather than a completion; and choose a special token, like a single dot, to indicate that no configuration is required. The change would look like this:

''':'
autocomplete()
{
    local COMMAND="$1"
    local CONFIG_FILE="$(readlink -f "$2")"
    eval "
    _complete_$COMMAND()
    {
        local THIS_FILE=\"\$(readlink -f \"\${BASH_SOURCE[0]}\")\"
        local RESULT=( \$(python \"\$THIS_FILE\" \"$CONFIG_FILE\" \
                                 \"\$COMP_LINE\" \"\$COMP_CWORD\") )
        local COMPOPT="\${RESULT[0]}"
        if [ -n \"\$COMPOPT\" -a \"\$COMPOPT\" != "." ]
        then
            compopt -o \"\$COMPOPT\"
        fi
        COMPREPLY=( \"\${RESULT[@]:1}\" )
    }
    complete -F _complete_$COMMAND $COMMAND
    "
}
return
# … same as before

That is, using the first result for compopt (unless it's empty or .), and the rest for COMPREPLY. As an aside, just appreciate for a moment that we didn't have to implement all our previous logic in Bash (I mean — fi? ${{RESULT[@]:1}}???). All we need now is to adjust our complete function:

def complete(completions):
    if nospace:
        print("nospace")
    else:
        print(".")
    print("\n".join(completions))

And everything works flawlessly:

$ autocomplete prog prog.completions # applies auto-completion
$ prog m<TAB> # becomes my-command
$ prog my-command <TAB> # becomes --l and shows:
--log  --level=
$ prog my-command --lo<TAB>   # becomes --log, with a space
$ prog my-command --log <TAB> # becomes --level=, with no space
$ prog my-command --log --level=<TAB>
debug  info  warning  error

Auto-Completing Paths

Next, we can similarly implement subcommands, arguments, and even variadic arguments, i.e. supporting the following syntax in our auto-completion configuration:

# git.completions
origin get          # subcommand
origin set <string> # subcommand with argument
add --all <path>*   # variadic argument
commit --message=<string>
push <origin> <branch>

However, a lot of it is just bookkeeping, so we don't bother with it here. What I'd like to focus on, like I said, are the complications that require creative solutions — and in the example above, that would be the <argument-kind> notation. How do we decide, based on the type specified between the angle brackets, how to auto-complete that argument's (or option's) value? To continue our previous example, suppose we'd want to add:

my-command --log --level=debug|info|warning|error --log-file=<path>

How do we provide completions for <path>? On the face of it, we can simply traverse our current directory's structure, stepping into whatever directories necessary to satisfy any existing prefix, and completing any of the matching entries therein. Let's edit our complete_values function accordingly:

import pathlib

def complete_values(values, prefix):
    completions = []
    for value in values:
        if value == "<path>":
            if "/" in prefix:
                directory, prefix = prefix.rsplit("/", 1)
                directory = pathlib.Path(directory)
            else:
                directory = pathlib.Path(".")
            completions.extend(str(file) for file in directory.iterdir()
                               if file.name.startswith(prefix))
        elif value.startswith(prefix):
            completions.append(value)
    return completions

However, this has the unfortunate side-effect of accumulating the entire path specified so far in each completion option:

$ prog my-command --log-file=<TAB>
dir/  file1  file2
$ prog my-command --log-file=d<TAB> # becomes dir/
$ prog my-command --log-file=dir/<TAB>
dir/file3  dir/file4

See the problem? Although we resolve the paths correctly, they're needlessly long, as they unnecessarily repeat the prefix. In regular Bash utilities, this is somehow avoided:

$ cat dir/<TAB>
file3  file4  # dir/ is not repeated

Upon further investigation, we discover that this is once again thanks to compopt. If we do compopt -o filenames, it tells Bash to strip the full paths we suggest into their bare file names. Since we've already established a channel to communicate these settings, all we need to do is:

filenames = False

def complete_values(values, prefix):
    global filenames
    completions = []
    for value in values:
        if value == "<path>":
            filenames = True
            # … same as before
            
def complete(completions):
    compopt = []
    if nospace:
        compopt.append("nospace")
    if filenames:
        compopt.append("filenames")
    if compopt:
        print(",".join(compopt))
    else:
        print(".")
    print("\n".join(completions))

And we get:

$ prog my-command --log-file=dir/<TAB>
file3  file4

Just like we wanted. But the problem runs deeper: we can support "standard" formats, like <string> (any value — so, effectively, no autocompletion) or <path> (what we just did); but how do we support something like <origin> or <branch> in our git example?

Dynamic Extensions

We can add them, of course — but whenever confronted with such a situation, before rushing to tailor solutions to particular requirements in a reactive way, I ask myself whether a proactive approach isn't in order. In this case, it definitely is: even if we add the git-specific considerations to our framework, tomorrow we might have:

show-user --user=<user>

Which might require a custom SQL query to fetch all the users from the database of that specific program, while another program might store its users in an entirely different format. In other words, when some completions are particular to the program being auto-completed, how can we extend its configuration to accommodate this dynamically? No matter how we spin it, it seems that to support all the possible cases, we'd have to develop an entire programming language.

Luckily, we already have a nice programming language on hand, and one that, as we've seen, yields itself to dynamic execution quite amicably. So instead of having our own completion syntax, let's rename prog.completions to prog_completions.py and do:

# my-command --log --level=debug|info|warning|error --log-file=<path>
# show-user --user=<user>

import sqlite3

def user(prefix):
    query = "SELECT name FROM users WHERE name LIKE ?"
    with sqlite3.connect("db.sqlite") as conn:
        return [row[0] for row in conn.execute(query, (f"{prefix}%",))]

Now, when we autocomplete prog prog_completions.py, our Bash script will pass this Python module to itself-as-Python, and then not only parse it, but load it. Completion specifications will appear as leading comments, and any extension (like <user>) will be matched to the functions defined below. Let's rework parse_commands to address this:

def parse_commands(path):
    commands = {}
    for line in open(path):
        if not line.startswith("#"):
            break # leading comments are over
        line = line.removeprefix("#").strip()
        # … same as before

And add dynamic loading:

extensions = {}

def main():
    commands = parse_commands(sys.argv[1])
    load_dynamic_extensions(sys.argv[1])
    # … same as before
    
def load_dynamic_extensions(path):
    code = open(path).read()
    namespace = {}
    exec(code, namespace)
    for name, extension in namespace.items():
        if callable(extension):
            extensions[f"<{name}>"] = extension

This will run our completion configuration as code and extract any functions defined in it, to be available when resolving values whose configuration matches their name. All that's left is amending complete_values to take these extensions into account, so if some <argument-kind> matches some <extension-name>, that extension (like our user function) is called to complete it:

def complete_values(values, prefix):
    completions = []
    for value in values:
        if value == "<path>":
            # … same as before
        elif value in extensions:
            completions.extend(extensions[value](prefix))
        elif value.startswith(prefix):
            completions.append(value)
    return completions

And lo and behold:

$ prog s<TAB>                 # becomes show-user
$ prog show-user <TAB>        # becomes --user=
$ prog show-user --user=<TAB> # and, depending on the names in our DB:
alice  bob  charlie

Consider what we just did. Developing the auto-completion framework sure took some work (although, honestly, it still clocks under 200 lines); but actually implementing an auto-completion specification for a program — including flags, valued options, paths, and a custom SQL query to autocomplete values from a database in the shell — that took a client of our framework a measly 5 lines of straightforward Python code, and a couple more for intuitively formatted comments.

We did this by shamelessly mixing Python and Bash — in the same polyglot script, no less — switching between them whenever one was better suited for the task at hand, and dynamically loading what custom configurations didn't fit within the domain-specific language we offhandedly implemented. All things considered, I hope the minute technicalities we had to deal with for that purpose were well worth the lesson.