Domain-Specific Languages

I really like Python; at this point it should be pretty obvious. For that reason, whenever the circumstances call for some sort of programmable format, I don't rush to invent my own mini-languages, but tend to start with solutions that, to use our newfound terminology, dynamically load a Python plugin – kind of like what we did with shell autocompletion. Sometimes it's enough, sometimes it's not; so in this chapter, I'd like to explore that tension.

Let's say we're building a search system, but don't want to commit to a particular database (today it's SQL, but tomorrow it might be MongoDB). So, we create a layer of abstraction where queries are represented as data, in which case YAML is my go-to format:

query:
    collection: users

It's basically a more readable version of JSON, easily convertible to and from native Python types:

>>> import yaml
>>> text = open("query.yaml").read()
>>> data = yaml.safe_load(text)
>>> data
{'query': {'collection': 'users'}}

Suppose the results are returned as dictionaries, and we want to post-process them somehow; like, change the username field to UserName, remove the password, and combine first_name and last_name into FullName. We can invent some sort of dialect to describe these modifications in our query:

collection: users
postprocess:
  - action: rename
    target: username
    to: UserName
  - action: remove
    target: password
  - action: combine
    targets:
      - first_name
      - last_name
    to: FullName

And while it looks alright, it's somewhat short-sighted: tomorrow we might need more functionality, like extracting the year from a birth_date stored as yyyy-mm-dd, or setting ResidesInUK to true if any address in addresses is in the UK. Not only will this require further development, but over time we'll end up with increasingly complicated and cumbersome syntax:

# … same as before
  - action: split
    target: birth_date
    delimiter: "-"
    index: 0
    to: Year
  - action: any
    target: addresses.country
    equals: UK
    to: ResidesInUK

What's more, users of our systems will have to recall this every time they write a query – effectively learning a weird, declarative programming language we made up as we went along, which is exactly what we wanted to avoid.

Python Plugins

So what do we do instead? Depends on who our users are. If we're building an internal tool intended for semi-technical professionals — the kind of people who can use SQL or Excel, but aren't programmers per se — I'd recommend going with Python plugins. From my experience, even without extensive experience with the language, such people can figure out their way around it, especially given a few templates to copy-paste and edit; even if they don't, ChatGPT does, and they can always ask it for help. The result:

collection: users
postprocess: |
    def postprocess(rows):
        processed = []
        for row in rows:
            processed.append({
                "UserName": row["username"],
                "FullName": row["first_name"] + " " + row["last_name"],
                "Year": int(row["birth_date"].split("-")[0]),
                "ResidesInUK": any(address["country"] == "UK"
                                    for address in row["addresses"]),
            })
        return processed

That is to say, there's no need to reinvent the wheel – Python is simple enough, powerful enough and well-known enough to fill in such gaps, especially if we're integrating it into another Python system, which can load and execute such plugins with ease:

def search(query):
    rows = fetch(query["collection"])
    postprocess = query.get("postprocess")
    if postprocess:
        namespace = {}
        exec(postprocess, namespace)
        rows = namespace["postprocess"](rows)
    return rows

A different scenario is when our users are not technical enough to work with code. Although, before you make this call, I sincerely encourage you to try: I've found that people might surprise you, especially when they want to get something done. Still, our next step before developing our own mini-language would be to see if we can encode our requirements in regular strings, which we can then parse with:

Regular Expressions

Let's revise our query to be less scary:

collection: users
postprocess:
    - rename username to UserName
    - remove password
    - combine first_name + last_name to FullName
    - get year from birth_date to Year
    - if any addresses.country is UK set ResidesInUK

This we can parse with regular expressions like:

rename_regex = r"rename (.*?) to (.*?)"

def rename(row, from_, to):
    row[to] = row.pop(from_)

In fact, since we'll be doing a lot of this, we might want to build a small framework around it — purely for ourselves, yeah? — so that adding new functionality will take a minute. Consider a decorator that binds a regular expression to a post-processing function, so that extending our pool of actions is just a matter of implementing that function and sticking @action("<regex>") on top. We can even simplify the pattern syntax and have the common (.*?) abbreviated to ?:

import re

actions = []

def action(pattern):
    regex = re.compile(pattern.replace("?", "(.*?)"))
    def bind(postprocess):
        def run_action(rows, instructions):
            for row in rows:
                for instruction in instructions:
                    if match := regex.fullmatch(instruction):
                        postprocess(row, *match.groups())
        actions.append(run_action)
        return postprocess
    return bind

This way, all of our logics are short and sweet:

@action("rename ? to ?")
def rename(row, from_, to):
    row[to] = row.pop(from_)
    
@action("remove ?")
def remove(row, field):
    row.pop(field)
    
@action("combine ? to ?")
def combine(row, fields, to):
    row[to] = ' '.join(row[f.strip()] for f in fields.split('+'))
    
@action("get year from ? to ?")
def get_year(row, field, to):
    row[to] = int(row[field].split("-")[0])
    
@action("if any ? is ? set ?")
def if_any(row, field, value, flag):
    row[flag] = any(v == value for v in resolve(row, field))

And note how there's no need to be overzealous in using regular expressions, either: a simple (.*?) can be enough to extract what we need, and when it's something like field+field+..., we don't have to come up with a clever way to capture each element, but can simply fields.split('+') the lot of them later. The one utility that we do need is a way to resolve "JSON paths" such as addresses.country – but that's something that might come in handy elsewhere as well, so I didn't want to bake it into if_any:

def resolve(row, json_path):
    if not json_path:
        return row
    if "." in json_path:
        field, json_path = json_path.split(".", 1)
    else:
        field, json_path = json_path, None
    value = row[field]
    if isinstance(value, list):
        return [resolve(item, json_path) for item in value]
    return resolve(value, json_path)

That's it; now that all our actions are primed and neatly tucked in a single list, all that's needed is to coordinate them with the actual instructions:

def search(query):
    rows = fetch(query["collection"])
    instructions = query.get("postprocess")
    if instructions:
        for action in actions:
            action(rows, instructions)
    return rows

As you can see, aspect-oriented thinking is not just for big projects; it can enhance our software design even in relatively mundane tasks.

Limited Evaluation

Sometimes, however, regular expressions are not enough. Imagine we'd want not only to post-process users, but also to filter them, and provide some sort of condition to that extent. Doing so in YAML is possible, but it ain't pretty — we have to split it into separate clauses, specifying the operation and operands for each:

collection: users
filter:
    # (user_age > 18) and (username != "admin")
    operation: and
    operands:
        - operation: greater_than
            operand1: user_age
            operand2: 18
        - operation: not_equal
            operand1: username
            operand2: admin

This is easy to consume, but hard to produce, so it's not very considerate to our users; let's think if we can do better. (user_age > 18) and (username != "admin") would be ideal — it is, after all, pretty standard mathematical notation, if only we could parse it. Unfortunately, regular expressions are not strong enough to handle this: we can tailor one for a specific scenario, of course, but it's a theoretical impossibility to support the general case, since regular expressions can't deal with recursion, such as arbitrary nesting might entail:

(user_age > 18) and ((username != "admin") or (username != "root"))

You know what can, though? — I know you do; it's the same answer as always. If we change our query format to:

collection: users
filter: (user_age > 18) and (username != 'admin')

We can simply evaluate that expression for each result, using its fields as the namespace, effectively having them play the role of variables in filter's "code":

def search(query):
    rows = fetch(query["collection"])
    # … postprocessing …
    filter_ = query.get("filter")
    if filter_:
        rows = [row for row in rows if eval(filter_, {}, row)]
    return rows

Note how we call eval with row as its second argument: this makes it treat the first one as globals and the second as locals. The reason we do this is so that Python, which naturally injects the builtins into the global scope before execution, doesn't pollute our actual rows.

Apropos builtins — in this case, we can make our evaluation reasonably secure by removing them, since they aren't really necessary, and I don't know of a hack that could hijack eval (that exception trick we saw earlier only works in exec, since it requires a with-statement):

eval(filter_, {"__builtins__": {}}, row)

But honestly, we can do better still. Filtering, after all, is best done in the database — rather than fetching everything and sieving through it afterwards, we should fetch just the relevant stuff to begin with. At first sight, this might seem impossible: eval only works on Python code and as part of a Python program, whereas our database, whether it's SQL, MongoDB or something else, probably won't run our Python as-is. What we need is a way of translating our code into another dialect – and to achieve that, we'll use a nifty trick: rather than evaluating the decision given a row, we'll construct an object that describes what decision to make; a delayed evaluation, if you will, which can be adjusted per database, generating an equivalent expression in whatever format it does understand.

If we're working with SQL, this is particularly easy: SQLAlchemy's table objects have a mapping of columns that already overload most operations, so we can compose them into expressions that can be converted to SQL out of the box:

>>> from sqlalchemy import create_engine, MetaData
>>> engine = create_engine("sqlite:///db.sqlite3")
>>> metadata = MetaData()
>>> metadata.reflect(bind=engine) # infer table structure from the database
>>> users = metadata.tables["users"]

>>> expression = (users.c.user_age > 18) & (users.c.username != "admin")
>>> expression # this is not a boolean, but an object:
<sqlalchemy.sql.elements.BooleanClauseList object at 0x…>
>>> # which can be converted to equivalent SQL:
>>> print(expression.compile(compile_kwargs={"literal_binds": True}))
users.user_age > 18 AND users.username != 'admin'

Again: since we want the database to do the filtering, we need to provide it with the appropriate query; and since this query hasn't run yet, we don't have its results on hand, and can't use the rows' fields to evaluate our decision. Instead, we'll use the table's columns as the variables in our namespace, so eval will construct an expression object like the one above. The only caveat is that SQLAlchemy uses & and | instead of and and or (since those are keywords, and as such, can't be overloaded), so we should account for that:

>>> code = "(user_age > 18) and (username != 'admin')"
>>> code = code.replace(" and ", " & ").replace(" or ", " | ")
>>> expression = eval(code, {"__builtins__": {}}, users.columns)
>>> print(expression.compile(compile_kwargs={"literal_binds": True}))
users.user_age > 18 AND users.username != 'admin'

This expression can be added as a WHERE clause in a SELECT statement, and the database will do the rest. Of course, should we decide to switch to MongoDB, we'll have to redo this — but if we want the filtering to happen in the backend, there's no way around such backend-specific adaptation.

So, how might this look in MongoDB? Its library doesn't provide us with magical columns like SQLAlchemy does – but this doesn't mean we can't create them ourselves. It's simply a matter of implementing the necessary operator overloads (although, for the sake of brevity, I'll only do the ones used by our example):

>>> class Expression:
...     def __gt__(self, other):
...         return BinaryExpression("gt", self, other)
...     def __ne__(self, other):
...         return BinaryExpression("ne", self, other)
...     def __and__(self, other):
...         return BinaryExpression("and", self, other)
...     def __or__(self, other):
...         return BinaryExpression("or", self, other)

>>> class BinaryExpression(Expression):
...     def __init__(self, op, arg1, arg2):
...         self.op, self.arg1, self.arg2 = op, arg1, arg2

>>> class Name(Expression):
...     def __init__(self, name):
...         self.name = name

This works the same way that columns do: we start with bare Name objects per field as our variables, and those get composed into binary expressions:

>>> fields = {"user_age": Name("user_age"), "username": Name("username")}
>>> expression = eval(code, {"__builtins__": {}}, fields)
>>> expression
<BinaryExpression object at 0x…>

Now we just need a way to convert this to MongoDB's query dialect, which would be {$and: [{user_age: {$gt: 18}, username: {$ne: "admin"}}]}. Since our expression objects are already properly nested, doing so recursively is pretty straightforward:

>>> class BinaryExpression(Expression):
...     # … same as before
...     def to_mongo(self):
...         arg1, arg2 = self.arg1, self.arg2
...         if isinstance(arg1, Expression):
...             arg1 = arg1.to_mongo()
...         if isinstance(arg2, Expression):
...             arg2 = arg2.to_mongo()
...         if self.op in ["and", "or"]:
...             return {f"${self.op}": [arg1, arg2]}
...         return {self.arg1: {f"${self.op}": arg2}}

>>> class Name(Expression):
...     # … same as before
...     def to_mongo(self):
...         return self.name

And we get:

>>> expression.to_mongo()
{'$and': [{'user_age': {'$gt': 18}}, {'username': {'$ne': 'admin'}}]}

That is, given a string with some condition, we found a way to evaluate it into intermediary representations that can be translated into SQL- and MongoDB-compatible queries both; adding new backends is equally feasible.

Context-Free Grammars

But what if Python's syntax itself is not enough – or rather, turns out to be too cumbersome to support all the conditions we want to allow? Suppose we want to filter all the users whose username matches the regular expression ^[aA] (that is, starts with A), and we'd like the ~= operator to mean "matches"?

username ~= "^[aA]"

Needless to say, this is not valid Python, so eval is not going to like it; but then, our users are not going to like the valid Python equivalent, either:

re.match(r"^[aA]", username)

We can, of course, content ourselves with what's possible and get creative — like, use @ to mean "matches", relying on the seldom used __matmul__ operator:

>>> class Expression(Expression):
...     # … same as before
...     def __matmul__(self, pattern):
...         return BinaryExpression("regex", self, pattern)

>>> expression = eval("username @ '^[aA]'", {"__builtins__": {}}, fields)
>>> expression.to_mongo()
{'username': {'$regex': '^[aA]'}}

But naturally, there's only so many operators we can overload.

At this point, it might be worth considering the next construct in the parsing hierarchy: context-free grammars, less known but more powerful than regular expressions. The PLY library I mentioned before is one way to define such grammars; another popular option is PyParsing, which was the de-facto standard for many years. Both, however, are pretty old, and provide a somewhat crude interface for the non-trivial task of articulating complex, recursive rules in Python. I prefer Lark: a newcomer that gave up on this altogether, realizing that the best way to define custom grammars is by using its own grammar-definition grammar instead; talk about meta.

The way it works is simple — at least, once we master Lark's meta-grammar. We're going to define a set of rules, outlining what potential forms our syntax might take. Starting from some root rule, we drill down into ever more specific configurations, until we arrive at mere tokens, which are simple enough to describe with plain old regular expressions. The core feature of CFGs that we were lacking before is that each rule can be defined in terms of other rules – even itself! – thus allowing nested, recursive structures.

A number with a balanced set of parentheses, for example, can be described with x = /[0-9]+/ | "(" x ")" – essentially saying "I'm either a list of digits, or myself between ( and )", with "myself" implying both a list of digits or, again, itself between ( and ), ad infinitum, matching 1, (1), ((1)) and so on. In our case, the grammar might look like this:

>>> import lark
>>> grammar = lark.Lark("""
...     %import common.WS
...     %import common.CNAME -> NAME
...     %import common.NUMBER
...     %ignore WS
...     ?start: expression
...     ?expression: or_ | and_ | comparison | "(" expression ")"
...     or_: expression ("or" expression)+
...     and_: expression ("and" expression)+
...     ?comparison: gt | ne | matches
...     gt: NAME ">" NUMBER
...     ne: NAME "!=" string
...     matches: NAME "~=" string
...     ?string: "'" /[^']+/ "'"
... """)

Briefly, the %import statements bring forth standard definitions like whitespace (WS), identifiers (CNAME) and numbers. Then, %ignore WS tells the grammar to discard whitespace, and the rest of the lines are the rules, which have to include start as their root. We set it to be an expression, which, in turn, can be or_, and_, comparison, or a parenthesized expression (that is, itself) – CFG's recursive definition in action. or_ and and_ are simply multiple expressions with or/and between them (note the recursion again); and comparison is a name followed by > NUMBER, != string, or ~= string, where string is a sequence of anything-except-quote between two quotes (i.e. a simple token, defined using a regular expression). As for the occasional ? prefix, I'll address it in a bit. All things considered, for defining something so nuanced and self-referential, this isn't a lot of work – so reading Lark's documentation is well worth it.

At this point, we can parse strings into trees, just like we did with ast – it's just that we get to do it according to our own custom rules rather than Python's:

>>> tree1 = grammar.parse("(user_age > 18) and (username != 'admin')")
>>> print(tree1.pretty())
and_
  gt
    user_age
    18
  ne
    username
    "admin"
>>> tree2 = grammar.parse("username ~= '^[aA]'")
>>> print(tree2.pretty())
matches
  username
  ^[aA]

And just like with ast, we can write transformers that use the visitor pattern to call specific methods on specific nodes, effectively traversing the tree to convert it into a more actionable object, like our MongoDB query:

>>> class MongoTransformer(lark.Transformer):
...     def or_(self, nodes):
...         return {"$or": list(nodes)}
...     def and_(self, nodes):
...         return {"$and": list(nodes)}
...     def gt(self, nodes):
...         name, number = nodes
...         return {name.value: {"$gt": int(number.value)}}
...     def ne(self, nodes):
...         name, string = nodes
...         return {name.value: {"$ne": string.value}}
...     def matches(self, nodes):
...         name, regex = nodes
...         return {name.value: {"$regex": regex.value}}

>>> mongo = MongoTransformer()
>>> mongo.transform(tree1)
{'$and': [{'user_age': {'$gt': 18}}, {'username': {'$ne': '"admin"'}}]}
>>> mongo.transform(tree2)
{'username': {'$regex': '^[aA]'}}

And for SQL:

>>> import functools, operator

>>> class SQLTransformer(lark.Transformer):
...     def __init__(self, table):
...         self.table = table
...     def or_(self, nodes):
...         return functools.reduce(operator.or_, nodes)
...     def and_(self, nodes):
...         return functools.reduce(operator.and_, nodes)
...     def gt(self, nodes):
...         name, number = nodes
...         return self.table.columns[name.value] > int(number.value)
...     def ne(self, nodes):
...         name, string = nodes
...         return self.table.columns[name.value] != string.value
...     def matches(self, nodes):
...         name, regex = nodes
...         return self.table.columns[name.value].regexp_match(regex.value)

>>> sql = SQLTransformer(users)
>>> sql.transform(tree1).compile(compile_kwargs={"literal_binds": True})
users.user_age > 18 AND users.username != '"admin"'
>>> sql.transform(tree2).compile(compile_kwargs={"literal_binds": True})
users.username REGEXP '^[aA]'

That is, we use the recursive traversal to construct the SQLAlchemy expression object from before, utilizing regexp_match for the ~= operator and functools.reduce to apply | and & operators on a list of expressions.

As for the ? prefix I skipped before – it's just a way to inline the results of rules with a single value. If we'd have string: instead of ?string:, a method like ne would get its string token wrapped in a (sub-)tree with that "string" rule inside it, which would complicate parsing, or necessitate a dedicated string visitor like string(self, nodes): return nodes[0]. The same goes for start, expression and comparison – rules whose sole purpose is to group other rules can be seamlessly inlined by Lark to construct simpler trees, provided we add ? before their name.

Extending Python

Let's recap: when we find ourselves inventing our own special syntax for users to code in, we're effectively creating what's known as a domain-specific language. At this point, it's best to pause and ask ourselves whether Python itself can be used, and only proceed if we're sure it can't. Then, the first thing to check is whether regular expressions suffice, and move on only if we're sure they don't. The final step, writing a context-free grammar, will definitely be enough – in fact, Python itself is such a grammar, albeit with a lot of very sophisticated machinery attached. For small cases like our condition language, implementing the necessary semantics is not a big deal; but if we find ourselves handling complex and stateful considerations like functions and classes, there is a more radical alternative we can consider: extending Python itself – as in, adding the ~= operator to its syntax.

This might sound mad, but Python is an open-source project, and there's nothing stopping us from making changes, compiling them, and getting an enhanced interpreter more to our liking. Once again, not necessarily something I'd use in production, but definitely an empowering exercise – so, let's go get the code:

$ git clone https://github.com/python/cpython.git
…
$ cd cpython/
$ git checkout 3.12
$ ./configure # prepare for compilation
…

Like I said, Python is a context-free grammar, which is defined across several files. First, Grammar/Tokens, a list of all the available tokens, is where we append our new symbol:

# …cropped…
AT                      '@'
ATEQUAL                 '@='
RARROW                  '->'
ELLIPSIS                '...'
COLONEQUAL              ':='
TILDEEQUAL              '~='
# …cropped…

And then Grammar/python.gram, which outlines the grammar itself, is where we specify how this symbol is used. We don't have to understand it in depth; one way to navigate such complexity is to look for something similar to what we need, copy it, and make slight modifications. Since we're adding a comparison operator, let's imitate LTE (<=):

# …cropped…
lte_bitwise_or[CmpopExprPair*]: '<=' a=bitwise_or
                                { _PyPegen_cmpop_expr_pair(p, LtE, a) }
rgx_bitwise_or[CmpopExprPair*]: '~=' a=bitwise_or
                                { _PyPegen_cmpop_expr_pair(p, RGX, a) }
lt_bitwise_or[CmpopExprPair*]: '<' a=bitwise_or
                                { _PyPegen_cmpop_expr_pair(p, Lt, a) }
# …cropped…

The syntax is somewhat similar to Lark: rule_name[type]: rule { callback }. Nested rules can be bound to parameters (a=bitwise_or) so as to be passed to the callback, and the nesting structure has to do with precedence: apparently, comparison operators are parsed right before bitwise-or, which is why they reference it inbetween their own tokens. The callback triggered is _PyPegen_cmpop_expr_pair(p, LtE, a), so we invent our own comparator, RGX, and all that's left is to include our new rule in the larger picture:

# …cropped…
compare_op_bitwise_or_pair[CmpopExprPair*]:
    | eq_bitwise_or
    | noteq_bitwise_or
    | lte_bitwise_or
    | rgx_bitwise_or
    | lt_bitwise_or
# …cropped…

Having augmented the grammar, we also need to adjust the code that handles it — and this is tricky, because we don't know in advance which parts of the interpreter deal with what rules (not to mention, this changes with each Python version). So, what I do is:

$ grep -rlIE 'lte_bitwise_or|LtE' . | uniq
./Lib/ast.py
./Lib/test/test_ast/test_ast.py # test
./Include/internal/pycore_ast.h # auto-generated
./Include/internal/pycore_ast_state.h # auto-generated
./Grammar/python.gram # already edited
./Parser/Python.asdl
./Parser/parser.c # auto-generated
./Doc/data/python3.12.abi # documentation
./Doc/library/ast.rst     # documentation
./Python/compile.c
./Python/ast_unparse.c
./Python/ast_opt.c
./Python/Python-ast.c

That is, search (grep) recursively (-r) for filenames only (-l), except binary files (-I), for anything that contains the (extended; -E) regular expression lte_bitwise_or|LtE, since those were the only special identifiers we duplicated and modified in the grammar. If we similarly duplicate and modify the code that handles them, there's no reason Python won't let us include this new syntax, although for now it will have the same functionality as LTE. We can ignore tests, documentation and auto-generated code (always check the top of the file for an indication), which leaves us with 6 files to consider:

# somewhere in ./Lib/ast.py:
    cmpops = {
        "Eq": "==",
        "NotEq": "!=",
        "Lt": "<",
        "LtE": "<=",
        "RGX": "~=",
        "Gt": ">",

# somewhere in ./Parser/Python.asdl:
    cmpop = Eq | NotEq | Lt | LtE | RGX | Gt | GtE | Is | IsNot …

// somewhere in ./Python/ast_unparse.c:
        case LtE:
            op = " <= ";
            break;
        case RGX:
            op = " ~= ";
            break;
        case Gt:
            op = " > ";
            break;

// somewhere in ./Python/ast_opt.c:
            case LtE:
            case RGX:
            case Gt:

The last two are a bit more verbose, so I won't paste them here, but the idea is the same: we find any reference to LtE or lte_bitwise_or, clone it, and swap in RGX or rgx_bitwise_or (we can also "fix" any <= to ~=, but these are just debug prints). In compile.c, we'll keep:

case RGX:
    cmp = Py_LE;
    break;

So as to retain the less-than functionality; if we'd venture to add actual Py_RGX logic, it would take us ever deeper into the interpreter's code, culminating in a new __regex__ magic method, which is a lot more work. And I know, I know, it's all eclectic and haphazard — in my defense, this used to be easier in earlier Python versions; and besides, we're done. At this point, we can compile everything:

$ make regen-token # regenerate the tokens
…
$ make regen-pegen # regenerate the parser
…
$ make regen-ast # regenerate the ast module
…
$ make -j # compile
…

And if we run the resulting ./python, we can do:

>>> 1 ~= 2 # the same as <=, yeah?
True

Transpiling Python

Of course, we're not really done yet; we've added new syntax, but no new functionality. Like I said, doing so in the interpreter is possible, but it's not easy – so instead, let me show you a different approach: transpilation, the conversion of one code into another. In our case, we want username ~= '^[aA]' to become re.match(username, '^[aA]') – as in, convert our special syntax into standard Python, which can be executed without any adjustments to achieve the same goal that our syntax stands for. We already know how such transformations can be achieved; all we need is for this:

>>> import ast
>>> print(ast.dump(ast.parse("username ~= '^[aA]'"), indent=2))
Module(
  body=[
    Expr(
      value=Compare(
        left=Name(id='username', ctx=Load()),
        ops=[RGX()],
        comparators=[Constant(value='^[aA]')]))],
  type_ignores=[])

To become this:

>>> code = "__import__('re').match('^[aA]', username)"
>>> print(ast.dump(ast.parse(code), indent=2))
Module(
  body=[
    Expr(
      value=Call(
        func=Attribute(
          value=Call(
            func=Name(id='__import__', ctx=Load()),
            args=[Constant(value='re')],
            keywords=[]),
          attr='match',
          ctx=Load()),
        args=[
          Constant(value='^[aA]'),
          Name(id='username', ctx=Load())],
        keywords=[]))],
  type_ignores=[])

And notice how I prefer __import__("re").match to re.match, because the latter assumes the availability of a global re name, which might not have been imported, while the former relies only on builtins to work. Well:

>>> class Transpiler(ast.NodeTransformer):
...     def visit_Compare(self, node):
...         if not isinstance(node.ops[0], ast.RGX):
...             return node
...         kw = {"lineno": node.lineno, "col_offset": node.col_offset}
...         return ast.Call(
...             func=ast.Attribute(
...                 value=ast.Call(
...                     func=ast.Name(id="__import__", ctx=ast.Load(), **kw),
...                     args=[ast.Constant(value="re", **kw)],
...                     **kw,
...                 ),
...                 attr="match",
...                 ctx=ast.Load(),
...                 **kw,
...             ),
...             args=[node.comparators[0], node.left],
...             **kw,
...         )

Once again, all this does — for our newly added RGX comparisons only — is replace x ~= y with __import__('re').match(y, x), effectively implementing the functionality of our new syntax using existing language constructs. This way, we only ever need to modify the interpreter to accept new symbols; their purpose, if you will, is merely to signal our intent, so we can intercept their nodes later and infuse them with actual meaning:

>>> def eval_condition(text, **namespace):
...     tree = Transpiler().visit(ast.parse(text))
...     expr = ast.Expression(tree.body[0].value)
...     code = compile(expr, "<string>", "eval")
...     return bool(eval(code, namespace))

>>> eval_condition("username ~= '^[aA]'", username="alice")
True
>>> eval_condition("username ~= '^[aA]'", username="bob")
False

And there you have it: your very own Python++.