Writing/Metaprogramming: An Adventure
Chapter 4 · 26 min read
Dynamic Code Execution
Let's continue our line thinking from the previous chapter: suppose we have a live system that we want to debug without restarting. We figured out how to inject a function call into a particular line inside another function — but in reality, we'd have to ask ourselves how to pass that other function in, and how to make sure it gets all the necessary information from the caller which we wish to debug.
Stack Traversal
The second challenge is actually easier, so let's start with that. At first glance, it seems like we'd have to support arguments — we'd have to invent a way to specify which ones we want, locate them in the caller's code object, and amend our bytecode accordingly, all of which is pretty complicated. So instead, let's stick to a no-arguments function, and make use of frame objects to extricate ourselves from its local scope.
We've already seen these objects when we were working with tracers: when linking
scopes, we actually used frame.f_back to "go up" to our caller's level, and
frame.f_locals to access its local variables. In that case, however, we got
our frame object as an argument to the tracer callback; here, we'll have to
fetch it ourselves, using something like inspect.currentframe(). So if our
caller function had some local name, like x, whose value we'd want to print:
>>> def f():
... x = 1
... y = 2
... return x + y
We could define a g that does so like this:
>>> import inspect
>>> def g():
... frame = inspect.currentframe()
... x = frame.f_back.f_locals["x"]
... print(f"{x=}")
And inject it after x is assigned:
>>> inject(f, 3, g)
>>> f()
x=1
3
More generally, frame objects let us traverse the call stack — that is, the data structure that holds information about which function called what function, which called what function, and so on. Note that this doesn't necessarily correspond to lexical scoping; if we have:
>>> def f():
... def g():
... frame = inspect.currentframe()
... print(f"g caller: {frame.f_back.f_code.co_name}")
... g()
>>> f()
g caller: f
Then the two are indeed the same — but only because g was called from f,
not because it was defined in it. We can see that if we change it to:
>>> def f():
... # … same as before
... return g
And call it from elsewhere:
>>> def h():
... g = f()
... g()
>>> h()
g caller: h
This should be obvious if you remember we're talking about the call stack; but since frames effectively represent scopes, this sometimes leads to confusion. Just make sure you understand that these are runtime scopes, not lexical ones: they represent the chain of execution, not definition.
Dynamic Code Execution
Back to our first challenge. So far, we've been working in the interpreter, so
we had no problem with f and g and h calling each other, since they were
all defined in the same global scope. If we'd have a live application, say
app.py, already running, we wouldn't be able to edit it and have our changes
kick in without restarting the program. Take this simple server, for example:
import flask
app = flask.Flask(__name__)
@app.get("/hello/<param>")
def get_hello(param):
return "Hello, world!"
if __name__ == "__main__":
app.run(port=8000)
And suppose we want to inject a piece of code into it, say from hook.py, that
would let us see what param is:
import inspect
def hook():
frame = inspect.currentframe()
param = frame.f_back.f_locals["param"]
print(f"{param=}")
How can we load it into an already running server, so as to inject it as the
first line of get_hello? Of course, we'd have to augment the server to support
this — but the point is to only do it once, in a generic way, which wouldn't
just inject hook into get_hello, but allow dynamic loading of any code into
any handler.
Conveniently, Python provides the builtin exec function, which takes a string
or a code object, and much like its name implies, executes it in a particular
namespace. To give an example, suppose we have the following code:
>>> code = """
... for i in range(n):
... print(i)
... """
It assumes a global variable n, so to make it work, we can pass in a
dictionary:
>>> exec(code, {"n": 3})
0
1
2
Alternatively, if our code serves to define some interesting value, like a function, we can fetch it from that dictionary after the evaluation is complete:
>>> code = """
... def add(x, y):
... return x + y
... """
>>> namespace = {}
>>> exec(code, namespace)
>>> add = namespace["add"]
>>> add(1, 2)
3
So effectively, we're dealing with a scope: values are read from it and written
to it as our code runs. Now all we need is an endpoint and a convention: a
special request to our server that will execute whatever we send to it in our
application's memory, and inject the result — let's say, a function called
hook — at the appropriate place:
# … same as before
@app.post("/inject/<handler>/<line>")
def post_inject(handler, line):
handler = globals()[handler]
code = flask.request.get_data().decode()
namespace = {}
exec(code, namespace)
hook = namespace["hook"]
globals()["hook"] = hook
inject(handler, int(line), hook)
return ""
def inject(into, line, hook):
# … magic from the previous chapter
Note that every hook we inject will override the previous one, but we can easily make sure that each is created under a unique name by adding a random suffix:
@app.post("/inject/<handler>/<line>")
def post_inject(handler, line):
# … same as before
hook = namespace["hook"]
hook.__name__ = f"hook_{secrets.token_hex(8)}"
globals()[hook.__name__] = hook
# … same as before
And now, watch this — we run our app, so it's live and listening for connections, and:
>>> import requests
>>> # run "python app.py" in the background
>>> requests.get("http://localhost:8000/hello/test")
<Response [200]>
>>> # at this point, nothing should be printed from our app except:
>>> # 127.0.0.1 … "GET /hello/test HTTP/1.1" 200 -
>>> # but if we do:
>>> code = open("hook.py").read()
>>> requests.post("http://localhost:8000/inject/get_hello/7", data=code)
<Response [200]>
>>> # and try again:
>>> requests.get('http://localhost:8000/hello/test')
<Response [200]>
>>> # we'll get this:
>>> # param=test ← that's our hook running in our app!
>>> # 127.0.0.1 … "GET /hello/test HTTP/1.1" 200 -
A Few Words on Security
If you think about it, it's pretty crazy: we have just sent arbitrary code from outside a program, and it seamlessly integrated it into itself, granting us complete freedom to inspect (and even modify) its state.
Needless to say, security-minded people will freak out; if you're one of them, I apologize for the heartburn that must have been building up while you were reading along. Rest assured, I don't recommend adding this code into any real server — it's basically handing the keys to the kingdom to any hacker with an internet connection. To be fair, it can be secured — the injection endpoint can only be available from inside a safe network, only after sufficient authentication, and only to limited personnel. More importantly, though, is that it doesn't really matter, because we're just playing, remember? – as such, we care about what can be done, not whether it should.
Still, since we're talking about security, let's understand the issue a bit
better. Sticklers might argue that no matter how much we harden our endpoint,
any code that uses exec is fundamentally unsafe, because it evaluates user
input as code — it usually follows that we shouldn't use exec, ever. And it's
true that if we can make do without it, we probably should avoid the risk — but
it's equally true that some things can't be done without it, and any risk can be
mitigated.
Suppose our server supports file uploads — let's say, user profile images, which
are stored in an images/ subdirectory next to app.py; and suppose we also
crop and resize them using some functions we keep in utils.py:
project/
app.py
utils.py
images/
# user profile images
Then I can trick the server by uploading a profile image called ../utils.py
that contains Python code, have it be saved to images/../utils.py — that is,
overwrite utils.py – and the next time app.py imports it, my code will be
executed. Admittedly, it's more elaborate than directly sending my code to
/inject and having it run, but the server is still hacked: uploaded files are
a form of user input, and importing code is a way of executing it.
So what do we do? Forbid file uploads, or avoid imports? — of course not; we
just make sure our scheme is otherwise secure, like generating the image names
ourselves, for example based on the username and file's mimetype (so if Alice
uploads image/jpeg data, it would be stored as alice.jpg). At this point, as
is often the case with cybersecurity, we enter a game of cat and mouse: I can
create a user whose name is ../utils, and upload my "image" as
application/x-python-code, resulting in the same overwrite as before. Next, we
might disallow dots and slashes in usernames, and/or limit the accepted
mimetypes to standard image formats only — but who's to say we haven't missed
something else? In truth, no system is ever fully secure.
Sandboxing
Let's play a similar game with exec. Suppose we want to prevent
important_file.txt from being overwritten; executing this code:
>>> code = "open('important_file.txt', 'w').write('hacked!')"
Will definitely overwrite it:
>>> exec(code, {})
>>> print(open("important_file.txt").read())
hacked!
But ask yourself why: this code actually relies on a global variable, open —
albeit a builtin one. Where did it come from, if we passed in an empty
dictionary?
Turns out, Python adds all these things by default, under the special
__builtins__ key:
>>> namespace = {}
>>> exec("x = 1", namespace)
>>> namespace
{'__builtins__': {…, 'open': <builtin open function>, …}, 'x': 1}
So to prevent it, we can pass in an empty __builtins__:
>>> exec(code, {"__builtins__": {}})
NameError: name 'open' is not defined
This way of executing code is actually quite secure, since without builtins,
there isn't much damage we can do. On the other hand, this might be too limited:
if you recall our simple hook from before, which merely printed the parameter,
it wouldn't work either, on account of print being one of these builtins — as
well as other completely harmless and arguably indispensable utilities like
int or len.
An alternative approach, then, would be to pass in a copy of the builtins dictionary, but remove any sharp objects from it, so to speak; in cybersecurity jargon, this is known as a sandbox:
>>> import builtins
>>> sandbox = vars(builtins).copy()
>>> del sandbox["open"]
>>> # so this works:
>>> exec("print('hello')", {"__builtins__": sandbox})
hello
>>> # but this doesn't:
>>> exec("open('important_file.txt')", {"__builtins__": sandbox})
NameError: name 'open' is not defined
Here, however, we enter the same cat-and-mouse game we've seen before — because
if I really wanted to open a file in Python, I could figure out ways to do it
without using open:
>>> code = """
... import pathlib
... path = pathlib.Path("important_file.txt")
... path.write_text("hacked!")
... """
>>> exec(code, {"__builtins__": sandbox})
>>> print(open("important_file.txt").read())
hacked!
To disallow imports, we can remove the __import__ builtin, which is what makes
import statements tick behind the scenes:
>>> del sandbox["__import__"]
>>> exec(code, {"__builtins__": sandbox})
ImportError: __import__ not found
But once again, if you recall our simple hook, it required inspect to acquire
a frame object. If we disallow imports (which is recommended; if we consider
builtins to be too powerful, what is there to say about the entire standard
library?), we need to provision the code with all the utilities we do allow,
like passing in the frame object ourselves:
>>> param = "test"
>>> code = """
... param = frame.f_locals["param"]
... print(f"{param=}")
... """
>>> import inspect
>>> frame = inspect.currentframe()
>>> exec(code, {"__builtins__": sandbox, "frame": frame})
param=test
This, however, is easier said than done; not only because we have to guess all the provisions that will be necessary in advance, and whoever's writing the hook needs to know all the global variables that will be magically available to their code when it runs — but because oftentimes, an attacker might use these very values to escape the sandbox:
>>> code = """
... imp = frame.f_builtins["__import__"]
... pathlib = imp("pathlib")
... pathlib.Path("important_file.txt").write_text("hacked!")
... """
>>> exec(code, {"__builtins__": sandbox, "frame": frame})
>>> print(open("important_file.txt").read())
hacked!
With frame objects it's particularly easy, since they're so well-connected to the rest of the runtime; but we can figure out a way from literally any object, provided that it has at least one custom method:
>>> class A:
... def f(self):
... pass
>>> a = A()
>>> code = """
... imp = a.__class__.f.__builtins__["__import__"]
... pathlib = imp("pathlib")
... pathlib.Path("important_file.txt").write_text("hacked!")
... """
>>> exec(code, {"__builtins__": sandbox, "a": a})
>>> print(open("important_file.txt").read())
hacked!
In fact, we can escape even with no object at all; go ahead and see if you can figure out how, it's a fun puzzle. Suppose we manage to raise an exception: an error object has a traceback, which has a frame, which we'll be able to use to crawl out of our sandbox like we did before.
Now, raising an error is easy; it's catching it that's hard. Doing so with the
regular except Exception as error won't work, since it relies on the
Exception builtin, and we removed that (and all other exception classes) from
our builtins. What we can do is use a context manager's __exit__ method: its
signature is such that if an error is raised inside its context, that
information is passed in. But now, creating a context manager won't work: to
define a class we need the __build_class__ builtin, or at least type, both
of which were removed as well. This, however, is something that can be recreated
from any value: after all, everything in Python is an object, and all objects
store their types in __class__, which are also objects, and their
(meta-)types are type. To recap: even with no builtins at all, we can still
acquire the standard metaclass, use it to construct a context manager, and use
that to catch an error out of which we can break free:
>>> code = """
... type = (1).__class__.__class__
... def __enter__(self):
... return self
... def __exit__(self, exception, error, traceback):
... imp = error.__traceback__.tb_frame.f_back.f_builtins["__import__"]
... pathlib = imp("pathlib")
... pathlib.Path("important_file.txt").write_text("hacked!")
... return True # suppress the error
... A = type("A", (), {"__enter__": __enter__, "__exit__": __exit__})
... with A():
... 1 / 0
... """
>>> exec(code, {"__builtins__": {}}) # no builtins at all!
>>> print(open("important_file.txt").read())
hacked!
Web Notebooks
So yeah — all things considered, I don't recommend sandboxing; unless it's
entirely limited, it's probably not limited at all, only a matter of how well
our hacker knows Python. But that's not to say that I'm against exec: the way
I think about it is as a powerful tool to run trusted code.
To wit, one of the most popular frameworks in the Python ecosystem is built around this functionality: Jupyter Notebooks (formerly known as IPython). It's a simple, brilliant idea: what if we could write and run our code in a web interface?
The benefits are many: we can seamlessly work on a remote machine in the cloud, which can be much more powerful than our own computer, come equipped with complex configurations we don't have to set up ourselves, have access to private resources we wouldn't want to download, and the list goes on.
But that's just the "backend" part, running the Python; the "frontend" part, of doing so in the browser, opens up a whole world of fascinating possibilities. For all of Python's strengths, it can't really compete — no language can — with web technologies when it comes to graphic user interfaces and visualizations. So not only can our Python sessions be had with almost no setup, they can now also be annotated with text, include images and plots, and be exported — either as static assets, or even as dynamic notebooks that can be calibrated and experimented with: tutorials, research papers or entire books can be written interactively, all thanks to that one little builtin.
Let's implement something of this sort ourselves. The main concept behind this
project is called a kernel: a Python interpreter exposed as a web server,
which receives the input we'd normally write after the >>> prompt as a
request, and returns the evaluated result we'd normally see printed as a
response. Actually, this is surprisingly simple (given that we won't implement
the UI itself), especially since we'll start with a handicap: rather than
displaying the result of the last expression, we'll require the input to
explicitly print the values it wants to see. And so:
import builtins
import flask
app = flask.Flask(__name__)
scope = {"__builtins__": vars(builtins).copy()}
@app.post("/")
def post_evaluate():
code = flask.request.get_data().decode()
results = []
def print(arg):
results.append(str(arg))
scope["__builtins__"]["print"] = print
exec(code, scope)
return "\n".join(results)
if __name__ == "__main__":
app.run(port=8000)
First, we create a persistent scope in which our executions will run; and then
we execute whatever input we receive into this scope. The only tricky part about
this snippet is that it replaces the builtin print function with our own
version — one that, rather than emitting the output to the screen, which would
happen server-side, captures it in a list and returns a concatenated result at
the end, so it can be displayed by the client. To demonstrate how it works:
>>> def run(code):
... response = requests.post("http://localhost:8000/", data=code)
... if response.content:
... print(response.content.decode())
>>> # run "python app.py" in the background
>>> run("x = 1") # no output
>>> run("y = 2") # no output
>>> run("print(x + y)")
3
Consider what happens here. For convenience, we've wrapped the request-response
boilerplate in a run function — but whenever we call it, we're actually
sending our code into a different program, which evaluates it on our behalf.
The first two assignments have no visible effect, but they definitely happen,
because the third one references those values, and we get the correct result.
Entire code blocks with multiple outputs are also supported:
>>> run("""
... n = x + y
... for i in range(n):
... print(f"line {i}")
... """)
line 0
line 1
line 2
And as you can see, our memory is still intact, since n is derived from x
and y. Of course, this code doesn't handle concurrent requests well (print
will be overwritten and the outputs will get mixed up), doesn't adhere to
print's actual signature, doesn't support cool tables, images or plots; but
whip up some basic HTML to make these calls in Javascript, and you will have a
full-fledged Python — in the browser (!) — in less than 20 lines of code (!!!).
Abstract Syntax Trees
But let's play some more. Real Jupyter notebooks are much smarter; you don't
have to use print, yet somehow, the last statement that you write gets
displayed, much like it would in the regular Python interpreter. In other words,
something like this:
>>> run("""
... x = 1
... y = 2
... x + y
... """)
Would be enough to get 3. How?
Well, in addition to exec, Python also provides the eval builtin function.
It can't handle statements like assignment, or entire blocks of code — but if we
give it a string with an expression, it can run it all the same and give back
the result:
>>> exec("1 + 2") # no result
>>> eval("1 + 2")
3
>>> # however:
>>> eval("""
... x = 1
... y = 2
... x + y
... """)
SyntaxError: invalid syntax
Of course, we can always resort to clever tricks in order to cram multiple
statements in a single expression — using the walrus operator instead of
assignment, the ternary operator instead of ifs, comprehensions instead of
loops, lambdas instead of functions; even classes can be created dynamically
with type. The above, for example, we can rewrite as:
>>> eval("(x := 1, y := 2, x + y)[-1]")
3
But that's hardly a way to live. Instead, we want to figure out how to combine
eval on the last statement with exec on all the rest — and it's not at all
obvious. Naively splitting off the last line won't do, since expressions can
easily span multiple lines; and parsing the actual syntax… well, how do we go
about that?
Luckily, Python has us covered once again: this time with the standard ast
module, which stands for abstract syntax tree — a data structure
representing parsed code, denoting what's nested in what, the type of each token
and so on. All we need to do is pass a string into it and get a handy tree
object:
>>> import ast
>>> tree = ast.parse("x = (1 + 2) * 3")
>>> tree
<ast.Module object at 0x…>
>>> # this is not very informative, so let's do:
>>> print(ast.dump(tree, indent=2))
Module(
body=[
Assign(
targets=[
Name(id='x', ctx=Store())],
value=BinOp(
left=BinOp(
left=Constant(value=1),
op=Add(),
right=Constant(value=2)),
op=Mult(),
right=Constant(value=3)))],
type_ignores=[])
As you can see, ast considers the whole thing a module, and within its body —
an assignment, whose targets (since there can be multiple) are the name x, and
whose value is a binary operator. The order of precedence, which follows from
the parentheses, is expressed through nesting: the assigned value is a
multiplication of another binary operation — this time, an addition of the
constants 1 and 2 — with the constant 3. If we were to try and decode even
this simple case ourselves, taking care of all this would require considerable
effort. In other words, we're off to a great start; but it still remains to be
seen how we can process this complex, recursive structure.
To demonstrate, consider the following use case: given some code, we want to
extract all the identifiers it uses. It's not so easy; a regular expression
like /[a-zA-Z_][a-zA-Z0-9_]*/ would yield a lot of false positives, since any
such token being an identifier or not depends entirely on its context. If we
take this, for example:
>>> code = "x = y.z + 's'"
Then x and y are identifiers, but z is an attribute, and s is part of a
string. In the AST, it's clear enough:
>>> tree = ast.parse(code)
>>> print(ast.dump(tree, indent=2))
Module(
body=[
Assign(
targets=[
Name(id='x', ctx=Store())],
value=BinOp(
left=Attribute(
value=Name(id='y', ctx=Load()),
attr='z',
ctx=Load()),
op=Add(),
right=Constant(value='s')))],
type_ignores=[])
But how do we work with this?
Transformers
There's a design pattern intended for this express purpose: it's called a
visitor. We won't discuss its details, however, since ast already
implements it for us as NodeTransformer; all we have to do is subclass it,
define a few special methods, and ask it to "visit" our tree. The transformer
will take care of the complicated traversal, invoking our methods only when it
encounters those parts of the tree, known as nodes, that we care about — and
giving us an opportunity to inspect (or even change) them.
The way it knows which methods to call on which nodes is encoded in each
method's name: it has to start with visit_, followed by the node type we'd
like it to handle — so in our case, Name. Such methods then receive a single
node object of that type; and note that the node structure will vary
accordingly, so to process it we need to be intimately familiar with Python's
data model. Alternatively, we can just print the tree like we did above, and see
that a Name node has an id and a ctx (indicating whether it's being
assigned or resolved). For now, we're only interested in collecting the first —
so let's give it a try:
>>> class Transformer(ast.NodeTransformer):
... def visit_Name(self, node):
... print(node.id)
... return node
>>> Transformer().visit(tree)
x
y
Going back to our actual problem, we need to do something more sophisticated:
detect the last expression (that's an Expr node), extract it, and ideally
remove it from the tree (by returning None instead of the original node), so
that we can exec the transformed code and eval the last expression
separately. Since we can't know in advance which expression is going to be the
last, let's go over the tree twice – once to record it, and once to remove the
last expression:
>>> class Transformer(ast.NodeTransformer):
... def __init__(self, remove=None):
... self.remove = remove
... self.last = None
... def visit_Expr(self, node):
... self.last = node
... return None if node is self.remove else node
>>> tree = ast.parse("""
... x = 1
... y = 2
... x + y
... """)
>>> t1 = Transformer()
>>> t1.visit(tree)
>>> t2 = Transformer(remove=t1.last)
>>> t2.visit(tree)
>>> print(ast.dump(t2.last, indent=2))
Expr(
value=BinOp(
left=Name(id='x', ctx=Load()),
op=Add(),
right=Name(id='y', ctx=Load())))
>>> print(ast.dump(tree, indent=2))
Module(
body=[
Assign(
targets=[
Name(id='x', ctx=Store())],
value=Constant(value=1)),
Assign(
targets=[
Name(id='y', ctx=Store())],
value=Constant(value=2))],
type_ignores=[])
Looks good. To execute it, though, we can't just pass the tree or the expression
node into exec or eval — both accept only strings and code objects. We need
to turn it into the latter, using the builtin compile function, which is happy
to work with ASTs (although it requires a couple more arguments, namely a
filename and the compilation mode):
>>> code = compile(tree, "<string>", "exec")
>>> expr = compile(t2.last, "<string>", "eval")
TypeError: expected Expression node, got Expr
Oops; that last bit is somewhat confusing, and took me a moment to figure out.
Turns out Expr is an expression statement, and eval (or compile in eval
mode) expects a plain expression. Once we know this, it's a simple conversion:
>>> expr = compile(ast.Expression(t2.last.value), "<string>", "eval")
And we're ready:
>>> namespace = {}
>>> exec(code, namespace)
>>> eval(expr, namespace)
3
A Different Approach
This was a cool exercise, but there are a lot of edge cases it doesn't address. It's enough to add a simple if-statement to break everything:
>>> tree = ast.parse("""
... x = 1
... y = 2
... if True:
... x + y
... """)
>>> # … same as before
>>> code = compile(tree, "<string>", "exec")
ValueError: empty body on If
Because by extracting x + y, we leave the if True hanging. We could try
replacing the last expression with a no-op node (that is, Pass) — and although
our code wouldn't break, it wouldn't be right either, since if we change it to
if False, we'd still evaluate the last expression, it being treated separately
and all. Accounting for all the possible contexts from which we might be
extracting an expression is no easy task – so instead, let's do something else.
You know how in the Python interpreter, we have the magic _ variable, which
stores the value of the last result?
>>> 1 + 2
3
>>> _
3
If we can implement a similar logic, we'll be able to execute the original
code-block as is, and simply fetch _ to get the last value, whatever it may
be. To do that, we'd have to replace Expr nodes with Assign ones; and since
we're talking about expression-statements only, there's no danger of us breaking
any syntax where plain expressions are used. The only new detail we have to take
into account when creating new nodes is that they should have a line number and
column offset (so that if an error occurs, tracebacks can intelligibly point out
the offending tokens) — but those we can just copy over from the original node:
>>> class Transformer(ast.NodeTransformer):
... def visit_Expr(self, node):
... return ast.Assign(
... targets=[ast.Name(
... id="_",
... ctx=ast.Store(),
... lineno=node.lineno,
... col_offset=node.col_offset,
... )],
... value=node.value,
... lineno=node.lineno,
... col_offset=node.col_offset,
... )
>>> Transformer().visit(tree)
>>> code = compile(tree, "<string>", "exec")
>>> namespace = {}
>>> exec(code, namespace)
>>> namespace.get("_")
3
Incidentally, when we integrate this with the kernel we developed earlier, we
can start adding all sorts of fun features based on the result type, since it's
no longer just a string to display. I mean, if it's a string, sure — we'll
display it; but if it's a pandas dataframe, which looks nicer as a table, we
can return it as a dictionary, and have our Javascript frontend render it more
beautifully than mere text:
import pandas as pd
@app.post("/")
def evaluate_handler():
code = flask.request.get_data().decode()
tree = Transformer().visit(ast.parse(code))
exec(compile(tree, "<string>", "exec"), scope)
result = scope.get("_")
if isinstance(result, pd.DataFrame):
return {"type": "table", "data": result.to_dict("records")}
return {"type": "string", "data": str(result)}
Assuming our UI uses React, it might look like this:
import React from "react";
const Result = ({ type, data }) => {
if (type === "string")
return <p>{data}</p>;
const columns = Object.keys(data[0]);
return (
<table>
<thead>
<tr>
{columns.map(col => (
<th key={col}>{col}</th>
))}
</tr>
</thead>
<tbody>
{data.map((row, rowIndex) => (
<tr key={rowIndex}>
{columns.map(col => (
<td key={col}>{row[col]}</td>
))}
</tr>
))}
</tbody>
</table>
);
};
What's more, if the code is attempting to plot something with matplotlib, we
can extract the image and attach it to our response as well:
import base64
import io
import matplotlib.pyplot as plt
@app.post("/")
def evaluate_handler():
# … same as before
if isinstance(result, pd.DataFrame):
response = {"type": "table", "data": result.to_dict("records")}
else:
response = {"type": "string", "data": str(result)}
fig = plt.gcf() # get active plot
if fig.get_axes(): # plot is not empty
image = io.BytesIO()
plt.savefig(image, format="png", bbox_inches="tight")
image_base64 = base64.b64encode(image.getvalue()).decode()
response["image"] = f"data:image/png;base64,{image_base64}"
return response
We use a BytesIO to trick matplotlib into saving the image into memory, then
convert the bytes to base64 to be able to send them as a JSON-friendly data
URI, and in the UI:
const ResultWithImage = ({ type, data, image }) => {
return (
<div>
{image && (
<img src={image} />
)}
<Result type={type} data={data} />
</div>
);
};
I'll stop here, but you get the idea: combining the best of both worlds, we get Python's programming environment with Javascript's visualization capabilities.