Zygo — a warm sandbox per request

Zygo forks a warm, sandboxed interpreter for every request: 1.4 ms through its API, and every request starts from a process that has never served one. Rootless, OCI images, and no daemon to install: the one long-lived process is a supervisor under your own user, not a system service.

This page is the first page of the Zygo book, also on the web at mhmtskrc2.github.io/zygo.

Try it now, once Zygo is installed — one program in a fresh sandbox, thrown away when it exits:

zygo run python:3.12-slim python3 -c 'print("hello")'   # pulls the image the first time

And what Zygo is for — a warm function, forked for every request:

zygo serve ./handler.py --name resize        # a warm zygote: ~150 ms, once
zygo exec resize '{"url": "..."}'            # a fresh, sandboxed process: 2.8 ms (1.4 by API)
zygo exec resize '{"url": "..."}'            # and again, from the same clean copy

Why

A warm worker that serves many requests is fast and dirty: request n sees whatever request n-1 left behind. A container per request is clean and slow. Zygo is the third thing. zygo serve starts an interpreter, lets it do its imports, and parks it inside a sandbox. zygo exec forks it. Each child has its own cgroup, deadline and secrets, and is thrown away afterwards.

The same import-heavy Python script, run fresh for each call on one host (a Linux 6.8 VM, 2 vCPU, aarch64; chapter 25 has the method, and make bench-embed repeats it):

Most of the 70.8 ms is Python importing those sixteen modules: the sandbox itself is about 3.6 ms, and python3 -c pass inside one is 12 ms. Through the HTTP API rather than the CLI, a warm request is 1.44 ms usually and 10.5 ms for 1 in 100, and one function sustains 1,108 requests a second. zygo bench all reproduces every number on your own machine.

The zygote idea is older than Zygo: Android starts apps that way, and serverless research forked handlers from a pre-imported process in SOCK (Oakes et al., USENIX ATC 2018) and restored them from a snapshot in Catalyzer (Du et al., ASPLOS 2020). Zygo’s part is the packaging — one static binary, rootless, every limit on, an agent protocol any language can speak — not the idea.

Why not bubblewrap, nsjail, nono, sandbox-runtime or kern?

They are good at what they do, and none of them keeps a sandboxed process warm and forks it per request.

Similar projects compares all of them but sandbox-runtime, with measurements against nsjail and kern.

The boundary

The default backend, ns, is the host kernel: namespaces, cgroup v2, a seccomp allowlist, Landlock, no capabilities and a read-only root. That is the right wall for code that is semi-trusted — your customers’ scripts, an agent’s tools. A kernel bug is a way through it, as it is for every container. The same spec also runs on gvisor (a kernel in user space) or vm (libkrun), one-shot only. Neither is a full wall for hostile code yet. Both lack a network. A rootless gvisor cannot enforce its limits, and vm has none inside the guest and a kernel you build yourself. For anonymous code, the honest answer today is a separate machine.

make escape-linux attempts 21 of the vectors in the threat model, with 0 escapes, and every syscall number is swept against the seccomp profiles. The same chapter lists the tenant-against-tenant vectors not attempted yet, and says where the boundary is weaker than it looks. No external audit has been done. Fork safety, question by question covers what a fork shares with its parent and what it does not.

Install

# Linux, x86_64 or aarch64: one static binary, checked against the release's checksums
url=https://github.com/mhmtskrc2/zygo/releases/latest/download
curl -fsSLO "$url/zygo-$(uname -m)-unknown-linux-musl.tar.gz"
curl -fsSL "$url/SHA256SUMS" | sha256sum -c --ignore-missing
tar xzf zygo-*-unknown-linux-musl.tar.gz && sudo install -m 0755 zygo-*/zygo /usr/local/bin/

brew install mhmtskrc2/zygo/zygo      # macOS: the shim, and a Linux VM it manages
cargo install zygo-cli                # from source

Linux needs kernel 5.3 or newer (6.1 recommended), unprivileged user namespaces and delegated cgroup v2. On a Mac every command runs in a Linux VM, about 22 ms away. There is also a signed container image. Getting started covers all of it, signatures included.

zygo doctor                           # can this host run sandboxes? prints the fix if not
zygo run --mem 128M --timeout 10s python:3.12-slim python3 -c 'print("hello")'   # pulls the image
echo 'def handler(event): return {"got": event}' > handler.py
zygo serve ./handler.py --name echo && zygo exec echo '{"n": 1}'   # serve never pulls
zygo stop --all                       # everything serve started; on a Mac, the VM too

What else is in the box

• One file per project. sandbox.toml declares functions, their images, limits, network and secrets; zygo up deploys it blue/green and pins image digests in zygo.lock. Chapter 20
• Every limit is on by default — memory, CPU, pids, wall clock, scratch, open files — and the deadline kills the request’s whole process tree.
• The network is off by default. egress is an allowlist of names; private ranges and the cloud metadata address stay closed. Chapter 14
• Secrets are files, written from outside the sandbox for one request: never environment variables, never in the warm process’s memory.
• Any language. Python and Node agents ship; anything else is a fresh process per request in a held sandbox (about 1.4 ms), or an agent of your own against the protocol.
• For programs: an HTTP API, dependency-free Python and Node clients (zygo-sdk), an Elixir client (zygo_sdk), and an MCP server. Chapter 17
• For a multi-tenant product: tenants with their own tokens and budgets, a script or a whole workspace (a tar) sent with each request, streamed output, and cancellation — all over the API.

Building on it? Read 13 (warm functions), then 17 (the API, the SDKs, MCP), then 23 (the threat model), in that order.

Status

v0.1.x: one machine; Linux in production, macOS for development. The warm path is ns-only by decision (ADR 0002). ROADMAP.md says what comes next.

Not for: an interactive session or a REPL (a request is one call, not a shell you keep); more than one machine (no scheduler, no cluster); GPUs; Windows without WSL2.

More

Contributing · Security policy · Changelog · Roadmap · Licence: Apache-2.0

Zygo is not affiliated with Zygo Corporation, the metrology company.

The Zygo book

← Previous page: the project README — the book’s first page, and the five-minute version of everything below.

All of Zygo’s documentation, in one place, written to be read from the start.

It begins with Container 101: what the Linux kernel gives you to build a sandbox, and what Docker builds from it. Then it explains Zygo — how it works, where it saves time and memory, and how it compares with everything around it. Then it teaches you to use it, and ends with the full reference: every command, every flag, every field, every file and every exit code. You do not need to know any of it already. If you can use a shell and you know what a process is, you can read every page.

Each section is short on purpose — a few sentences, a picture where one helps, and a link to the detail. Read Parts I and II once; after that, the headings work as a reference. This book is the source of truth: when Zygo changes, the book changes in the same commit (AGENTS.md).

Part I — Container 101

Part II — Zygo, explained

Part III — Using Zygo

Part IV — Reference

Part V — Security and speed

Part VI — Decisions

Where to start

  first time here? ────────────────▶ the README, then this page
  new to containers? ──────────────▶ Part I, then Part II
  know Docker, new to Zygo? ───────▶ chapters 6, 7, 10, then 11
  want to use it today? ───────────▶ chapter 11, then 13
  looking something up? ───────────▶ Part IV
  deciding whether to trust it? ───▶ chapter 23

Three sentences to hold on to

1. A container is not a thing in the kernel. It is a normal process with several limits put on it; the kernel has no idea what a “container” is.
2. The limits are cheap; the tools around them are not. Setting up the limits takes about a millisecond. Docker takes hundreds, and most of that is programs talking to programs.
3. Zygo removes the tools, then removes the start-up. It sets the limits up itself, in one process, and then keeps a ready copy of your program waiting, so a request costs one fork().

How the numbers are used

Every number about Zygo in this book comes from chapter 25, which names the machines and the commands. Numbers about other projects are their own claims or commonly measured ranges, and they are marked that way. zygo bench all repeats Zygo’s numbers on your own machine.

1. The kernel and the process

This chapter is about the basics that come before namespaces and containers: the kernel, system calls, processes, fork and exec, users and root. They are old and simple, and they are the ground everything else stands on. A namespace, a cgroup, a container and a Zygo sandbox are all just new rules added to these same old pieces. If a later chapter feels confusing, the missing piece is usually here.

  chapter 1:  kernel · syscalls · processes · fork/exec · users · capabilities
                  ▲
  chapter 2–4:    │ namespaces, cgroups and filters are rules ON these pieces
                  ▲
  chapter 5:      │ a container = one process + those rules + a file tree
                  ▲
  chapter 6:      │ a Zygo sandbox = the same, with no daemon, kept warm

The kernel and user space

The kernel is the one program that talks to the hardware. It owns the memory, the CPU time, the disks and the network cards, and it decides who gets which. Everything else — your shell, Python, a database — runs in user space and cannot touch the hardware directly. When a program in user space needs something, it has to ask the kernel. That split is the first and oldest boundary on the machine, and every sandbox in this book is a way of making the kernel say “no” more often.

 USER SPACE     ┌───────┐  ┌────────┐  ┌──────────┐  ┌──────────┐
 (programs)     │ shell │  │ python │  │ postgres │  │ your job │
                └───┬───┘  └───┬────┘  └────┬─────┘  └────┬─────┘
                    │ syscalls │            │             │
 ═══════════════════╪══════════╪════════════╪═════════════╪══════ the boundary
                    ▼          ▼            ▼             ▼
 KERNEL         ┌─────────────────────────────────────────────────┐
                │ processes · memory · files · network · devices  │
                └─────────────────────────────────────────────────┘
 HARDWARE          CPU          RAM          disk         network card

System calls

The way a program asks the kernel for something is a system call, or syscall. Opening a file is openat, reading from it is read, starting a process is clone, and there are about 450 of them on a modern Linux. A syscall is a small, well-defined door: the program puts a number and some arguments in place and the kernel answers. Because every request passes through these doors, the kernel can check each one. It is also why the number of doors matters for safety — each is code in the kernel that a hostile program can try to confuse.

Processes

A process is a running program: its memory, its open files, its user, and the kernel’s notes about it. Every process has a number, the pid, and a parent. The first process on the machine is pid 1, and every other one descends from it, so the processes form a tree. ps -ef --forest shows that tree. A sandbox is, at the end, a branch of this tree that the kernel treats differently from the rest.

fork and exec

Linux starts a new program in two steps. fork makes a copy of the calling process — same memory, same open files — and both copies carry on from the same line. exec (execve) then replaces the copy’s memory with a new program loaded from disk. A shell that runs ls forks itself, and the child execs ls. The split looks strange, but it gives you a moment between the two steps where the child can change its own situation — close files, change user, enter a sandbox — before the new program starts. Every sandbox in this book does its work in that moment.

 shell (pid 100)
    │
    │ fork()
    ├──────────────────▶ copy of shell (pid 101)
    │                        │
    │                        │  ◀── the moment: change user, close files,
    │                        │      enter namespaces, join a cgroup,
    │                        │      install filters …
    │                        │
    │                        │ execve("/bin/ls")
    │                        ▼
    │                     ls (still pid 101)
    │                        │ exit
    ▼ wait() ◀───────────────┘

Copy-on-write

A fork does not really copy the memory. Parent and child share the same physical pages, marked read-only, and a page is copied only when one of them writes to it. So forking a 50 MB process takes well under a millisecond and costs almost no new memory. The child still behaves as if it had its own full copy: what it writes, the parent never sees. This one trick is the base of Zygo’s warm path, and chapter 6 comes back to it.

 right after fork                   after the child writes to page B
 ────────────────                   ────────────────────────────────
     parent   child                      parent   child
        │       │                           │       │
        ▼       ▼                           ▼       ▼
     ┌───┬───┬───┐                       ┌───┬───┬───┐   ┌────┐
     │ A │ B │ C │                       │ A │ B │ C │   │ B' │
     └───┴───┴───┘                       └───┴───┴───┘   └────┘
   one set of pages, shared           parent sees  A  B  C
   and marked read-only               child sees   A  B' C   (one page copied)

Users and root

Every process runs as a user, which the kernel knows as a number, the uid. Files have an owner uid and permission bits, and the kernel checks them on every access. uid 0, root, has always been special: for most checks, root simply passes. That made root the prize of every attack, because one bug in a root program gave away the whole machine. Much of the history of Linux security is the slow work of making root less all-powerful.

Capabilities

Capabilities split root’s power into about forty named pieces. CAP_NET_ADMIN lets a process change the network, CAP_SYS_ADMIN lets it mount file systems and much more, CAP_KILL lets it signal anyone. A process can hold some pieces and not others, and it can drop the ones it does not need, for good. A sandbox usually drops all of them. Zygo does: a program inside it has an empty capability set.

Everything is a file, and /proc

Linux shows a lot of its state as files. /proc is a file system the kernel makes up on the fly: /proc/1234/ describes process 1234, /proc/meminfo describes memory. /sys does the same for devices and, as chapter 3 shows, for control groups. This matters for sandboxes in two ways. Many controls are set by writing a small file, which is fast and needs no special tool. And a sandbox must be careful what parts of /proc and /sys it shows, because some of those files are doors of their own.

One kernel

Here is the fact the rest of the book keeps coming back to. On one Linux machine, every process — in a container or not — talks to the same kernel. Namespaces, control groups and filters are all rules inside that kernel. If the kernel itself has a bug that lets a process break its rules, every rule fails at once. That is why some sandboxes add a second kernel (gVisor) or a whole virtual machine (Firecracker, Zygo’s vm backend); chapter 10 compares them.

  containers / Zygo `ns`        gVisor                   microVM (Firecracker, Zygo `vm`)
  ──────────────────────        ──────                   ────────────────────────────────
  ┌─────┐ ┌─────┐ ┌─────┐       ┌─────┐                  ┌─────┐          ┌─────┐
  │ app │ │ app │ │ app │       │ app │                  │ app │          │ app │
  └──┬──┘ └──┬──┘ └──┬──┘       └──┬──┘                  └──┬──┘          └──┬──┘
     │       │       │          ┌──▼─────────────┐       ┌──▼────────┐    ┌──▼────────┐
     │       │       │          │ user-space     │       │ guest     │    │ guest     │
     │       │       │          │ kernel         │       │ kernel    │    │ kernel    │
     │       │       │          └──┬─────────────┘       └──┬────────┘    └──┬────────┘
  ┌──▼───────▼───────▼──┐       ┌──▼─────────────┐       ┌──▼─────────────────▼───────┐
  │     host kernel     │       │  host kernel   │       │  host kernel + KVM         │
  └─────────────────────┘       └────────────────┘       └────────────────────────────┘
  one kernel bug = all out      a bug must pass two      a bug must pass the guest
                                                         kernel AND the hardware wall

2. Namespaces

A namespace changes what a process can see. It does not limit how much it can use — that is control groups, the next chapter.

What a namespace is

Normally every process sees the same machine: the same list of processes, the same files, the same network, the same host name. A namespace gives a group of processes their own copy of one of those things. Inside a new PID namespace, for example, a process sees only its own children and thinks it is pid 1. Nothing is copied and nothing is emulated; the kernel just keeps a separate table and answers from it. Linux has eight kinds, and each one can be used alone or with the others. A “container” is simply a process that has its own copy of most of them.

            the host sees                         the sandbox sees
 ┌──────────────────────────────────┐     ┌─────────────────────────────┐
 │ pids   1 systemd … 4200 zygo     │     │ pids   1 python   2 sh      │
 │        4201 python  4202 sh      │     │                             │
 │ files  /home /etc /var …         │     │ files  / of the image only  │
 │ net    eth0 wlan0 lo  + LAN      │     │ net    lo                   │
 │ name   my-laptop                 │     │ name   sandbox              │
 │ users  0 root … 1000 you         │     │ users  0 root  (= you, 1000)│
 └──────────────────────────────────┘     └─────────────────────────────┘
        same processes 4201 and 4202 — two different views of them

Mount namespace

The mount namespace gives a process its own list of mounted file systems. It can mount and unmount things, and the rest of the machine does not see it. This was the first namespace, added in 2002, and it is why the flag is still called just CLONE_NEWNS — nobody expected others. A sandbox uses it to build a whole new file tree from an image, then switch its root to that tree, so the host’s files are simply not there. Chapter 4 explains how the switch is done.

PID namespace

The PID namespace gives a process its own process numbers. The first process inside becomes pid 1, and it can only see and signal processes in the same namespace, or in namespaces below it. From outside, the same processes are still visible, with their normal host numbers. So a sandboxed program cannot list, watch or kill anything on the host. When pid 1 of the namespace exits, the kernel kills everything else inside it, which makes clean-up easy.

Network namespace

The network namespace gives a process its own network: its own interfaces, its own addresses, its own routes and its own firewall rules. A new one has only a loopback interface (lo), so by default the process can reach nothing but itself. To give it more, something outside has to connect it — a virtual cable to the host, or a program that moves packets in user space. Chapter 4 covers both, and Zygo’s choice.

UTS namespace

UTS is an old name for “the system’s name”. This namespace gives a process its own host name and domain name. It is the smallest of the eight and hides no secret; it mostly stops a program from learning the host’s name, or changing it. Container tools set it so that hostname inside prints something sensible.

IPC namespace

IPC means “inter-process communication”. This namespace gives a process its own System V message queues, shared memory segments and POSIX message queues. Without it, two programs on one machine could meet through a shared memory segment with a guessable key. It is old and rarely thought about, but a sandbox that skips it leaves a small side door open.

User namespace

The user namespace gives a process its own list of users. Inside it, a process can be uid 0 — root — while outside it is still your own normal uid. A uid map (/proc/<pid>/uid_map) says which inside number matches which outside number. Root inside a user namespace has full capabilities, but only over things that belong to that namespace: its own mounts, its own network, never the host’s. This is the namespace that lets a normal user create all the others, and it is what makes rootless containers — and all of Zygo — possible.

  inside the sandbox          uid_map            on the host
  ──────────────────     ─────────────────     ──────────────
  uid 0    "root"   ───▶   0     1000   1  ───▶  uid 1000 (you)
  uid 1000 "app"    ───▶   (a second line, via newuidmap, if the host allows)
  any other uid     ───▶   not mapped: shows as "nobody", owns nothing

  root inside  = every capability over the sandbox's own mounts, network, …
               = no power at all over anything that belongs to the host

Cgroup namespace

The cgroup namespace changes how a process sees the control-group tree from chapter 3. Inside it, the process’s own group looks like the top of the tree, so it cannot learn where it sits on the host. It hides information; it does not set any limit. Zygo creates one, and in addition does not mount the cgroup file system inside the sandbox at all.

Time namespace

The newest one, added in Linux 5.6. It lets a process see a different value for the clocks that count time since boot. Its main use is moving a running container to another machine without its clocks jumping. Sandboxes for short jobs rarely need it, and Zygo does not create one.

Creating and joining

Three syscalls do the work. clone (and the newer clone3) starts a child in new namespaces. unshare moves the calling process into new namespaces. setns joins a namespace that already exists, given a file that points to it, such as /proc/<pid>/ns/net. Zygo uses clone3 to build a sandbox, because it makes the child pid 1 of its new PID namespace in one step. It uses setns on the warm path, to put a new process into a sandbox that is already built.

  clone3(flags)            unshare(flags)            setns(fd)
  ─────────────            ──────────────            ─────────
  parent                   process                   process      existing
    │                        │                         │          namespace
    └─▶ child, born         moves itself into         └──────────▶ ┌─────┐
        inside NEW          NEW namespaces                         │ net │
        namespaces                                                 └─────┘
  used by Zygo to          used by `unshare(1)`      used by Zygo to enter
  BUILD a sandbox          and many tools            a WARM sandbox

What namespaces do not do

Namespaces hide things; they do not count or limit anything. A process in its own namespaces can still use all the memory, fill the process table with a fork bomb, or keep every CPU busy. Namespaces also do nothing about which syscalls a process may call, so the whole kernel is still in reach. Those gaps are filled by control groups (chapter 3) and by seccomp and Landlock (chapter 4). You need all of them together; any one alone is not a sandbox.

Seeing them yourself

lsns lists the namespaces on the machine. ls -l /proc/self/ns shows the ones your shell is in, each as a number. unshare --user --map-root-user --pid --fork --mount-proc bash starts a shell where you are root and pid 1 — without being root on the host. Type ps inside it and you will see two processes. That small command is most of the idea behind every tool in this book.

3. Control groups

Namespaces change what a process can see. Control groups, or cgroups, limit what it can use.

What a cgroup is

A cgroup is a group of processes that the kernel counts and limits together. You can say “this group may use 256 MB of memory, half a CPU and at most 64 processes”, and the kernel holds every process in the group to it. A child process starts in its parent’s group, so a program cannot escape its limit by forking. Without cgroups, one bad program could use up the whole machine; with them, it can use up only its own share.

The tree, as files

Cgroups are shown as folders under /sys/fs/cgroup. Each folder is a group, and a folder inside it is a smaller group inside the bigger one. You create a group with mkdir, move a process into it by writing its pid to cgroup.procs, and set a limit by writing to a file such as memory.max. There is no special tool and no daemon; it is plain file work. Limits nest: a group can never use more than its parent allows, whatever its own files say.

/sys/fs/cgroup/                                 the whole machine
├── system.slice/                               system services
└── user.slice/
    └── user-1000.slice/
        └── user@1000.service/                  ◀── handed to you (delegation)
            └── zygo.slice/                     memory.max = RAM − reserve
                ├── system/                     the supervisor, protected
                └── tenants/
                    └── acme/                   one customer's budget
                        └── resize/             one function: cpu.max, pids.max
                            └── g4242-1/        one warm sandbox
                                ├── zygote      the warm process: memory.max = mem
                                ├── req-01f3…   one request ─┐ each has its own group and
                                └── req-01f4…   one request ─┘ its own memory.max = mem

This is Zygo’s real tree; crates/zygo-core/src/cgroup.rs describes each level. The memory limit is on the leaves on purpose. The function’s group holds the warm process and every request at once, so a memory.max there would be one budget for all of them, and a single request going over it would take the others with it. On its own group, a request that goes over mem is killed alone, and the warm process and the requests beside it go on.

Controllers

Each kind of resource is handled by a controller. The ones a sandbox cares about most are these:

When a group goes over memory.max, the kernel’s OOM killer (“out of memory”) kills a process in that group — not somewhere else on the machine. When it hits pids.max, fork just fails, which is how a fork bomb is stopped.

Version 1 and version 2

Linux has two versions of cgroups. Version 1 had a separate tree for each controller, which was flexible but confusing, and hard to hand to a normal user safely. Version 2 has one tree for everything, with clearer rules. Most current distributions use version 2 only. Zygo needs version 2; zygo doctor checks for it and says so if it is missing.

Delegation: cgroups without root

The cgroup tree belongs to root, so how can a normal user create groups? The answer is delegation: the system gives a user one branch of the tree, and the user can do what they like inside it. On a systemd machine every login already has such a branch, under user@<uid>.service. Zygo creates its whole tree inside that branch, which is why it needs no root at all. Troubleshooting covers the machines where the branch is missing or missing a controller.

Killing a whole group

Killing one process is easy; killing everything it started is not, because a process can fork faster than you can list its children. Cgroup version 2 has a file for this, cgroup.kill: write 1 to it and the kernel kills every process in the group at once (Linux 5.14 and newer). Zygo uses it for every deadline: when a request runs out of time, one file write ends the request and all its children, and nothing is left over.

  req-01f3/                      echo 1 > req-01f3/cgroup.kill
  ├── python  (the request)
  │   ├── sh                     ─▶  all of them, gone in one step,
  │   │   └── curl                   even ones forked a moment ago
  │   └── python (a worker)
  └── … (a fork bomb in progress)

Counting, not only limiting

A cgroup also keeps numbers. memory.peak says the most memory the group ever used, memory.events says whether the OOM killer fired, and cpu.stat says how much CPU was used and how often the group was held back by its quota. Zygo reads these after each request to report why it ended: out of time, out of memory, or neither. Its benchmark also reads cpu.stat, and refuses to judge a latency number when the group was held back, because that number would describe the limit rather than the code.

Freezing

The cgroup.freeze file stops every process in a group without killing it, and a second write lets them carry on. The processes keep their memory; they simply get no CPU time. Zygo uses this to pause a warm function that has been idle for a while. Waking it again costs one write, much less than starting it over.

Namespaces and cgroups, side by side

Neither one limits which syscalls a process may call. That is the next chapter.

4. The other locks

Namespaces decide what a process sees, and cgroups decide how much it uses. This chapter is about the rest: what it is allowed to do, and what its world is built from.

Why more locks are needed

A process in its own namespaces, inside a tight cgroup, can still call almost every syscall the kernel has. Some of those syscalls are large and complex, and most serious container escapes of the last ten years went through one of them. So a sandbox adds locks that work at a finer level: which powers the process holds, which syscalls it may make, and which files it may touch. Each lock is simple on its own. Together they mean that one mistake is not enough to get out.

  ┌──────────────────────────────────────────────────────────────┐
  │ namespaces        what it can SEE                            │
  │  ┌────────────────────────────────────────────────────────┐  │
  │  │ cgroup          how much it can USE                    │  │
  │  │  ┌──────────────────────────────────────────────────┐  │  │
  │  │  │ capabilities + no_new_privs  what POWER it holds │  │  │
  │  │  │  ┌────────────────────────────────────────────┐  │  │  │
  │  │  │  │ Landlock      which FILES / PORTS          │  │  │  │
  │  │  │  │  ┌──────────────────────────────────────┐  │  │  │  │
  │  │  │  │  │ seccomp     which SYSCALLS           │  │  │  │  │
  │  │  │  │  │         ┌──────────────┐             │  │  │  │  │
  │  │  │  │  │         │ your program │             │  │  │  │  │
  │  │  │  │  │         └──────────────┘             │  │  │  │  │
  │  │  │  │  └──────────────────────────────────────┘  │  │  │  │
  │  │  │  └────────────────────────────────────────────┘  │  │  │
  │  │  └──────────────────────────────────────────────────┘  │  │
  │  └────────────────────────────────────────────────────────┘  │
  └──────────────────────────────────────────────────────────────┘
      to get out, a program has to beat every layer — or the kernel itself

Dropping capabilities

As chapter 1 said, capabilities are root’s power cut into pieces. Inside a user namespace a process can hold all of them — over that namespace. A sandbox then drops every one it does not need, from every set the kernel keeps, so they cannot come back. Docker keeps fourteen by default, to be useful to normal software. Zygo keeps none, because a function has no need to change the network or create device files.

no_new_privs

Some programs on disk are marked setuid: they run as their owner, often root, whoever starts them. sudo and passwd work this way. Inside a sandbox, that would be a way to gain power back. The no_new_privs flag, set once on a process, tells the kernel that nothing this process or its children exec may ever gain power, setuid or not. It cannot be unset. Every serious sandbox sets it, and seccomp needs it before a normal user may install a filter.

seccomp

seccomp (“secure computing”) lets a process install a small filter that the kernel runs on every syscall it makes. The filter is a tiny program, written in classic BPF, that looks at the syscall number and its arguments and answers “allow”, “fail with an error” or “kill”. Once installed it cannot be removed, and children inherit it. Docker’s default filter lists what is blocked and allows about 350 syscalls. Zygo’s lists what is allowed — about 215 names, of which 190 exist on an arm64 kernel and all on x86_64 — so a syscall added to the kernel next year is blocked until someone chooses to allow it. Seccomp profiles has the full lists.

  program ── syscall(nr, args) ──▶ ┌─────────────────────────┐
                                   │ seccomp filter (BPF)    │
                                   │  read, write, openat …  │──▶ allow ──▶ kernel does it
                                   │  clone + CLONE_NEW*     │──▶ EPERM ──▶ "not permitted"
                                   │  bpf, io_uring, ptrace, │──▶ EPERM
                                   │  mount, unshare …       │
                                   │  anything not listed    │──▶ EPERM
                                   └─────────────────────────┘

Landlock

Landlock, added in Linux 5.13, lets a normal process limit its own access to files, and from 6.7 its network connections too. The process says, for example, “from now on I may read under /usr and write under /tmp, and nothing else”, and the kernel enforces it, even against root inside the sandbox. Like seccomp, it can only be tightened, never loosened. It is a second wall behind the mount namespace: if a mistake ever made a host path visible inside, Landlock would still refuse to open it. Zygo applies it wherever the kernel has it.

AppArmor and SELinux

These are security modules: rules, loaded by the system’s administrator, that the kernel checks for every process. Docker on Ubuntu loads an AppArmor profile for each container; on Fedora and Red Hat, SELinux labels do the same job. They are strong, but they need root to set up, so a rootless tool like Zygo cannot use them for its own sandboxes. They can get in its way, though: Ubuntu’s AppArmor rules limit unprivileged user namespaces, and troubleshooting explains the fix.

Resource limits (rlimits)

rlimits are the old, per-process limits that came before cgroups: how many files a process may have open, how big a file it may write, how much stack it may use. You see them with ulimit -a. They are weaker than cgroups because they count per process, not per group. They are still useful for the few things cgroups do not cover, such as open files. Zygo sets nofile (1024 by default) this way.

The root filesystem

Inside a mount namespace a sandbox builds a new file tree, usually from an image, and makes it the root. The old way is chroot, which only changes where path lookups start and has well-known ways out. The better way is pivot_root: it swaps the old root for the new one in the mount namespace, and then the old root can be unmounted, so the host’s files are not just hidden but gone from this view. Zygo uses pivot_root, mounts the root read-only, and gives the sandbox one small writable place, /tmp, in memory.

Layers: overlayfs, bind mounts and tmpfs

Three kinds of mount do most of the work. overlayfs stacks folders on top of each other and shows them as one, which is how an image made of several layers becomes one tree without copying anything; a normal user may use it from Linux 5.11. A bind mount shows an existing file or folder at a second place — this is how your code gets into a sandbox, and it can be made read-only. tmpfs is a file system in memory, which vanishes when the last process using it exits. Build the root from overlayfs, add your files with bind mounts, give it a tmpfs to write to, and you have a container’s file system.

  what the program sees as /             where it really comes from
  ──────────────────────────             ──────────────────────────
  /app/handler.py   (read-only)   ◀───── bind mount of ./handler.py on the host
  /tmp              (writable)    ◀───── tmpfs, in memory, 64M, gone at exit
  /usr /lib /bin …  (read-only)   ◀───── overlayfs of the image's layers:
                                           ┌───────────────────────┐
                                           │ layer 3  pip packages │
                                           │ layer 2  python       │
                                           │ layer 1  debian base  │
                                           └───────────────────────┘
                                           stored once, shared by every sandbox

The network

A new network namespace has only loopback. There are two common ways to connect it. The first is a veth pair: a virtual cable with one end inside and one on the host, joined to a bridge with NAT — Docker’s way, which needs root on the host side. The second is a program that moves packets between the namespace and the host’s normal sockets, in user space and as your own user; slirp4netns and pasta do this. Zygo uses pasta, and puts an nftables firewall inside the sandbox’s own namespace, where your user is allowed to. That firewall is how “this function may reach api.example.com:443 and nothing else” is enforced.

  Docker (bridge, root on the host)            Zygo (pasta, your own user)
  ─────────────────────────────────            ───────────────────────────
  ┌ container netns ┐                          ┌ sandbox netns ─────────────┐
  │ eth0            │                          │ tap0                       │
  └──┬──────────────┘                          │ nftables: allow only       │
     │ veth pair                               │   api.example.com:443      │
  ┌──▼──────────────┐                          │ no 10.x / 192.168.x /      │
  │ docker0 bridge  │ ◀─ iptables NAT          │   169.254.169.254          │
  └──┬──────────────┘                          └──┬─────────────────────────┘
     ▼                                            │ packets as data
  host network: LAN, cloud metadata,           ┌──▼──────────────┐
  the internet — all reachable by default      │ pasta (as you)  │──▶ normal sockets
                                               └─────────────────┘    on the host

Putting it together

A sandbox on Linux is all of these, set up in the right order in the moment between fork and exec:

clone3 into new namespaces (user, mount, pid, net, ipc, uts, cgroup)
  → write the uid map                     (from the parent)
  → join a cgroup with limits
  → build the root: overlayfs + bind mounts + tmpfs, then pivot_root
  → set rlimits, drop every capability, set no_new_privs
  → install Landlock, then the seccomp filter
  → execve your program

Each step is cheap — a namespace set is about a millisecond, a cgroup write a tenth of that, a filter microseconds. The whole list is the sandbox. The rest of this book is about who runs this list, how often, and what they put around it.

5. Docker

Docker did not invent any of the parts in the last three chapters. What it did was put them behind one simple command, and add a way to ship software that the whole industry now uses.

The problem Docker solved

Before 2013, putting a program on a server meant installing its language, its libraries and its settings on that server, and hoping they matched the developer’s machine. Docker’s answer was to ship the program with its whole file system, as one thing you can copy, and to run it in namespaces so it did not see or break the rest of the server. “It works on my machine” became “it works in this image”. The isolation was a side benefit; the packaging was the revolution.

Images and layers

An image is a file system plus a little metadata: which command to run, which user, which environment. It is stored as a stack of layers, each a tar file of changes on top of the one below. Two images based on the same python:3.12 share those lower layers on disk and on the network. A layer is named by the hash of its content, so it can never change without getting a new name. That is why an image pulled today by digest is the same, byte for byte, as the one pulled next year.

        myapp:1.0                  otherapp:2.3
    ┌─────────────────┐        ┌─────────────────┐
    │ COPY . /app     │        │ COPY . /app     │   each layer is named by
    ├─────────────────┤        ├─────────────────┤   the hash of its content:
    │ pip install …   │        │ pip install …   │   sha256:9f…, sha256:77…
    └────────┬────────┘        └────────┬────────┘
             └────────────┬─────────────┘
                 ┌────────▼────────┐
                 │ python:3.12     │   shared: stored and
                 ├─────────────────┤   downloaded once
                 │ debian:bookworm │
                 └─────────────────┘

Why a sandbox brings its own files

A common first question: Python is already installed on my machine, so why does a container need an image with another Python in it? The answer is that a program is never one file. /usr/bin/python3 needs libpython, the C library, about a thousand files of standard library, SSL certificates, time zone data and more, spread over /usr, /lib and /etc. To run the host’s Python inside a sandbox, you would have to show the sandbox all of those host folders — and hiding the host’s files is the sandbox’s first job. An image solves this: it is a complete, separate set of files, so the sandbox can see everything it needs and nothing of the host.

  USING THE HOST'S PYTHON                     USING AN IMAGE
  ───────────────────────                     ──────────────
  ┌ sandbox ────────────────────────┐         ┌ sandbox ────────────────────────┐
  │ /usr/bin/python3  ◀── host      │         │ /usr/bin/python3  ◀── image     │
  │ /usr/lib/python3.12 ◀── host    │         │ /usr/lib/python3.12 ◀── image   │
  │ /lib (libc, libssl) ◀── host    │         │ /lib (libc, libssl) ◀── image   │
  │ /etc (certs, config) ◀── host   │         │ /etc (certs, config) ◀── image  │
  │ app.py                          │         │ app.py                          │
  └─────────────────────────────────┘         └─────────────────────────────────┘
  the sandbox can read big parts of           the sandbox sees a complete world,
  the host, and they change with apt          and none of the host's files

An image is also a promise

There is a second reason, just as important. The Python on your laptop is 3.12, the one on the server is 3.10, and apt upgrade can change either one tomorrow. A program that works on one machine may fail on the other, and you cannot tell why. An image named by its digest is the same bytes on every machine, today and next year. And an image costs almost nothing per run: it is downloaded once, stored once, and shared read-only by every container that uses it.

The Dockerfile

A Dockerfile is a recipe for building an image, one step per line: FROM python:3.12, RUN pip install requests, COPY . /app. Each step runs in a temporary container and its changes become a new layer. It is simple and very widely known. The cost is that every change of a dependency means a new build, a new image and a new push. Chapter 6 shows how Zygo avoids this.

OCI: the standards

The Open Container Initiative turned Docker’s formats into open standards. The image spec says how an image and its layers are stored; the distribution spec says how a registry serves them; the runtime spec says how a folder plus a config.json becomes a running container. Because of OCI, an image built by Docker runs under Podman, Kubernetes, or Zygo, and comes from any registry: Docker Hub, GitHub, your own. Zygo reads OCI images and uses none of Docker’s own code.

A container is a process

Run docker run -d nginx and then ps -ef on the host: nginx is there, as a normal process with a normal pid. There is no box and no small machine. The kernel sees a process with its own namespaces, in a cgroup, with a seccomp filter and an AppArmor profile — exactly the list at the end of chapter 4. “Container” is a word for that process plus the metadata Docker keeps about it.

The chain behind docker run

docker itself does almost nothing. It sends your request over a socket to dockerd, the daemon, which runs as root. dockerd asks containerd to create the container. containerd starts a containerd-shim, which runs runc. runc does the real work — namespaces, cgroup, mounts, filters — then execs your program and exits. The shim stays, for as long as the container lives, to hold its input, output and exit code.

docker ─▶ dockerd ─▶ containerd ─▶ containerd-shim ─▶ runc ─▶ your program
 (CLI)    (root)      (root)        (stays)           (exits)

Five programs and three hand-overs for one command. Each exists for a good reason — upgrades without stopping containers, Kubernetes, many clients — and each costs time.

Where Docker’s time goes

docker run of a small Python script takes about 300 to 1000 milliseconds. The isolation itself — the part runc sets up — is around one millisecond of that. The rest is the chain above, creating the container’s record and its writable layer, setting up the network bridge, and then starting Python and its imports from nothing. docker exec into a running container skips some of this and still costs 50 to 100 ms. For a service that runs for weeks, none of it matters. For a function that runs for 10 ms, it is almost all of the cost.

  one `docker run` of a 10 ms Python function, roughly

                ~250 ms           ~1 ms         ~220 ms         ~10 ms
  ┌────────────────────────────────┬─┬───────────────────────────┬──┐
  │ docker → dockerd → containerd  │▒│ Python starts and imports │██│
  │ → shim; record, layer, network │▒│ its modules from nothing  │██│
  └────────────────────────────────┴─┴───────────────────────────┴──┘
  ├────────────┼────────────┼─────────────┼────────────┼────────────┤
  0           100          200           300          400         500 ms

  ▒  the isolation itself
  ██ your code

What a container leaves behind

A container is an object, not only a process. When the program exits, the container stays in docker ps -a, with its writable layer on disk and its logs in the daemon, until someone runs docker rm, or passed --rm. This is right for services: you can restart them, read their logs, step inside. For a job that runs once, it is something to clean up.

Docker’s defaults

Docker’s defaults are made for running software you chose. The root file system is writable, fourteen capabilities are kept, the process runs as root inside, the network is a bridge that can reach your local network, and there is no memory, CPU or process limit unless you ask. You can make a container very tight — --read-only --cap-drop ALL --network none --memory ... — but you have to know every flag. For running code you did not write, the defaults point the wrong way. The comparison lists them side by side with Zygo’s.

Rootless Docker and Podman

The root daemon has always been Docker’s weak spot: access to its socket is the same as root on the host. Rootless Docker runs the whole chain as a normal user inside a user namespace. Podman goes further: no daemon at all, each container started by the podman command itself, with a small helper, conmon, staying behind like the shim. Both are real steps forward. Both still create container objects and still start your program from nothing, so the per-run cost stays in the hundreds of milliseconds.

What Docker is for

Docker is the right tool for building images, running long-lived services, connecting them with networks, publishing ports and restarting them when they fail. Zygo does none of that, and uses Docker’s images happily. The question this book cares about is narrower: what if the thing you run is a short function, called thousands of times, written by someone you do not fully trust? The next chapter is the answer Zygo gives.

6. How Zygo works

Zygo uses the same kernel parts as Docker. What it changes is who sets them up, how often, and what is already waiting when a request arrives.

Zygo in one picture

                    ┌──────────────────────── zygo (your user, no root) ──────────────────────────┐
                    │                                                                             │
   zygo run ───────▶│  ONE-SHOT: build a sandbox ─▶ run the program ─▶ exit, nothing left  ~12 ms │
                    │                                                                             │
   zygo serve ─────▶│  WARM: build a sandbox once, start the interpreter, import the handler      │
                    │        and park it as a "zygote"                                   ~150 ms  │
                    │                                                                             │
   zygo exec ──────▶│        fork the zygote ─▶ the child runs one request ─▶ exit        ~1.4 ms │
   zygo exec ──────▶│        fork the zygote ─▶ the child runs one request ─▶ exit        ~1.4 ms │
                    │                                                                             │
                    └─────────────────────────────────────────────────────────────────────────────┘

There are two halves. The one-shot half is a better docker run: a fresh sandbox per program, built in one process. The warm half is the reason the project exists: a sandbox that is built once and then copied for every request. The rest of the chapter takes them in that order.

The one-shot sandbox: zygo run

zygo run python:3.12 python3 app.py does the whole list from chapter 4 itself. It reads the image from its own store, works out a mount plan, calls clone3 into seven new namespaces, and the child builds its root, joins its cgroup, drops every capability, installs Landlock and seccomp, and calls execve. There is no daemon, no RPC, and no container record. When the program exits, the kernel removes the namespaces and the tmpfs, Zygo removes the cgroup, and nothing is left to clean up. With the image already pulled, this takes a median of 12 ms on a Linux 6.8 VM. That is python3 -c pass. The 70.8 ms on the README’s table is the same command running a script that imports sixteen modules: the sandbox costs the same 3.6 ms, and the rest is Python doing the imports — the work a warm function does once (chapter 25).

  docker run                                   zygo run
  ──────────                                   ────────
  docker ─▶ dockerd ─▶ containerd              zygo
             ─▶ shim ─▶ runc ─▶ program          └─ clone3 ─▶ child: mounts, cgroup,
                                                              caps, Landlock, seccomp
  record + layer left: docker rm                              ─▶ execve ─▶ program
  300–1000 ms                                  nothing left · ~12 ms

The idea of a zygote

The name comes from Android. Starting a Java app from nothing is slow, so Android starts one process, the Zygote, loads the common libraries into it, and then forks it every time an app opens. Each app gets a ready-made process in a few milliseconds, and the loaded libraries are shared through copy-on-write. Zygo applies the same trick to a sandbox. The expensive part of running a Python function is not the sandbox, it is Python itself — starting the interpreter and importing modules often takes hundreds of milliseconds. So Zygo does that once, inside the sandbox, and forks the result.

The same idea reached serverless before Zygo. SOCK (Oakes et al., SOCK: Rapid Task Provisioning with Serverless-Optimized Containers, USENIX ATC 2018) forks Python handlers from a zygote that has already imported their packages, and Catalyzer (Du et al., Catalyzer: Sub-millisecond Startup for Serverless Computing with Initialization-less Booting, ASPLOS 2020) restores a function from a snapshot instead of starting it. Zygo’s contribution is the packaging — one static binary, rootless, every limit on, an agent protocol any language can speak — not the idea.

The warm path

zygo serve ./handler.py builds a sandbox, starts a small agent inside it, and the agent imports your handler and waits. That waiting process is the zygote. For each request the zygote calls fork(); the child runs your handler(event) once, sends back the result, and exits. The child starts with everything already loaded, because it is a copy of a process that had loaded it. And it leaves nothing behind, because it is thrown away.

            ┌──────────────── one warm sandbox (namespaces, root, filters) ─────────────────┐
            │                                                                               │
            │   zygote: python + imports done + handler loaded — never runs a request       │
            │     │                                                                         │
 request 1 ─┼──▶  ├── fork ─▶ child 1: handler(event) ─▶ reply ─▶ exit  (its writes: gone)  │
 request 2 ─┼──▶  ├── fork ─▶ child 2: handler(event) ─▶ reply ─▶ exit  (starts clean)      │
 request 3 ─┼──▶  └── fork ─▶ child 3: handler(event) ─▶ reply ─▶ exit  (starts clean)      │
            │                                                                               │
            └───────────────────────────────────────────────────────────────────────────────┘

Why a fork is clean

A normal worker process that serves many requests slowly collects state: a global that one request changed, an open connection, a patched function, a file in /tmp. Request n runs in whatever request n−1 left. Zygo’s children never share that problem, because each one is a copy of the zygote, and the zygote has never served a request. Files are the one place that needs more than a fork: each request gets its own temporary folder, named by TMPDIR and removed afterwards, because the sandbox’s /tmp is shared by every request in it. So a request sees exactly what the zygote had after its imports — every time, whatever the requests before it did. You get the speed of a shared worker and the cleanliness of a fresh container.

  a shared worker                          zygo exec
  ───────────────                          ─────────
  worker ─ req 1 ─ req 2 ─ req 3 ─ …       zygote ──┬─ copy ─ req 1 ✗
     state piles up: req 3 sees                     ├─ copy ─ req 2 ✗
     what req 1 and 2 left behind                   └─ copy ─ req 3 ✗
                                           every copy starts from the same clean point

One cgroup per request

Each child is put in a cgroup of its own, under its function’s group — you saw the tree in chapter 3. That gives every request its own memory limit, process limit and CPU share. When a request runs past its deadline, one write to its cgroup.kill ends it and every process it started, and the zygote keeps serving. When it runs out of memory, the kernel kills that request, not the zygote and not a neighbour. After each request Zygo reads the group’s numbers and reports the result: timed_out, oom_killed, peak memory, wall time.

Warm-exec, for programs that start fast

A Go or Rust binary starts in a millisecond, so there is nothing to keep warm inside it. For these Zygo keeps only the sandbox warm: namespaces, mounts and filters built once. Each request is a new process that enters the sandbox with setns and execs your cmd, reading the event on standard input and writing the result on standard output. This costs about 1.4 ms, measured with sh -c cat as the program; a bigger program adds its own start. It works for any language and any image, with no agent at all.

The agent and its protocol

The agent is the small program inside the sandbox that loads the handler, forks, and passes events and results. It talks to Zygo over a simple, documented wire protocol (spec/protocol.md), not through a library. So any language can have an agent: Zygo ships one for Python and one for Node, and zygo agent test checks a new one against the same conversation. A runtime pool is the same idea with no handler loaded in advance: the script arrives with the request, which lets one warm zygote serve thousands of different scripts for about 0.5 ms more.

The supervisor

Someone has to keep zygotes alive, hand out requests, watch deadlines and collect results. That is the supervisor: a normal process under your own user, which the first zygo serve starts when it needs one. It is not a system service and it never runs as root. It owns the cgroup tree, keeps a reserve of memory for itself so that busy tenants cannot starve it, and serves the CLI, the HTTP API and the SDKs. If it goes, the sandboxes it started go with it; nothing is left running on its own.

  CLI · HTTP API · Python/Node SDK · MCP
                  │
          ┌───────▼────────┐    memory reserved,
          │   supervisor   │    so tenants cannot starve it
          │  (your user)   │
          └──┬─────┬─────┬─┘
             │     │     │   deadlines, cgroups, secrets, logs
      ┌──────▼┐ ┌──▼────┐ ┌▼──────┐
      │zygote │ │zygote │ │sandbox│   one per function (and script version)
      │resize │ │fetch  │ │parse  │
      └───────┘ └───────┘ └───────┘

Images without a Dockerfile

Zygo pulls normal OCI images from any registry, into a store under your home folder. What you would put in a Dockerfile goes in sandbox.toml instead: requirements = "./requirements.txt" becomes a Python venv, built once inside a sandbox with the image’s own pip, and system = ["libwebp7"] becomes an extra layer with those apt packages. Each is keyed on the image digest and the list, built once, and shared by every function that asks for the same thing. The image itself is never changed, and there is no build step for you to run or to push.

   python:3.12-slim  (from the registry, never changed)
          │
          ├── + bytecode layer   (.pyc for the standard library, built once)
          ├── + system layer     (apt: libwebp7)        key: image digest + list
          └── + /venv            (pip: requirements)    key: image digest + lockfile
                     │
            shared by every function that names the same image and the same list

Why not zygo ./venv/bin/python app.py?

It looks simpler: you already have a Python and a venv, so why name an image at all? Because a venv is not a Python. venv/bin/python is only a link to the host’s interpreter, and the venv folder holds only the extra packages; the interpreter, its standard library and the C libraries under it all stay in the host’s /usr and /lib. To run it, Zygo would have to show the sandbox those host folders, which is exactly what a sandbox exists to hide (chapter 5). So Zygo has no mode that uses the host’s own files as the root, on purpose: every sandbox starts from an image, and sees of your machine only what you --mount, read-only unless you say :rw.

  what you think a venv is          what a venv really is
  ────────────────────────          ─────────────────────
  ┌ venv/ ──────────────┐           ┌ venv/ ──────────────┐
  │ python              │           │ bin/python ─────────┼──▶ /usr/bin/python3.12   (host)
  │ everything it needs │           │ lib/…/site-packages │      ├─▶ /usr/lib/python3.12/ (host)
  └─────────────────────┘           └─────────────────────┘      └─▶ /lib/libc.so …       (host)

Why not zygo /usr/bin/python3 app.py?

Then skip the venv and point at the system’s own Python directly? It fails for the same reason, and trying it shows why, one wall at a time. The file /usr/bin/python3 is small; the Python it starts is not. On an Ubuntu 24.04 machine it needs five shared libraries from the host’s /lib, and then its standard library — 1,194 files, 29 MB, in the host’s /usr/lib/python3.12. Mount only the binary into an image, and each missing piece stops it in turn.

  zygo run --mount /usr/bin/python3:/opt/python3 IMAGE /opt/python3 -c "print(1)"

  wall 1  the loader    alpine:3               "the program does not exist"   exit 125
          (musl image, and the binary asks for glibc's loader)
  wall 2  the libraries debian:bookworm-slim   "libexpat.so.1: cannot open    exit 127
                        python:3.12-slim        shared object file"
          (the image has glibc, but not the exact libraries this build wants)
  wall 3  the stdlib    python:3.12-slim       "No module named 'encodings'"  exit 1
                        + libexpat mounted too
          (it looks for /usr/lib/python3.12; the image keeps its own elsewhere)

Measured on the Lima VM from chapter 25, Ubuntu 24.04, Python 3.12.3, aarch64. Notice the last row: even an image with the same Python version does not help, because the host’s binary looks for the host’s files, in the host’s places.

And if you mounted all of it?

You could keep mounting — /usr/lib/python3.12, then the libraries, then /etc/ssl — until it starts. By then the sandbox can read most of the host’s /usr and /lib: every program installed, every version, which is a map for anyone looking for a weak spot. Worse, those files are not yours to hold still: the next apt upgrade replaces them under a zygote that is still running, and a warm function breaks in a way nobody can reproduce. The system Python is also the operating system’s own tool — apt and other system programs depend on it, and on Debian and Ubuntu pip refuses to install into it (the “externally managed environment” error). The image’s Python belongs to your function alone, and never changes unless you change its name.

  the system's Python                     the image's Python
  ───────────────────                     ──────────────────
  owned by the OS, used by apt            owned by your function
  changes with every apt upgrade          changes only when you change its name
  different on every machine              the same bytes everywhere (a digest)
  shows the sandbox the host's /usr       shows the sandbox nothing of the host

What you do instead, for Python

You keep the parts that are yours and take the rest from an image. Your code comes in with --mount ./app.py:/app/app.py. Your packages come from --requirements requirements.txt: Zygo builds a venv once, inside the image, with the image’s own pip, and shares it with every run that asks for the same list. The Python itself comes from the image, the same bytes on every machine. Nothing here costs time per request: the image is pulled once, the venv is built once, and both are mounted read-only in a few milliseconds.

# not:  zygo ./venv/bin/python app.py
zygo run --mount ./app.py:/app/app.py --requirements requirements.txt \
    python:3.12-slim python3 /app/app.py

Why not zygo ./my_executable_binary?

Now take a compiled program instead — a Go server, a C tool. There is no interpreter this time, so does it still need an image? It depends on how the program was linked, which means how it finds the library code it uses. A dynamically linked program keeps only its own code; when it starts, a small loader (/lib64/ld-linux-x86-64.so.2) finds libc.so and friends on the machine and joins them in. That is the default for C (gcc app.c), and for Go when it uses cgo. Such a program has exactly the venv’s problem: its libraries live in the host’s /lib, and the sandbox is not meant to see it. A statically linked program carries every library inside its own file, so it needs almost nothing — but “almost” is not “nothing”.

  dynamically linked (gcc app.c)          statically linked (CGO_ENABLED=0 go build)
  ──────────────────────────────          ─────────────────────────────────────────
  ┌ my_app ───────┐                       ┌ my_app ────────────────────────┐
  │ your code     │──▶ ld-linux.so (host) │ your code                      │
  └───────────────┘──▶ libc.so.6  (host)  │ + the Go runtime               │
                   ──▶ libssl.so  (host)  │ + every library it uses        │
                                          └────────────────────────────────┘
  needs the host's /lib inside            still wants a few files around it:
  the sandbox: the venv problem again     /etc/ssl/certs, /usr/share/zoneinfo, …

What a static binary still needs

Even a fully static program expects a small world around it. To call an HTTPS API it reads CA certificates from /etc/ssl/certs; to show local time it reads /usr/share/zoneinfo; to turn a uid into a name it reads /etc/passwd; to resolve a host name it reads /etc/resolv.conf; if it runs sh -c, it needs a /bin/sh. And every sandbox needs a root to stand on: somewhere to put /tmp, /proc and /dev. An image gives all of this in a few megabytes, the same on every machine, downloaded once. So Zygo keeps one rule with no exceptions: the root always comes from an image, and from your machine a sandbox sees only what you --mount.

What you do instead, for Go and C

Mount the binary into a small image and name it as the command. Which image depends on how the binary was linked, and file ./my_app tells you: it says either statically linked or dynamically linked, interpreter ….

file ./my_app                              # "statically linked"? then any small image works
zygo run --mount ./my_app:/app/my_app alpine:3 /app/my_app --port 8080

# a dynamic glibc binary: pick an image with glibc, not Alpine
zygo run --mount ./my_app:/app/my_app debian:bookworm-slim /app/my_app

The mistake to avoid is a glibc binary in alpine:3: it fails with “the program does not exist”, because the loader it asks for is not in the image (chapter 12). And for a binary you call again and again, do not pay even the 12 ms of a fresh sandbox: keep the sandbox warm with cmd, and each call costs about 1.4 ms plus the program’s own start (warm-exec).

[fn.parse]
image  = "alpine:3"
mounts = ["./bin/parse:/app/parse:ro"]   # a static Go or C binary you built
cmd    = ["/app/parse"]                  # event on stdin, JSON result on stdout

Secrets

A secret, such as an API key, is never put in the environment and never in the zygote’s memory. While a request runs, the supervisor writes it as a file, /run/secrets/NAME, from outside the sandbox, readable only inside that function’s sandbox, and removes it when the function’s last request ends. A request that is compromised can read the secrets its own function was given, while it runs, and never one that another function uses.

The network, off by default

A sandbox starts with no network at all. network = "egress" with allow = ["api.example.com:443"] opens exactly that name and port: Zygo runs its own small DNS resolver inside the sandbox, and when a name on the list is asked for, its addresses are added to the nftables filter before the answer goes back. A name not on the list does not even resolve. Private addresses and the cloud metadata address stay closed in every mode unless you ask for them by a flag named for exactly that. Chapter 14 has the details.

Safe by default, loosened by name

Every limit has a value even if you set none: 256 MB of memory, one CPU, 64 processes, 30 seconds, up to 64 MB of scratch, 1024 open files. The root is read-only, no capability is kept, and the seccomp allowlist is on. To remove a limit you type a flag whose name says what it does — --allow-unlimited, --allow-host-net, --allow-private-net. Docker makes you add safety one flag at a time; Zygo makes you remove it one flag at a time. The principles gives the principle and its cost.

Three backends, one command

isolation = "ns" | "gvisor" | "vm" moves the wall without changing anything else. ns is everything above: the host kernel with its locks, and the only backend with warm functions. gvisor puts gVisor’s user-space kernel between your program and the host. vm boots a small virtual machine with its own kernel through libkrun, for code you trust least, at about six times the start-up cost. ADR 0002 explains why the warm path stays on ns.

On a Mac

Sandboxes are a Linux feature, so on macOS Zygo starts and manages a small Linux virtual machine through Lima. The zygo command on your Mac forwards every sandbox command into it, with the same arguments, folder and streams. Crossing into the VM adds about 22 ms per command; from a program, through the API or the SDKs, the millisecond warm path is still there.

7. Where the time and memory are saved

Zygo is not faster because it found a faster kernel feature. It is faster because it stops doing work that does not need to be done on every request. This chapter lists each piece of that work, how much it costs, and what Zygo does instead.

The whole picture

  what one request pays, for a small Python function (not to scale)
  ─────────────────────────────────────────────────────────────────────────────
  docker run   ░░░░░░░ chain ░░░░░░░▒▓▓▓▓ Python + imports ▓▓▓▓█   300–1000 ms
  docker exec  ░░ daemon ░░█                                        50–100 ms, shared state
  zygo run     ▒▓▓▓▓ Python + imports ▓▓▓▓█                         12 ms, Python included
  zygo exec    ▪█                                                   1.4 ms, clean state
  ─────────────────────────────────────────────────────────────────────────────
  ░ programs talking to programs   ▒ the isolation itself (about the same everywhere)
  ▓ interpreter start-up           ▪ a fork            █ your code

The isolation is not what costs. Each saving below removes one of the other blocks.

Saving 1: no chain of programs

docker run passes through a CLI, a root daemon, containerd, a shim and runc, with a socket or a new process at each step, and each of them keeps its own records. zygo run is one process that makes the syscalls itself. There is nothing to ask and nothing to wait for. This alone takes a one-shot sandbox from hundreds of milliseconds down to about 12 ms with Python in it. The sandbox itself — the namespace set, the cgroup and the mounts — is about 3.6 ms of that, work the kernel has to do whoever asks for it; the other 8 to 9 ms is Python starting (chapter 25).

Saving 2: no container object to create or remove

Docker makes a record for every container, a writable layer on disk, and log files, and later has to delete them. Zygo makes none of these. The root is read-only and shared; the one writable place is a tmpfs that disappears when the process does. There is no zygo rm because there is nothing to remove. On a busy machine this also saves disk writes and the slow build-up of old containers that someone has to clean.

Saving 3: the interpreter starts once, not per request

This is the big one. A Python function that imports a few modules needs roughly 100 to 500 ms before it can run a single line of your code. A container per request pays that every time, however fast the container is. A Zygo zygote pays it once, at zygo serve, and every request after that is a fork() of a process where it is already done: about 1.4 ms. The same holds for Node with a large dependency tree, or any runtime whose start-up is slow.

  per-request cost, same handler, same host (from the embedder's benchmark)
  ────────────────────────────────────────────────────────────────────────
  a container per request   ████████████████████████████████████████  542.4 ms
  a one-shot sandbox        █████▏                                     70.8 ms
  a warm fork               ▏                                           2.8 ms

The one-shot row is still slow here because it starts Python each time, and the warm row includes starting the zygo CLI itself — through the API it is closer to the 1.4 ms above. The embedder’s benchmark has the full setup.

Saving 4: memory is shared, not copied

A forked child shares every page of the zygote until it writes to one (chapter 1). A request that reads a lot and writes little costs very little new memory. The zygotes also share with each other: a hundred warm Python scripts on the same image share almost all of the interpreter’s pages through the image’s files. Measured, each warm script costs about 11 MB of its own memory, not the 21 MB its process seems to use, so around 300 warm scripts fit in 4 GB. ADR 0005 has the numbers.

Saving 5: compiled bytecode is built once

The official python:*-slim images ship no compiled .pyc files, and a read-only root means Python cannot save the ones it compiles. So every run compiled every module it imported again — import ssl alone took 66 ms. Zygo compiles the standard library once, into a layer of its own, the first time it sees such an image. Importing ten common modules went from 165 ms to 35 ms. What Zygo costs has the details.

Saving 6: no image builds for dependencies

With Docker, a new Python package means a new Dockerfile step, a build, a new image and often a push. With Zygo, you list the packages and it builds a venv once, keyed on the image and the exact list, and shares it with every function that asks for the same thing. Changing a function’s code touches no image at all. This saves the build time, the registry space, and the “which image has which version” work that grows with every function.

Saving 7: clean-up is one write

Killing a process tree safely is hard: children can fork while you list them. Zygo puts each request in its own cgroup and ends it with one write to cgroup.kill. No scanning, no race, no leftover process. A timeout is cheap, certain, and it never touches the zygote or another request.

Saving 8: nothing to run or guard

There is no root daemon to keep running, patch, watch and protect. Access to Docker’s socket is the same as root on the host; Zygo has no such socket. The supervisor is a normal process under your user, started when needed. That is a saving in people’s time, not in milliseconds — but on a real team it is often the largest one.

The advantages, in one table

What is not saved

Your own code costs what it costs; Zygo only removes the work around it. A warm function uses memory while it waits — about 11 to 21 MB for a Python zygote — and after cold_after it is dropped and the next request pays the warm-up again. The ns backend shares the host’s kernel, so a kernel bug still defeats it, as it defeats every container. And Zygo is one machine: it does not spread work across servers, publish ports, or run long-lived services. The threat model and the comparison say where the edges are.

8. The rules Zygo is built on

Zygo is built on eight rules, or principles. Each one is a choice, and each choice has a cost. This chapter puts the cost right next to the rule, because a rule whose cost is hidden is only a slogan.

The eight rules at a glance

The rules fall into three groups: how a sandbox is made, how it stays fast, and how it stays safe and honest. The table is the short version; the rest of the chapter takes them one by one.

  ┌──────────── how it is made ────────────┐   ┌──────────── how it is fast ────────────┐
  │ P1 one process, no container machinery │   │ P2 the sandbox is ready before a call  │
  │ P4 any OCI image, dependencies in spec │   │ P3 one flag picks where the wall is    │
  │ P5 your user, no root, no root service │   │                                        │
  └────────────────────────────────────────┘   └────────────────────────────────────────┘
  ┌──────────────────────────── how it stays safe and honest ─────────────────────────────┐
  │ P6 locked by default · P7 every limit has a value · P8 every claim measured or marked │
  └───────────────────────────────────────────────────────────────────────────────────────┘

P1: a sandbox is a locked-down process, not a container

A container in Docker’s sense is an object managed by a chain of programs. Zygo skips that chain. It builds the sandbox from kernel parts — namespaces (separate views of the system), a cgroup (a group with resource limits), seccomp (a filter on system calls) and Landlock (a limit on which files a process may touch) — inside one process, with no RPC (a call to another program over a socket). You met all of these in chapters 2 to 4.

The steps are short. Zygo calls clone3 to make a child process in seven new namespaces. The child calls pivot_root to make the image its root folder, drops all its capabilities (pieces of root’s power), installs a BPF filter (the small program that seccomp runs on each system call), and calls execve to become your program. There is no daemon (a long-running background service) to talk to, no shim (a helper process that babysits a container), and no trip to an image service while a request waits.

  zygo ──clone3──▶ child in 7 new namespaces
                     │
                     ├─ pivot_root onto the image
                     ├─ drop every capability
                     ├─ install the seccomp BPF filter (and Landlock)
                     └─ execve ─▶ your program
  no daemon · no shim · no RPC · no image service on the way

What it costs. The wall is the Linux kernel. A bug that lets a program gain kernel privileges defeats every lock at once. That is why the vm backend exists, for code you did not write. zygo doctor and the security chapter say this plainly.

P2: the sandbox waits warm

Nothing is built while a request waits. There are two ways Zygo does this. For a compiled program the sandbox is held: namespaces, mounts and locks are set up once, and each request is a fresh process that enters them. That costs about 1.4 ms, plus the program’s own start. For an interpreter such as Python, the sandbox holds an agent, a small program that has already imported your handler. Each request is a fork() of it — a copy of the process. The copy is copy-on-write: memory is shared until one side changes it, so nothing is copied up front. And because each request is a fresh process, no state carries over from the last one.

Measured: usually 1.4 ms, on the machines named in chapter 25. That is at a steady 250 requests a second, the rate the latency test holds so the host is never the limit. The most one warm function sustained, with four clients sending, is 1,108 a second.

  compiled program (warm-exec)            interpreter (agent)
  ────────────────────────────            ───────────────────
  held sandbox                            held sandbox + agent with handler loaded
     │                                       │
     ├─ new process enters ─▶ run ~1.4 ms    ├─ fork ─▶ handler(event) ~1.4 ms
     └─ new process enters ─▶ run            └─ fork ─▶ handler(event)

What it costs. A warm function stays in memory. A Python zygote (the loaded process that is forked for each request) uses about 20 MB, of which about 11 MB is its own and the rest is shared with other zygotes. The idle rules pause it after idle_timeout: it is frozen but still in memory. They drop it after cold_after, and the next request then pays the warm-up again.

  serving ──(no calls for idle_timeout)──▶ paused: frozen, still ~20 MB in RAM
     ▲                                          │
     │                                    (no calls for cold_after)
     │                                          ▼
     └──────── next call pays warm-up ─── dropped: memory freed

P3: isolation is one flag

isolation = "ns" | "gvisor" | "vm". The spec, the command and the protocol stay the same; only the backend changes, and the backend decides where the wall is drawn. ns uses the host kernel. vm uses KVM (the kernel’s built-in support for virtual machines) with a guest kernel of its own. gvisor uses a kernel that runs in user space.

  isolation = "ns"       your program ─▶ host kernel
  isolation = "gvisor"   your program ─▶ gVisor (user-space kernel) ─▶ host kernel
  isolation = "vm"       your program ─▶ guest kernel ─▶ KVM ─▶ host kernel

What it costs. ns and gvisor are built. vm boots a guest and runs one-shot sandboxes, at about six times the setup cost of ns (422 ms against 73 ms on a Raspberry Pi 5, chapter 25). The guest writes to a private layer bounded by scratch; it has no network and no warm functions. zygo backend install gvisor downloads the gVisor runtime. The same command run on both backends gives the same answer, while uname -r inside reports 4.19.0-gvisor instead of the host’s kernel — which is the point.

gvisor is one-shot only so far. A warm function must be entered for each request; on ns that is setns into namespaces the supervisor holds, and gVisor’s wall is not those namespaces. So Zygo refuses a warm function on gvisor instead of quietly running something weaker, and it refuses a networked sandbox there too. Rootless runsc (gVisor’s runtime) also cannot write cgroups, so limits there are only advisory, and Zygo says so on every start. For a warm function today, that means ns.

P4: OCI images, no Dockerfile

Any OCI image (the standard image format Docker also uses) from any registry can be the file system. What a Dockerfile would add — Python packages, apt packages — goes in the spec instead. Zygo builds it once, inside the image, as a venv (a Python folder of installed packages) or as a derived layer (an extra image layer made by running the install). Every function that names the same thing shares the result. The image itself is never changed.

  image (never changed) ─┬─▶ + venv from requirements.txt ──▶ mounted at /venv
                         └─▶ + derived layer from apt list ──▶ shared by all who ask
  built once, keyed on the image digest and the list

What it costs. A derived layer is a copy of the image, an install, and a diff; about five seconds the first time. Overlayfs (a file system that stacks folders) would be faster, but rootless overlayfs needs kernel 5.11, and not every host Zygo runs on has it.

P5: no root, no daemon

Zygo needs no root at any point and no system service. The supervisor is a process in your own login session that zygo serve starts when it needs one. Outbound networking uses pasta (a user-space network helper) and nftables (the kernel’s packet filter) inside the sandbox’s own namespace. This works because entering a user namespace you created gives you a full set of capabilities inside it, and nowhere else.

  your login session (your uid, no root)
  └─ zygo serve ─▶ supervisor
                    └─ sandbox: own user namespace = full caps HERE only
                         ├─ nftables rules for the allowlist
                         └─ pasta ─▶ the outside network

What it costs. Some features need a program on the host: pasta and nft for outbound networking, and newuidmap for a separate range of user ids per tenant. If one is missing, Zygo refuses to start and names the package. It never silently falls back to something weaker.

P6: safe by default

With no flags, the network is off, the root file system is read-only, the capability set is empty, the seccomp allowlist is on, and every limit has a value. To loosen any of it you must type a flag: --allow-host-net, --allow-private-net, --allow-unlimited. zygo up does not accept these flags, so a spec that needs one has to be served on purpose.

  default                         to loosen, you type
  ───────                         ───────────────────
  network off                ──▶  --allow-host-net, --allow-private-net
  seccomp allowlist on       ──▶  a looser profile, by name
  every limit has a value    ──▶  --allow-unlimited
  (none of these flags work with zygo up)

What it costs. The first thing a new user hits is a limit. The error names the field and the fix.

P7: every limit is required

Memory, CPU, process count, wall-clock time, scratch space, open files, and, for a networked function, connections: each has a default, and there is no “unlimited” without the flag. Disk I/O and bandwidth are the two exceptions, below. The deadline kills the request’s whole process tree (the process and every child it started), not a single process. The per-request cgroup is what makes that one write, to its cgroup.kill file.

  request cgroup ── cgroup.kill ◀── one write at the deadline
     ├─ handler process        ✗
     │   └─ child it spawned   ✗
     └─ another child          ✗   the whole tree ends; the zygote keeps serving

What it costs. Disk I/O and bandwidth have no default limit, because a good value depends on the device. Zygo warns when either is missing.

P8: honest about what is not proven

Every claim in the README is either measured or marked. The test suites try the thing instead of reading a setting. A negative check first proves that the thing really ran. A latency number says whether it hit a limit. Chapter 25 records what was measured and on what hardware. Before these rules existed, several checks passed — or failed — for the wrong reason.

  weak test                           Zygo's test
  ─────────                           ───────────
  read a setting ─▶ pass              1. a negative check proves the probe really ran
                                      2. attempt the forbidden thing ─▶ must fail
                                      3. a latency number says if it hit a limit

What it costs. The status text is long, and it says “not built” more often than a launch page would.

The two warm modes, side by side

P2 has two shapes. Warm-exec keeps only the sandbox ready and runs your cmd fresh each time. Agent mode also keeps a loaded interpreter ready and forks it. Chapter 13 shows how to write each one.

  warm-exec:  [ sandbox ready ] ──▶ start your program ──▶ run   (fast for Go, Rust, C)
  agent:      [ sandbox ready + Python loaded ] ──▶ fork ──▶ run (skips the slow start)

9. FreeBSD jails, and Zygo

Linux was not first. FreeBSD had a working “container” in 2000, thirteen years before Docker, and many ideas in this book are easier to see there.

Where jails came from

Jails arrived in FreeBSD 4.0, in March 2000, written by Poul-Henning Kamp for a hosting company that wanted to give each customer “root” without giving them the machine. The paper that describes them has a telling title: Jails: Confining the omnipotent root. The idea was to take chroot, which only changes where file paths start, and close every way out of it. A jail became one clear thing in the kernel: a group of processes with its own files, its own host name, its own addresses, and a root user who is not really root.

How a jail works

A jail is one object in the kernel, made with the jail(2) syscall, usually through the jail(8) tool and a config file, /etc/jail.conf. Every process carries a pointer to its jail, and the kernel checks that pointer wherever it matters. A jailed process sees only processes in its own jail. Root inside may not mount file systems, load kernel modules, change the network, or reach raw devices. jls lists the jails; jexec runs a command inside one.

  FreeBSD: one kernel object                  Linux: many parts, put together
  ──────────────────────────                  ───────────────────────────────
  ┌──────────── jail ────────────┐            mount ns ─┐
  │ root path   /jails/web       │            pid ns   ─┤
  │ host name   web.example      │            net ns   ─┤
  │ addresses   10.0.0.5 / vnet  │            uts ns   ─┤
  │ root is limited              │            ipc ns   ─┼─▶  what a tool
  │ processes see only the jail  │            user ns  ─┤    calls a
  │ limits via rctl              │            cgroup   ─┤    "container"
  └──────────────────────────────┘            caps     ─┤    or "sandbox"
       made by jail(2), by root                seccomp  ─┤
                                               Landlock ─┘

What jails have grown since

Jails kept growing. VNET (since FreeBSD 12 in the default kernel) gives a jail a whole network stack of its own, like a Linux network namespace. rctl limits a jail’s memory, CPU and process count, like cgroups. Jails can nest inside other jails. Capsicum, a separate FreeBSD feature, lets a process lock itself down to only the files and sockets it already holds — close in spirit to seccomp and Landlock together. Tools such as iocage and Bastille manage jails the way Docker manages containers, and Podman now runs OCI images on FreeBSD with jails underneath.

One idea, two designs

FreeBSD and Linux solved the same problem in opposite ways. FreeBSD made one strong, complete object: you ask for a jail and you get all of it. Linux made many small parts — each namespace, cgroups, seccomp — and left it to tools to put them together. The FreeBSD way is easier to reason about: there is one thing to check, and it is hard to forget a piece. The Linux way is more flexible — a tool can take only a network namespace, or only a cgroup — but every tool must assemble the parts correctly, and a missing part is a hole. Much of Zygo’s test suite exists because of that second fact.

What Zygo shares with a jail

At the level of “what does the program inside see”, a Zygo ns sandbox and a jail are very close. Both give a group of processes its own root file tree, its own view of processes, its own network (or none), its own host name, and a root user without real power. Both share the host’s kernel, so both are only as strong as that kernel. Both are cheap: no second kernel, no virtual hardware. And both come from the same instinct: the process, not a machine, is the thing to confine.

Where they differ

The biggest difference: who holds the key

On FreeBSD, making a jail needs root, and root inside a jail is the host’s uid 0, only with fewer rights. The safety comes from the kernel’s list of what jailed root may not do. On Linux with a user namespace, “root” inside is just your user outside, so there is nothing of the host’s to lose even if a check were missed. That is why Zygo can run without any privilege, and why a normal user can start a thousand sandboxes without asking an admin. The cost is on the other side: user namespaces open a lot of kernel code to normal users, which is exactly why Zygo’s seccomp filter refuses to let a sandbox create new ones.

Can Zygo be called “jails for Linux”?

Partly — and it is worth being exact. As a picture it fits well: a program locked in its own small world, sharing the kernel, cheap to make. People who know FreeBSD will understand Zygo’s ns backend in one sentence that way. But a jail is usually a long-lived home for a service, built by an admin, and Zygo is the opposite: short-lived, built by any user, often one per request, and it does not run services at all. The honest phrase is “a throwaway jail for every request”: the jail’s walls, with the lifetime of a function call.

Other relatives

FreeBSD was not alone. Solaris Zones (2005) took the same idea further, with its own resource controls and a strong admin model. On Linux, Linux-VServer and OpenVZ were jail-like kernel patches used by hosting companies long before namespaces were finished. LXC (2008) was the first widely used tool to build jail-like “system containers” from the new Linux parts, and Docker began as a layer on top of it. Every one of these, like Zygo, shares one kernel between everything it hosts.

10. Similar projects, and Docker side by side

Many projects build a sandbox from the parts in chapters 2 to 4. They differ in three questions, and once you ask those, most comparisons answer themselves. This chapter asks them, sets docker run and zygo run side by side in detail, and then walks through the projects one by one.

The three questions

1. Where is the wall? The host kernel with locks on it, a second kernel in user space, or a virtual machine with its own kernel.
2. What is ready when a request arrives? Nothing — the sandbox is built each time; the sandbox — a new process enters it; or the sandbox and the loaded program — a copy is forked.
3. Who has to be root? The tool, a daemon, the admin once, or nobody.

Numbers for Zygo in this chapter are the project’s own measurements; chapter 25 names the hosts and the commands. The claims about other projects are theirs or commonly measured, not measured here. They all move quickly; check before quoting.

The map

The three questions give every project a place. The table below puts the first two on a grid: each row is where the wall is, each column is what is already waiting when a request comes in. Read a column from left to right as “less work per request”: on the left everything is built for each call, on the right the program is already loaded and only copied.

  what is ready when a request arrives?

  NOTHING                    THE SANDBOX                 THE SANDBOX + THE PROGRAM
  ───────                    ───────────                 ─────────────────────────
  build the sandbox          enter the sandbox           fork the loaded program
  start the program          start the program           run the request
  run the request            run the request
  tear it all down

  most tools work here       `docker exec`, warm-exec    Zygo `exec`; Sandlock, Zeroboot
  slowest per call                                       fastest per call

The right-hand column held research when Zygo started, not a product: SOCK (USENIX ATC 2018), built into the OpenLambda research platform, forks Python handlers from a zygote on the host kernel (chapter 6). That column is the space Zygo was built for. Since 2026 two more projects fork there, each from a different row: Sandlock and Zeroboot, below. Everything else on this page is a good tool for a nearby job.

What one call costs, tool by tool

The chart shows roughly what one call to a small Python function costs with each tool. The scale is logarithmic: each mark to the right is ten times slower than the one before, so a bar twice as long is not twice as slow but many times slower. A solid bar (█) is the usual cost; a light part (▒) is the range above it. Every bar except the first includes starting Python itself; the first does not, because the warm zygote started Python long before the request.

                          1 ms          10 ms         100 ms        1 s
                          │·············│·············│·············│
Zygo exec (warm fork)     ██                                           1.4 ms
Zygo run, nsjail, kern    ███████████████▒▒▒▒▒▒                        12–40 ms
gVisor                    █████████████████████████▒▒▒▒▒▒              ~60–160 ms
Firecracker microVM       ██████████████████████████████▒▒▒▒           ~140–250 ms
docker run                ███████████████████████████████████▒▒▒▒▒▒▒   300–1000 ms

The Zygo numbers are measured by this project: zygo exec is the warm path’s median, and zygo run the median of python3 -c pass with the image already pulled (chapter 25). The one-shot row is one cluster on purpose. zygo run is not drawn faster than nsjail or kern, because it is not: inside Windmill, nsjail and Zygo in its place cost the same 19–20 ms per job (measured below), and on the Raspberry Pi kern box and zygo run tied (chapter 25). The light part of that bar is where the same tools land with a heavier script or an older kernel. The rows below it are each project’s own claims, here to show the order of size. The lesson is in the shape: the one-shot tools cluster together, because they all pay for building a sandbox and starting Python, and only a warm fork leaves that cluster.

The request path

This table compares what each tool costs on every request, and what it costs once, up front. The Zygo columns are measured; the other columns are the projects’ documented or commonly measured figures.

How to read the Zygo numbers: we timed many requests. “Usually” is what a normal request costs. “1 in 100” is what the slowest requests cost — only one request in a hundred was slower than that.

  fast ◀──────────────── 100 requests, sorted ────────────────▶ slow
        · · · · · · · · · · · · ▲ · · · · · · · · · · · · · · ▲ ·
                                usually                  1 in 100

These are the time a tool adds. Your own code’s time comes on top.

¹ On Linux 6.x, 1 request in 100 waits about 9 ms for the kernel to move it into its cgroup; on Linux 5.10 the same is 2.6 ms (warm) and 3.2 ms (pool). Chapter 25 explains it. Zygo numbers are from a Lima VM, Linux 6.8, 25 September 2026.

² Clean as in a fresh process forked from a zygote that has never served a request. A fork still shares what the zygote had before any request: its memory layout, a socket opened at import time, a literal /tmp/... path. Fork safety goes through each.

The first row is the whole argument. A container’s cost is the machinery around it and the cold start of the interpreter. Zygo takes the machinery off the request path entirely, and it pays the interpreter’s start only once, in a warm zygote. That is not a new observation: SOCK (USENIX ATC 2018) and Catalyzer (ASPLOS 2020) made the same one for serverless platforms, with a zygote and a snapshot respectively (chapter 6).

The three Zygo columns are three answers to “what is warm?”. zygo run keeps nothing warm. A pool keeps the interpreter and its dependencies warm, but no code: each request brings its own script, so one pool serves thousands of different scripts, for about 0.5 ms more than a function. A warm function keeps one handler loaded, which is the fastest, but costs one zygote per script (chapter 13).

  what is warm?        zygo run          pool                 warm function
                       ────────          ────                 ─────────────
  sandbox              built each time   ready                ready
  interpreter + deps   started each time ready                ready
  your code            loaded each time  loaded per request   ready
  per request          12 ms             1.9 ms               1.4 ms

docker run and zygo run, side by side

Both accept the same sentence — IMAGE [COMMAND…] — and both use OCI images from the same registries. The likeness ends there. The next sections compare them on who runs the command, what is left behind, the defaults, networking, images, and the flags you already know.

Who runs the command

docker run is a client. The request crosses a unix socket to dockerd, a root daemon. dockerd hands it to containerd, which starts a containerd-shim, which runs runc. runc sets the container up and exits; the shim stays for as long as the container lives. That is five processes, three RPC boundaries (calls from one program to another), all of them root. Nearly all of the 300–1000 ms is that chain; the isolation itself — the namespaces, the cgroup, the seccomp filter — is about a millisecond of it. Chapter 5 walks the chain step by step.

zygo run is one process, with no RPC and no root. zygo itself calls clone3 into a fresh set of namespaces. The child applies the mount plan, calls pivot_root, joins its cgroup, drops every capability, installs seccomp and Landlock, and calls execve. With the image cached, that takes a median of 12 ms. ps shows zygo with your program directly under it, and nothing in between.

  what ps shows for docker run                what ps shows for zygo run
  ────────────────────────────                ──────────────────────────
  docker (client, your shell)                 zygo (your user)
  dockerd            (root)                     └─ your program
  containerd         (root)
  containerd-shim    (root)
    └─ your program
  (runc started it and has already exited)

What is left behind

docker run creates an object. When the process exits, the container stays (docker ps -a), and its writable layer stays on disk. Something has to docker rm it, or you must have passed --rm. Logs pile up in the daemon, there are restart policies, and docker exec gets you back inside.

zygo run is a process. When it exits, the namespaces, the cgroup and the tmpfs go with it; there is nothing to remove, and there is no zygo rm. Standard input, output and the exit code pass straight through. The one addition: a sandbox that Zygo or the kernel killed exits with code 137, and --outcome file.json says which it was, with timed_out, oom_killed, peak_rss_kb and wall_ms. --dry-run --json prints the mount plan and the cgroup values without running anything; Docker has nothing like it.

The container you keep around and step into is a separate idea in Zygo: zygo serve and zygo exec, a warm function. It looks like docker exec, except that every request is a fresh process and costs about 1.4 ms.

  docker run IMAGE                          zygo run IMAGE
  after exit:                               after exit:
  ┌──────────────────────────────────┐      ┌──────────────────────────────────┐
  │ container record (docker ps -a)  │      │ nothing                          │
  │ writable layer on disk           │      │ exit code passed through         │
  │ logs in the daemon               │      │ 137 if killed; --outcome says    │
  │ → needs docker rm (or --rm)      │      │ timed_out / oom_killed           │
  └──────────────────────────────────┘      └──────────────────────────────────┘

The defaults

Docker’s defaults are made for software you chose; Zygo’s are made for code you did not. This is what you get with no flags at all.

A denylist names what is forbidden and allows the rest; an allowlist names what is allowed and forbids the rest. release_agent is an old cgroup file that has been used to escape containers, so Zygo does not show the cgroup file system inside at all.

Adding locks versus loosening them

In Docker you add safety: --cap-drop ALL --read-only --security-opt no-new-privileges --memory --pids-limit --network none. In Zygo you loosen it, and every flag that does so says it in its name: --allow-unlimited, --allow-host-net, --allow-private-net. There is no way to turn a limit off without one of them.

This is not “Zygo is safer than Docker”. Docker’s defaults suit software you chose; Zygo’s suit code you did not. The point is that each set of defaults leans the way its use case needs. And every row in the Zygo column is something a test attempts, not a setting a test reads.

  Docker:  open ──(--cap-drop ALL, --read-only, --memory, --network none, …)──▶ tight
  Zygo:    tight ──(--allow-unlimited, --allow-host-net, --allow-private-net)──▶ looser

Networking: four modes, none of them a server

Docker’s network is built for a container that is a service: a bridge with NAT (address translation, so many containers share the host’s address), -p 8080:80 to publish a port, containers that find each other by name, and --network host when you want none of it.

Zygo has four modes, and none of them accepts a connection. none is the default. egress is an allowlist by name — allow = ["api.example.com:443", "*.cdn.example.com:443"] — enforced by nftables inside the sandbox’s own namespace, with a DNS resolver of Zygo’s own. A name off the list does not resolve; a name on it has its addresses added to the filter before the answer goes back. full is the public internet. host means no network namespace and needs --allow-host-net.

Private address ranges and 169.254.169.254 (the cloud metadata address, which often hands out credentials) stay refused in every namespaced mode. If pasta or nft is missing, a networked sandbox does not start, instead of starting without its filter. There is no port publishing; Zygo does not run services. Chapter 14 covers the modes in use.

  mode      reaches                                          needs
  ────      ───────                                          ─────
  none      loopback only                                    (default)
  egress    only the host:port names on the allow list       pasta + nft
  full      the public internet; never private or metadata   pasta + nft
  host      everything the host reaches (no namespace)       --allow-host-net

What the sandbox can reach

The same probe, on the same host, was run through Zygo’s --net full and Docker’s --network bridge: a small script that tries to open a connection to each target and reports what the kernel answers.

                      ┌─────────────────────┐
  Docker bridge ─────▶│ host's Postgres     │◀──── reached
                 ────▶│ LAN router          │◀──── reached
                 ────▶│ metadata address    │◀──── routed (refused by host)
                      └─────────────────────┘
  Zygo --net full ──▶ public internet only; private and metadata: no route

Closing the gap in Docker: four rules

Docker needs four firewall rules to close what Zygo closes by default. They drop traffic from containers to the metadata range and the three private ranges.

iptables -I DOCKER-USER -d 169.254.0.0/16 -j DROP
iptables -I DOCKER-USER -d 10.0.0.0/8     -j DROP
iptables -I DOCKER-USER -d 172.16.0.0/12  -j DROP
iptables -I DOCKER-USER -d 192.168.0.0/16 -j DROP

In a product where the code inside the sandbox is written by a tenant (a customer of the product), this is not a matter of taste.

The egress allowlist, end to end

The same host, the same probes, four configurations: a normal HTTPS request, and a raw socket to a public DNS server.

One named host opens and nothing else does, with no proxy process anywhere.

Images and dependencies

Both pull the same images from the same registries, and zygo run pulls on first use exactly as docker run does. The stores differ. Docker’s is /var/lib/docker and belongs to root. Zygo’s is under ~/.local/share/zygo, content-addressed (each file is named by a hash of its contents), with the layers unpacked. It uses rootless overlayfs on kernel 5.11 and newer, and a flattened copy below that. zygo login stores a password for a private registry, and an existing ~/.docker/config.json is read too.

The real difference is the Dockerfile. In Docker, adding a dependency means building an image. In Zygo the image is never touched: --requirements ./requirements.txt builds a venv inside a sandbox with the image’s own pip and mounts it at /venv, and system = ["libwebp7"] installs apt packages into a derived OCI layer. Both are keyed on the image digest and the list, built once, and shared by everything that names the same thing. Chapter 15 has the details.

  Docker                                   Zygo
  Dockerfile ─▶ docker build ─▶ new image   image (untouched)
                                              ├─ --requirements ─▶ venv at /venv
                                              └─ system = [...] ─▶ derived layer
                                            built once per (image digest, list)

The flags you already know

Docker

Docker is a general-purpose system for packaging and running services. Zygo uses its images and none of its runtime. If you need docker build, docker compose, long-running services, port publishing, networks between containers or restart policies, you need Docker. Chapter 5 explains how it works.

nsjail

nsjail, from Google, is the closest older relative of Zygo’s one-shot half. It puts a process into namespaces, cgroups, rlimits and a seccomp filter, which you write in a small policy language called Kafel. It can run a command once, run it again and again, or listen on a TCP port and start a fresh jail for every connection, which made it the standard tool for hosting CTF challenges. Windmill and other job runners can wrap each job in it. It has no images — you give it a folder or bind-mount the host’s — and no warm path: every run starts the program from nothing. Choose it when you want a battle-tested, very configurable jail around a command and you manage the root file system yourself. Zygo in nsjail’s place inside Windmill’s workers cost the same per job; the measurement is further down.

bubblewrap

bubblewrap (bwrap) is the small sandbox tool under Flatpak. It creates namespaces, builds a root from the bind mounts you list, and then runs a command — nothing more, on purpose. It has no cgroup limits and no seccomp policy of its own; you pass a compiled filter if you want one. That smallness is its strength: it is easy to audit and it runs everywhere, which is why many desktop and developer tools use it as a building block. Zygo covers what bubblewrap leaves to the caller: images, limits, the filter, the network allowlist, and the warm path.

firejail

firejail sandboxes desktop programs — a browser, a PDF reader — using ready profiles for hundreds of applications. It is installed setuid root, so any user can start a sandbox, but its own code runs with root’s power, and bugs in it have mattered in the past. It is made for confining the apps on your desktop, not for running server-side functions at high rates. Zygo needs no setuid program at all, because user namespaces give it what it needs.

minijail

minijail is Google’s sandbox library and tool for ChromeOS and Android system services. It applies namespaces, capabilities, seccomp and user changes to a service before it starts, from a policy file. It is part of the operating system’s own plumbing and is written for that setting: known services, written by the same team. Zygo is aimed at the reverse: code written by someone else, arriving at request time.

systemd-nspawn and systemd-run

systemd-nspawn starts a whole operating-system tree in namespaces — “chroot on steroids”, in its own words — and is good for booting a distribution in a container for testing. systemd-run can put any command in a transient unit with limits and many sandbox options. Both mostly need root, and both think in units and machines rather than requests. Zygo uses systemd only where it has to: to get a delegated cgroup under your login.

The three you are really choosing between

Docker, Firecracker, gVisor and Lambda are the landmarks; they are not the shortlist. Someone looking for “run this agent’s code somewhere safe” ends up comparing Zygo with three much closer projects: kern, nono and microsandbox. In two of the three cases the honest answer is that they solve a different problem. Two more, Sandlock and Zeroboot, fork a warm process as Zygo does and have a section of their own below. Their claims below are theirs, not measured here, and every one of them moves quickly.

kern

kern is the closest match to Zygo’s one-shot half today: rootless, daemonless, one static Rust binary, OCI images, namespaces with a seccomp allowlist and cgroup v2, a box made and thrown away per call in single-digit milliseconds by its own figures. If you want docker run without the daemon and without the 300 ms, both projects answer the same question, and kern’s answer is a good one. Measured side by side on a real Python workload, the two are within 3–5% of each other; the numbers are further down.

They part ways on what happens next. kern makes the box cheap enough to throw away every time, so there is nothing to keep warm. Zygo notes that the expensive part is not the box but the interpreter inside it: a Python process with its imports done is 150 ms or more that a per-call box pays again on every call, whatever the box costs. zygo serve pays it once, and zygo exec forks into it for 1.4 ms, with request n running on a copy of the memory the zygote had before request n−1 existed. That warm path, the protocol behind it, and the per-request cgroup, deadline and secrets that hang off it are Zygo’s real subject. The one-shot runner is the part that had to exist underneath it.

kern also has something Zygo lacks: virtual resource slices (vcpu:, vdisk:, vgpio:) declared in a config file and attachable to a bare host process. Zygo has no equivalent and no plans for one.

  kern, per call                         Zygo exec, per call
  ┌──────────────────────────────┐       ┌──────────────────────────────┐
  │ make box (cheap)             │       │ fork warm zygote     ~1.4 ms │
  │ Python start+imports ~150 ms │       │ (Python + imports paid once) │
  │ run, then destroy box        │       │ run, then child exits        │
  └──────────────────────────────┘       └──────────────────────────────┘

nono

nono is not so much a rival as a different layer, and the words overlap, so it is worth saying so. nono applies Landlock and seccomp — Seatbelt on macOS — to a process you are starting anyway: your coding agent, running as you, with your files. There is no image, no namespace, no cgroup and no runtime. What it gives you is that the agent cannot read ~/.ssh or reach a host you did not allow, enforced by the kernel and impossible to undo once applied. It works natively on a Mac.

That is the right tool for confining an agent that is meant to edit your working folder. It is the wrong one for running code the agent wrote, because that code still runs as you, in your files, with your environment — a narrower version of you, but still you. Zygo is the other half: the agent stays outside, and the code it generates goes into a sandbox with its own root file system, its own pid namespace, a memory limit and a deadline. The two fit together, and on a developer’s machine using both is reasonable.

  ┌─────────── nono: confines the agent, running as you ───────────┐
  │  coding agent (can edit your folder, cannot read ~/.ssh)       │
  │      │                                                         │
  │      └─ generated code ─▶ zygo run / zygo exec                 │
  └──────────────────────────────┬─────────────────────────────────┘
                                 ▼
             ┌─────── Zygo sandbox: own root, own pids ─────────┐
             │ memory limit · deadline · cannot see your files  │
             └──────────────────────────────────────────────────┘

microsandbox

microsandbox is the closest project to Zygo’s vm backend, and further along that road. It is microVM-first: every sandbox is a libkrun guest with its own kernel — libkrun is the same library behind Zygo’s vm backend. It supports OCI images, claims a boot under 100 ms, and can snapshot and branch a live sandbox, which Zygo cannot do at all. It ships Python, TypeScript and Rust SDKs and an MCP server, as Zygo does.

The difference is where the default sits. microsandbox’s wall is hardware for everything. Zygo’s default is the host kernel, with hardware available as --isolation vm for the work that needs it — the same spec and the same command. That is a real trade, and it does not go one way. A microVM per request is a wall a kernel bug does not cross, and Zygo’s ns backend is one kernel away from the host, as chapter 23 says in as many words. What Zygo has instead is the warm path — 1.4 ms, a fork, clean state — which a VM per request cannot reach, and which is the only reason the project exists.

If your code is truly hostile and 100 ms per call is affordable, microsandbox’s default is the safer one. If you run a thousand short calls a minute from your own users’ scripts, Zygo’s is the faster one, and --isolation vm is there for the part that is not safe to run on ns. Zygo’s vm backend is also, today, much less than microsandbox: it boots a guest and runs one-shot sandboxes, and warm functions and networking inside the guest are not built; ADR 0002 says why.

Sandlock and Zeroboot: two other forks

Two projects from 2026 fork a warm process for each call, as zygo exec does. Each does it from a different row of the map. Their numbers below are their own claims, from their pages, not measured here.

Sandlock (Multikernel) confines a process with Landlock and seccomp — no namespaces, no cgroups, no image. Its template mode starts a Python process once, lets it run its init(), then forks it for each call: about 0.7 ms a clone by its own figures, with the interpreter’s state and imported modules shared copy-on-write. It is a library with Rust, Python and Go bindings, a CLI, and a shim that lets it stand in as an OCI runtime; its README names Linux 6.12 for the current release. On the map it is the bottom row: the clone runs as you, in your filesystem narrowed by Landlock, with no root of its own, no pid namespace and no cgroup. Zygo’s fork lands in a sandbox with its own root, its own pid namespace, a cgroup and a deadline per request, and the supervisor, tenants, secrets and HTTP API around it.

Zeroboot snapshots a Firecracker microVM with the runtime loaded and maps the snapshot’s memory copy-on-write for every new VM: 0.79 ms usually and 1.74 ms for 1 in 100 by its own figures, each fork a VM with a kernel of its own. A fork has no network — serial I/O only — and one vCPU; it needs KVM, and the project calls itself a working prototype that is not production- hardened. It is the top row’s answer to the same question, and the one to watch for T3 code, where Zygo has only a one-shot vm today.

  Sandlock clone                  Zygo exec                       Zeroboot fork
  ──────────────                  ─────────                       ─────────────
  fork a confined process         fork a sandboxed zygote         fork a VM snapshot
  Landlock + seccomp              namespaces, cgroup, seccomp,    KVM, own kernel
  your files, narrowed            Landlock; own root, own pids    no network, 1 vCPU
  ~0.7 ms (its claim)             1.4 ms (measured, chapter 25)   0.8 ms (its claim)

gVisor

gVisor is a kernel written in Go that runs in user space. Your program’s syscalls go to it, not to the host, and it answers most of them itself, using only a small, filtered set of host syscalls. That gives a much smaller attack surface than the host kernel, without needing KVM or a virtual machine. The cost is speed on syscall-heavy work, and some programs that need rare kernel features. Its runtime, runsc, is an OCI runtime. Zygo’s gvisor backend uses it for one-shot runs: zygo backend install gvisor, then zygo run --isolation gvisor. Warm functions stay an ns feature, for the reason given in chapter 8.

Firecracker, Cloud Hypervisor, and platforms on them

Firecracker is the microVM monitor behind AWS Lambda and Fargate: a tiny virtual machine per workload, with a real kernel of its own, booting in about 125 ms or restoring from a memory snapshot faster still. Cloud Hypervisor is a similar microVM monitor. They give the strongest wall on this page, at a boot cost per VM, and they need KVM, so they rarely run inside another cloud VM. Platforms such as E2B build hosted sandboxes for AI agents on top of Firecracker. Its snapshot restore is the one thing here that resembles Zygo’s fork, one level down: a whole VM restored instead of a process copied.

Zygo’s vm backend is built on libkrun for the same purpose: anonymous code, not your own. It boots a guest and runs one-shot sandboxes, with a private writable layer over the image. Warm functions and guest networking are refused on it by decision, not by omission; ADR 0002 explains why.

Kata Containers

Kata Containers runs each container, or each Kubernetes pod, inside a lightweight virtual machine, while still looking like a normal container to Kubernetes. It gives you a hardware wall without changing how you deploy. The price is a VM’s start-up time and memory for every pod. It is built for long-lived services on a cluster, where Zygo is built for short calls on one machine.

AWS Lambda and its relatives

Lambda and similar services are managed platforms. Zygo is a local runtime with a similar shape — a function, a warm instance, a request — and no platform around it: no billing, no scaling across machines, and no ingress (accepting connections from outside).

Hosted sandboxes for agents: E2B, Modal, Daytona and the rest

Since 2025 a new group of products sells a sandbox for an AI agent: E2B, Modal Sandboxes, Daytona, Vercel Sandbox, Cloudflare Sandboxes, Blaxel, Docker’s own Sandboxes, and more every quarter. The README names three of them; this section says where they sit on the map above, because the word “sandbox” covers two different things.

What they are. Each gives an agent a session: a machine of its own with a filesystem, a shell, packages it can install, ports it can expose, and a lifetime of minutes to hours, billed by the second. The wall is a microVM (Firecracker at E2B and Vercel, a custom monitor at Docker) or gVisor (Modal), and the session can often be paused, snapshotted and resumed. They run in the vendor’s cloud; some can be self-hosted, at the cost of running their control plane. Everything in this paragraph is their own description, not measured here.

Where they sit. On the map they are the top-left cell — a virtual machine, built for each session — with one addition: the session stays. That is the right shape for an agent that writes code, runs it, reads the error and tries again for half an hour. It is the wrong shape for what Zygo is for: a function that runs for milliseconds, thousands of times, and must start clean each time. A session per call would cost a VM boot, or a snapshot restore, per call.

  a hosted agent sandbox                 a Zygo warm function
  ──────────────────────                 ────────────────────
  one session, minutes to hours          one call, milliseconds
  state kept between commands            no state between calls
  a VM (or gVisor) per session           a fork per call, on one kernel
  in their cloud, billed per second      on your machine, no billing
  pause, snapshot, resume                nothing to resume: warm again
  ports, a shell, a desktop              no ingress, no shell in the request

Choosing. If an agent needs a machine to work in — install packages, run a server, keep files between steps — use one of these, or microsandbox on your own hardware. If a program needs to run many small pieces of untrusted code — an agent’s tool calls, a customer’s plugin, a workflow step — and each must be cheap and clean, that is Zygo. The two combine: an agent living in a hosted session can still call a Zygo function for the tool that must answer in a millisecond.

runc, crun and youki

These are the low-level OCI runtimes: given a folder and a config.json, they do the list at the end of chapter 4 and exec the program. runc is written in Go, crun in C, youki in Rust. They are the part of Docker and Podman closest to zygo run, and on their own they are fast. But they expect someone else to prepare the bundle, pull images, keep records and clean up — which is the chain from chapter 5. Zygo does its own launching, so it needs none of them.

Workflow engines: Windmill and friends

Windmill, Temporal and similar workflow engines run users’ scripts, and they need a sandbox for each run. Today that is often nsjail or a container per job. These engines are exactly the embedder Zygo is designed for — the program that builds Zygo into itself. A worker calls zygo serve once per script, or uses the SDK, and each script run becomes a fork() instead of a container. examples/workflow-engine/ is such a worker.

A prefork pool of your own

The first question an embedder asks is a fair one: why not fork the warm interpreter yourself? Python can. A prefork pool loads the code once and forks workers from it. Python’s multiprocessing has a forkserver start method that imports the modules you name once and forks a child from them for each task. gunicorn --preload --max-requests 1 loads the application in its master and replaces each worker after one request. Either gives the speed trick zygo exec uses: the imports are paid once, and each request runs in a fresh copy.

What neither gives is a wall. The child runs as your user, in your file system, on your network, with no memory, process or time limit of its own unless you add one. For code your own team wrote, that is enough, and simpler than Zygo: use it. For code a customer or an agent wrote, the fork is the easy half. The hard half is what Zygo puts around the copy before it runs, which chapter 6 walks through. Node cannot fork a running process at all, so there a prefork pool is a pool of pre-loaded workers, as Zygo’s Node agent keeps (chapter 13).

  your own prefork pool                  zygo exec
  ─────────────────────                  ─────────
  fork the warm interpreter              fork the warm interpreter
  the child runs as you                  own user, pid, mount and network namespaces
  your files, your network               its own root; network off unless allowed
  limits: whatever you add               a cgroup and a deadline per request
  no syscall filter                      seccomp allowlist, Landlock
  right for your own code                right for code others wrote

WebAssembly runtimes

Wasmtime, Wasmer, Spin and Extism run code compiled to WebAssembly (Wasm): a portable instruction format that runs inside the runtime’s own process. A Wasm module can touch only the memory it was given and the host functions it was handed, so the wall has no kernel in it at all. A new instance starts in well under a millisecond, by their own figures. Tools such as Wizer can even run a module’s start-up once and save the memory it leaves, which is the Wasm form of a zygote.

The price is the compile step. Your code, and every library it imports, must be built for Wasm. Pure Python and JavaScript run on interpreters compiled to Wasm, but a package with native parts, such as numpy or a database driver, needs a Wasm build of its own. Some projects, Pyodide for one, ship builds of popular packages; most of PyPI and npm has none. A module sees files and the network only as far as the host opens them to it. Zygo makes the opposite trade: any program in any OCI image runs unchanged, and the wall is the host kernel, with what that costs (chapter 23). If your plugins are small, self-contained and built with a toolchain you control, Wasm is the stronger boundary. If they install whatever they import, that is Zygo’s column.

Measured: Zygo against nsjail and kern

nsjail and kern are the two one-shot runners closest to zygo run, so both were measured against it under load, on 24 September 2026. Every run happened in the same VM: Ubuntu 24.04, kernel 6.8, aarch64, 2 vCPU, 4 GB, on an M1 Max. Every binary ran from the VM’s own disk. The load generator ran outside the VM, so its CPU counts for nobody. “CPU per job” is the CPU of the whole stack being measured, divided by the jobs it finished. Unlike the numbers in chapter 25, these tables cannot be repeated from this repository: the load generator and the raw results were not kept (what these numbers are not).

Inside Windmill, in place of nsjail

Windmill CE v1.817 with three general workers ran a trivial Python script and a CPU-bound one (about 20 ms of Python) through nsjail, using Windmill’s own nsjail config. Then the same workers ran them through Zygo in nsjail’s place, and nothing else changed. Zygo ran two ways:

• Zygo as nsjail: the zygo binary, installed under the name nsjail, read nsjail’s command line and config itself. This translation was built for the benchmark only and is not in Zygo today.
• Zygo through a script: a shell stand-in translated the config and called zygo run. This is what works with Zygo as it ships.

Per job, 40 runs in a row inside a worker:

Under load, the whole Windmill stack. nsjail was measured twice; its ranges cover both runs.

Level per job, and 1–6% behind at saturation, while doing more. Zygo gave every job a cgroup with memory and process limits, a seccomp allowlist and a Landlock ruleset, and Windmill’s nsjail config sets none of those. The likely cost at saturation is the cgroup Zygo creates and removes per job: lru_gen_online_memcg, cgroup_addrm_files and tg_set_cfs_bandwidth show in perf. That was not measured on its own. Below saturation the two cannot be told apart. Through a shell script, the same swap costs about 15% of throughput, because the script’s sh, awk, grep and env add about 4 ms to every job. Idle memory does not move, because neither sandbox stays resident between jobs.

Running it found three defects in Zygo, all fixed:

• Two runs could share a staging directory. The three workers shared one Zygo store, but each had its own PID namespace, and a staging root was named after the PID. 2 jobs in 200 failed. Every per-process name now carries the PID and 64 random bits.
• A writable mount of a single file never started. Landlock was given directory rights on a regular file, and the kernel answers that with EINVAL. nsjail configs hand a job its result.json exactly this way. A rule on a file is now narrowed to the file rights.
• The stand-in script itself cost 4 ms a job, as described above.

Inside n8n, as its Code-node runner

n8n runs a Code node through a task runner, a process apart from n8n. examples/n8n-runner is one that sends each task to a Zygo runtime pool, and was measured against n8n 2.38.7’s own runners behind the same n8n (chapter 25 has the method and every table).

The two languages go opposite ways for one reason. n8n’s JavaScript runner runs every task in one Node process, which is cheap and shares everything; its Python runner starts a process per task and pays about 200 ms of CPU for it. Zygo pays for a process per task in both, about 5 ms in Python and 25 ms in Node, which cannot be forked. The Zygo runner covers the Code node’s items and both run modes, not n8n’s RPC helpers or binary data.

An embedder’s harness, against kern

The workload was a real embedder’s Python harness: it reads an event, runs a user’s handler, and writes the result through a read-write scratch mount, under the embedder’s limits. /bin/true measured the runtime alone. There were 64 runs at each concurrency from 1 to 32, twice. The ranges cover both passes and every concurrency level. kern bc822de ran with --security-profile untrusted.

python:3.12-slim ships no bytecode. Zygo compiles it once into a layer of its own, automatically (chapter 15). kern does not, so it is shown both as it ships and with an image precompiled by hand.

On the real workload Zygo is within 3–5% of kern at its best, and more than twice as fast as kern as it ships. On an empty program kern is about 2 ms of CPU per run cheaper. That is the start-up floor: a 7.9 MB binary against 2.2 MB, a larger plan (≈1.5 ms), and Landlock, which kern does not apply by default and Zygo keeps.

Before this comparison, Zygo made 42 harness runs a second at 45 ms of CPU each. The comparison found these, all fixed:

1. Zygo could not run from a delegated cgroup that also held its caller, a systemd unit with Delegate=yes or a service in a container. It exited 125. zygo.slice now goes to the top of the tree delegated to the user.
2. Every run re-executed under a transient systemd scope, about 10 ms of CPU, because the delegation check asked about the wrong cgroup.
3. The host probe ran on every run. It is now cached per boot for up to ten minutes.
4. Zygo moved its own process into a cgroup on every run, 1.9–6.0 ms of a 6–11 ms start. It now moves only when it is in the way.
5. The sandbox child was moved into its cgroup after clone3, which waits out an RCU grace period after a quiet spell. It is now born there with CLONE_INTO_CGROUP, the order kern uses, which also puts the limits on from the child’s first instruction. The sandbox start went from 6–11 ms to 3–4 ms.
6. The exit wait slept with a backoff, so a program that exited at 13 ms was noticed at 25 ms. It now uses pidfd_open and poll.
7. Every run built a registry client, with TLS roots and a multi-threaded runtime, to read one local file.
8. A failed bytecode build was retried on every run, at 170–200 ms each. It is now remembered for an hour.

A run now starts 4 processes, down from 13 at the worst.

What these numbers are not

• They are all cold. Every job started a fresh interpreter. Zygo’s warm path, a fork into a zygote that has already imported everything, was not part of either comparison. Neither nsjail nor kern has one to compare it with.
• One VM, one kernel. On kernel 5.10, in Docker Desktop’s VM, the same sandbox made Python about twice as slow as a plain container did. That cost belongs to the old kernel, and it would have been measured against every namespace-based runner alike.
• The raw results and the load generator were not kept. The tables here are their summary, and they cannot be repeated from this repository. The numbers in chapter 25 can: make bench-record writes them as JSON, and bench/ holds the records so far.

Everything in one table

What Zygo does not do

• Run on macOS or Windows natively. Sandboxes are Linux; on a Mac, Zygo manages a Linux VM for you.
• Provide ingress. No mode accepts connections; a function is called through the CLI, the SDKs or Zygo’s own HTTP API.
• Scale past one machine. Capacity is a per-host budget, and requests past it get HTTP 429 (too many requests).
• Hide the kernel. The ns backend is one kernel, and every chapter of this book says so.

Choosing, in short

For long-lived services, use Docker, Podman or Kubernetes. For hostile code where 100 ms or more per call is fine, use a VM wall: Firecracker, Kata, microsandbox, or Zygo’s vm backend. For confining a tool you already run, look at nono or bubblewrap. For a quick throwaway sandbox around one command, nsjail, kern and zygo run all do well. For your own code, a prefork pool is enough; for small plugins built for Wasm, a Wasm runtime is the stronger wall. For many short calls to code other people wrote, where each call must start clean and costs must stay in milliseconds, that is the right-hand column of the map, and that is Zygo.

In one sentence: docker run asks a root daemon to create a container object from an image; zygo run runs a program as a locked-down process under your own user — the same kernel parts, the opposite defaults, no chain in between, and nothing left behind.

11. Getting started

This chapter takes you from nothing to three warm functions behind an HTTP API. You install Zygo, ask it whether your machine can run sandboxes, run one program in a sandbox, and then keep a function warm and call it. Each step is short, and each one links to the chapter that explains it in full.

The road through this chapter

  install ──▶ zygo doctor ──▶ zygo run ──▶ zygo serve ──▶ sandbox.toml ──▶ zygo api
  (a file)    (can this       (one        (one warm      (three          (call them
               host do it?)    sandbox)    function)      functions)       over HTTP)

You can stop after any step. A CI job that only needs zygo run never has to learn about warm functions, and a Mac user can do all of it without setting up Linux by hand.

The five-minute version

If you would rather type first and read later, this is the whole road in one block. Every line is explained further down the page.

# install: Linux, x86_64 or aarch64 (a Mac: brew install mhmtskrc2/zygo/zygo)
url=https://github.com/mhmtskrc2/zygo/releases/latest/download
curl -fsSLO "$url/zygo-$(uname -m)-unknown-linux-musl.tar.gz"
curl -fsSL "$url/SHA256SUMS" | sha256sum -c --ignore-missing
tar xzf zygo-*-unknown-linux-musl.tar.gz && sudo install -m 0755 zygo-*/zygo /usr/local/bin/

zygo doctor                                   # can this host run sandboxes? --fix if not
zygo run python:3.12-slim python3 -c 'print("hello")'   # one sandbox, thrown away

echo 'def handler(event): return {"got": event}' > handler.py
zygo serve ./handler.py --name echo           # warm it once: about 150 ms
zygo exec echo '{"n": 1}'                     # a fresh fork of it: about 1.4 ms
zygo stop echo

If zygo doctor reports something red, its section below says what each line means, and zygo doctor --fix applies the usual fixes for you.

What you need

Zygo is one static binary: a single file that carries everything it needs, with no libraries or runtimes to install beside it. It builds sandboxes out of Linux kernel features, so the sandboxes themselves always run on Linux.

On Linux you also need unprivileged user namespaces (a normal user may make the private views from chapter 2) and cgroup v2 delegation (a normal user may own part of the cgroup tree from chapter 3). Most modern distributions have the first; the second often needs one small fix, which zygo doctor can apply for you.

Installing on Linux

Release builds exist for x86_64 and aarch64 (64-bit ARM, such as a Raspberry Pi 5 or a Graviton server). Both are static musl builds: musl is a small C library that is linked into the binary, so it does not depend on the host’s own libraries.

url=https://github.com/mhmtskrc2/zygo/releases/latest/download
curl -fsSLO "$url/zygo-$(uname -m)-unknown-linux-musl.tar.gz"
curl -fsSL "$url/SHA256SUMS" | sha256sum -c --ignore-missing   # prints "OK"
tar xzf zygo-*-unknown-linux-musl.tar.gz
sudo install -m 0755 zygo-*/zygo /usr/local/bin/zygo

uname -m prints x86_64 or aarch64, which picks the right file. The second line checks the archive against the SHA256SUMS file that every release carries, so you know it is the one that was published. The sudo is only for copying the file into /usr/local/bin; Zygo itself never runs as root.

SHA256SUMS is itself signed, in every release after 0.1.1. With cosign installed, this proves the checksums came from this project’s release workflow, not only from the same download page:

curl -fsSLO "$url/SHA256SUMS"
curl -fsSLO "$url/SHA256SUMS.sigstore.json"
cosign verify-blob SHA256SUMS --bundle SHA256SUMS.sigstore.json \
  --certificate-identity-regexp '^https://github\.com/.*/\.github/workflows/release\.yml@refs/tags/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Each release also carries an SBOM — a list of every library compiled into the binary, with its version — as zygo-<version>.cdx.json, in the CycloneDX format that security scanners read.

Installing on macOS

On a Mac, Homebrew installs three things: a small zygo shim for macOS (a thin program that passes your commands on), the Linux build of Zygo that the shim forwards into, and Lima, the tool that starts the Linux VM.

brew install mhmtskrc2/zygo/zygo

The formula lives in a tap (a small GitHub repository of Homebrew formulas), mhmtskrc2/homebrew-zygo; Homebrew finds it from the name. The release build writes the formula there, with checksums computed from the same archives, not typed in by hand. The section How Zygo runs on a Mac below explains what the VM is and what it costs.

Other ways to install

# from crates.io, anywhere with a Rust toolchain
cargo install zygo-cli

# from a checkout of the repository
cargo build --release
sudo install -m 0755 target/release/zygo /usr/local/bin/zygo

cargo install zygo-cli builds the default set of features. The vm backend is not in it. That backend links a virtual machine monitor from a pinned git tag, and a crate published on crates.io may not point at a git tag. So make vm-build from a checkout is the way to get it, and zygo doctor tells you this on a host where vm would otherwise work. Chapter 16 explains the backends.

The container image

Zygo also ships as a container image, ghcr.io/mhmtskrc2/zygo. It is Alpine Linux plus the static zygo binary and the few helpers networking needs. It does not need --privileged, but it does need a few specific things from Docker, because Zygo builds sandboxes inside it:

docker run --user 0:0 --security-opt seccomp=unconfined \
    --security-opt systempaths=unconfined --security-opt apparmor=unconfined \
    --cgroupns=host --cgroup-parent=/zygo -v /sys/fs/cgroup/zygo:/sys/fs/cgroup/zygo:rw \
    -p 7700:7700 -e ZYGO_API_TOKEN=... ghcr.io/mhmtskrc2/zygo

Sandboxes with a network need --device /dev/net/tun as well, and a volume keeps pulled images across restarts; chapter 16 has the full command. zygo doctor names anything that is missing, inside the container just as on a host. packaging/oci/ has the details, and chapter 16 covers running it in production.

On Windows, through WSL2

WSL2 is a real Linux kernel in a small VM that Windows manages, so Zygo runs there as on any Linux host. The project has not tested it yet — no CI runner or development machine here is Windows — so treat this as the route that should work, and tell us if it does not. Three settings matter:

1. A distribution with systemd. In Ubuntu 24.04 under WSL2, put this in /etc/wsl.conf, then run wsl --shutdown from Windows:

   [boot]
   systemd=true
   

   systemd is what gives your user a delegated cgroup, as on a desktop Linux.

2. cgroup v2 only. WSL2 can mount the old cgroup v1 hierarchy beside v2, and Zygo needs v2 on its own. In %UserProfile%\.wslconfig on the Windows side, then wsl --shutdown again:

   [wsl2]
   kernelCommandLine = cgroup_no_v1=all
   

3. Install as on Linux, with the commands above, then run zygo doctor. It checks the kernel version, user namespaces and cgroup delegation there, and zygo doctor --fix applies what it can.

WSL2’s kernel is 5.15 or newer, well above Zygo’s 5.3 floor; below 6.1 doctor calls it degraded, as on any host, and it reports whether the kernel was built with Landlock. There is no /dev/kvm by default, so the vm backend is off. Troubleshooting covers what doctor reports.

In a dev container or a Codespace

The repository has a .devcontainer for VS Code’s Reopen in Container and for GitHub Codespaces. It builds Zygo, installs the tools the tests and networked sandboxes need, and runs zygo doctor. Sandboxes work inside it: one-shot runs, warm functions and egress networking were checked in it.

It needs two things a default container does not give. The container is started --privileged, so Zygo may create user namespaces and mount inside them. And a small script arranges the container’s own cgroup tree at every start, because cgroup v2 hands controllers only to a cgroup with no processes in it. zygo in that container is a wrapper that starts the real binary in the empty cgroup the script left for it.

  /sys/fs/cgroup   (the container's own root: memory, pids, cpu for children)
  ├── init/        every process the container started, and each new shell
  └── launch/      empty until zygo starts in it
      └── zygo.slice/ …  the sandboxes, as on a host

What does not work there: the vm backend (no /dev/kvm), and on a host whose kernel is older than 5.13, Landlock; zygo doctor names both. It is a development machine, not a production shape — chapter 16 covers running Zygo in a container for real, without --privileged.

How Zygo runs on a Mac

macOS has no namespaces or cgroups, so a Mac cannot build a Linux sandbox itself. Zygo solves this by keeping a small Linux virtual machine (VM): a whole computer simulated in software, with its own Linux kernel. You never log into it. The zygo command on your Mac passes every sandbox command to a Linux zygo inside the VM, with the same arguments, the same working folder and the same input and output. The exit status comes back out to your shell.

  your Mac                                     Lima VM "zygo" (Ubuntu 24.04)
  ┌───────────────────────────────┐            ┌─────────────────────────────────┐
  │ $ zygo run python:3.12 ...    │   SSH      │ zygo run python:3.12 ...        │
  │   zygo (macOS shim) ──────────┼───────────▶│   └─▶ sandbox (namespaces,      │
  │                               │  ~22 ms    │        cgroup, seccomp ...)     │
  │ output + exit status ◀────────┼────────────┼── output + exit status          │
  │                               │            │                                 │
  │ /Users/you/project  ◀─────────┼── same ────┼─▶ /Users/you/project            │
  │                               │   path     │   (your home, mounted inside)   │
  └───────────────────────────────┘            └─────────────────────────────────┘

The Mac VM in detail

Your home folder is mounted inside the VM at the same path, and it is writable. So ./handler.py is one file, seen from two sides. That is also the limit, and Zygo enforces it: every host path must be under $HOME. A command run from outside your home folder is refused, and the message names both folders instead of quietly running somewhere else.

A few commands never need Linux, so they run on the Mac itself: zygo completion, zygo doctor, zygo agent test and zygo api --openapi. If the VM cannot be reached, a command exits with status 111; chapter 22 says what to do.

What the Mac VM costs

A one-shot zygo run python:3.12-slim true typed in a Mac shell takes about 29 ms end to end, when a supervisor is running in the VM. About 6 ms of that is the sandbox, and about 22 ms is the trip into the VM. true starts no interpreter; the 12 ms quoted elsewhere for a one-shot run is python3 -c pass, with Python’s own start inside it. These are medians of nine runs on the Mac that chapter 25 names. The same run through Docker Desktop on the same Mac took 397 ms.

Why does a one-shot run care about a supervisor? For its cgroup. Without one, zygo run on a systemd login must make a transient scope of its own and re-execute itself inside it, which costs about 12 ms more (25.7 against 13.5 ms inside the Lima VM, chapter 25). With a supervisor running, the run is handed to its cgroup tree instead.

The trip is paid per command, not per request. The millisecond warm path is still there when you call functions through the HTTP API or the SDKs, because then the calls happen inside the VM and the connection pays the trip once. zygo api running inside the VM is the answer for anything that must be fast on a Mac.

The Mac VM from a checkout

Without Homebrew you need the same two parts: Lima, and a Linux build of Zygo to put inside the VM.

brew install lima            # what starts the VM
make guest-build             # the Linux build that runs inside it, compiled in the VM

make guest-build needs nothing but the VM. It installs a Rust toolchain inside the VM on first use. On a Mac that already has Docker, make tests/linux/bin/zygo-linux-musl builds the same binary in a Docker container instead.

Checking the host: zygo doctor

zygo doctor is the first thing to run on any new machine. It checks everything a sandbox needs, prints one line per check, and prints the fix for anything that is missing. It tries each thing rather than reading a setting. For example, it really builds a user namespace and mounts inside it. So a kernel that has a feature switched on but ignores it is caught here, not later inside a sandbox.

zygo doctor           # can this host run sandboxes?
zygo doctor --json    # the same report, for a script or a health check

On a Mac, doctor checks both sides: the Mac’s side (can it reach the VM?) and the VM’s kernel. You get one report with one verdict.

What doctor checks

The last lines of the report list the backends this host can use right now: the host has what each one needs and this build of Zygo includes it.

Reading the report

Each line ends in one of four words. The exit status of zygo doctor is 0 exactly when no line says FAIL, and the JSON field ok says the same thing.

network = "none", the default, needs no networking tools at all. So a missing pasta is absent, not FAIL, and only matters when you turn a sandbox’s network on.

Which kernel version gives what

Zygo needs kernel 5.3 or newer. Later kernels unlock extra features rather than block Zygo. 6.1 or newer is recommended, because from there cgroup.kill and memory.peak are both present. Landlock’s network rules come later, in 6.7: below it, bind and connect are limited by the network namespace and the firewall alone.

  5.3 ─────────── 5.11 ─────────── 5.13 ─────────── 6.1 ─────────── 6.7 ─────────▶
  minimum         overlayfs in a   Landlock         recommended:    Landlock
  to run          user namespace   (file access     cgroup.kill,    network rules
                  (below: layers   rules)           memory.peak
                  are flattened,
                  more disk, slower
                  first run)

Letting doctor fix things: --fix

zygo doctor --fix applies the fixes it knows. It first prints a plan: each change, why it is needed, what it costs, and the exact commands. Then it asks apply these? [y/N]. Nothing happens unless you type y. In a script, where nobody can answer, add --yes.

zygo doctor --fix           # show the plan, ask, then apply
zygo doctor --fix --yes     # the same, without asking

Ubuntu and Debian: two AppArmor rules

AppArmor is a Linux security module that limits what programs may do. Ubuntu and Debian ship two AppArmor rules that get in Zygo’s way. Both come from the distribution, not from Zygo.

The first is kernel.apparmor_restrict_unprivileged_userns=1. It lets a normal process create a user namespace, and then refuses the first mount inside it. That mount is the first thing every sandbox does. zygo doctor finds this by trying the mount, and zygo doctor --fix installs an AppArmor profile that lets the zygo binary alone past it. The restriction stays on for every other program. Chapter 23 says what the rule protects.

The second is an AppArmor profile for pasta, the program Zygo uses to give a sandbox a network. Where that profile is enforced, it keeps pasta out of the sandbox’s user namespace. Then network = "egress" and "full" cannot start, even though /dev/net/tun works. The error says so and names aa-complain. The default, network = "none", does not use pasta at all.

cgroup delegation, the usual gap

On a machine with systemd, delegation is the most common missing piece, and it has two halves. First, your user manager (the systemd process that looks after your login) has to pass the controllers down to you. This is the fix doctor --fix writes, and you can also do it by hand:

mkdir -p ~/.config/systemd/user/user@.service.d
printf '[Service]\nDelegate=cpu cpuset io memory pids\n' \
    > ~/.config/systemd/user/user@.service.d/delegate.conf
systemctl --user daemon-reexec

Second, the process that runs Zygo has to sit in a delegated cgroup, and a login shell over SSH does not. Zygo handles this half itself, the way Podman does. A command that builds a sandbox restarts itself inside a temporary systemd scope of its own, which is a delegated cgroup made for one command. Chapter 12 has the details.

  your SSH login shell's cgroup      (not delegated: cannot hold a sandbox)
     │
     └─ zygo run ...  ──re-exec──▶  systemd-run --user --scope -p Delegate=yes
                                        └─ zygo run ...   (delegated: builds the sandbox)

One result surprises people. zygo doctor run from a plain SSH session can report cgroup v2 … FAIL, because that is true of the cgroup it stands in. zygo run from the same shell still works, because it moves into a scope first. Chapter 22 has the rest.

Your first one-shot sandbox

A one-shot sandbox is built for one program and thrown away when the program ends. This is zygo run, and it reads like docker run:

zygo run python:3.12-slim python3 -c 'print("hello")'

The first run downloads the image, just as docker run does. After that, a run takes about 12 ms (median, with the image already pulled, measured on the hosts in chapter 25). The sandbox has a read-only root, no network, no capabilities, and memory, CPU and process limits. A few more to try:

zygo run --mem 128M --pids 16 --timeout 10s alpine:3 /bin/sh   # tighter limits
zygo run --tty alpine:3 /bin/sh                                # with a terminal of its own
zygo run --dry-run --json python:3.12-slim                     # the plan, without running it

Chapter 12 covers zygo run in full: mounts, environment, exit codes and more.

Your first warm function

A warm function is a sandbox that is built once and kept ready, so each request skips the start-up. You write a handler: a function that takes one event and returns a result.

mkdir demo && cd demo
cat > handler.py <<'EOF'
def handler(event):
    return {"doubled": event.get("n", 0) * 2}
EOF

zygo serve handler.py --name double      # python:3.12-slim is in the store from the run above
zygo exec double '{"n": 21}'             # {"doubled": 42}
zygo ps                                  # what is running

serve starts a supervisor in your session if none is running: the background process that keeps warm functions alive. Then it warms a zygote: the Python interpreter with handler.py already imported, waiting. Every exec is a fork() of it, which means a copy made in a moment. So there is no import cost per request and no state left over between requests. Chapter 6 explains the idea.

What the warm function costs

  zygo serve                         zygo exec          zygo exec          zygo exec
  ├─ build sandbox                   ├─ fork            ├─ fork            ├─ fork
  ├─ start Python, import handler    └─ reply           └─ reply           └─ reply
  └─ park the zygote                   ~1.4 ms            ~1.4 ms            ~1.4 ms
     ~150 ms, paid once

On a Raspberry Pi 5, a Python handler that imports nothing is warm in about 150 ms, including starting the supervisor. The median overhead per request is about 1.4 ms, measured on a Lima VM on an Apple M1 Max. Both numbers and their machines are in chapter 25. The same trick works from a one-line file:

echo 'def handler(event): return {"got": event}' > handler.py
zygo serve ./handler.py --name echo
zygo exec echo '{"n": 1}'
zygo bench all            # every published number, measured again on this host

A project with three functions

A real project describes its functions in a file called sandbox.toml, so you do not repeat flags. Here are three functions that each show a different shape: a Python function with system packages, a Go binary, and a function that may call one outside API.

# sandbox.toml
[defaults]
image = "python:3.12-slim"
mem   = "256M"

[fn.resize]
entry        = "./resize.py"
requirements = "./requirements.txt"    # a venv, built once inside the image
system       = ["libwebp7"]            # apt packages, once, as a layer
mem          = "512M"

[fn.parse]
image  = "alpine:3"                     # no runtime → warm-exec
mounts = ["./bin/parse:/app/parse:ro"]  # a static Go binary
cmd    = ["/app/parse"]

[fn.fetch]
entry   = "./fetch.py"
network = "egress"
allow   = ["api.example.com:443", "*.cdn.example.com:443"]
secrets = ["API_KEY"]

[defaults] applies to every function; each [fn.NAME] table adds to it or overrides it. Chapter 20 lists every field.

What each function in the file does

  sandbox.toml
     │
     ├─ [fn.resize] ──▶ python:3.12-slim + libwebp7 layer + /venv ──▶ zygote: fork per request
     ├─ [fn.parse]  ──▶ alpine:3 + your Go binary, mounted ─────────▶ sandbox: exec per request
     └─ [fn.fetch]  ──▶ python:3.12-slim, network: only the allow list,
                        /run/secrets/API_KEY per request ───────────▶ zygote: fork per request

Chapter 13 covers the shapes, chapter 14 the network and secrets, and chapter 15 the venv and the apt layer.

Bringing the project up

zygo pull alpine:3                     # up never pulls; python:3.12-slim is already there
export API_KEY=…                       # read from this shell, delivered as a file
zygo up                                # every [fn.*] warm
zygo exec fetch '{"path": "/v1/ping"}'
zygo logs fetch -f                     # follow the function's log
zygo down                              # stop them all

zygo up warms every function in the file. It never pulls an image — a deploy should not quietly depend on a registry — so pull first, as above; zygo run is the one command that pulls for you (chapter 15). Run up again after an edit, and only the functions that changed are restarted.

Calling the functions over HTTP

zygo api puts an HTTP server in front of the functions. By default it listens on 127.0.0.1:7700, and every caller needs a bearer token: a secret string sent in the Authorization header.

export ZYGO_API_TOKEN=$(openssl rand -hex 16)
zygo api                               # 127.0.0.1:7700

curl -H "Authorization: Bearer $ZYGO_API_TOKEN" \
     -d '{"n": 4}' http://127.0.0.1:7700/fn/double

The token is needed on every kind of listener, unix sockets included. --no-auth turns it off, and Zygo refuses --no-auth anywhere except a unix socket or a loopback address such as 127.0.0.1.

  curl / your app ──HTTP──▶ zygo api ──▶ supervisor ──▶ double  (zygote)
      Authorization:        127.0.0.1:7700              resize  (zygote)
      Bearer <token>                                    parse   (warm sandbox)
                                                        fetch   (zygote)

What the API answers

Chapter 17 has every route, status code and header, and the Python, Node and Elixir clients.

Sending metrics to OpenTelemetry

Instead of waiting to be scraped at /metrics, Zygo can push the same numbers to an OpenTelemetry collector. OpenTelemetry (OTel) is a common standard for sending metrics and traces to monitoring tools.

zygo api --otlp-endpoint http://localhost:4318

OTEL_EXPORTER_OTLP_ENDPOINT does the same as the flag. The numbers go as OTLP/HTTP JSON once a minute, and OTEL_EXPORTER_OTLP_HEADERS adds a header, for example for authentication. Only metrics are sent. There are no per-request traces (spans) yet.

When something is wrong

zygo logs fetch --failed                    # the requests that failed, with their stderr
zygo shell fetch                            # a shell inside the warm sandbox
zygo spec explain fetch                     # every limit and mount, fully resolved
zygo run --dry-run --json python:3.12-slim  # the mount plan, without running it
zygo doctor                                 # is the host still able to run sandboxes?

Chapter 22 lists the errors people actually meet, with the fix for each.

Where to go next

The examples/ folder has complete projects: a small web API, a webhook, a CI job, an LLM tool, a Go program and a shell script as warm-exec functions, a plugin host in Python and in Node, a workflow-engine worker, an n8n Code-node runner, a Kubernetes deployment, and an agent in POSIX sh. Its README says what each one shows.

12. One-shot sandboxes: zygo run

zygo run builds a fresh sandbox, runs one program in it, and removes the sandbox when the program ends. It reads like docker run, but it has safe defaults and leaves nothing behind. This chapter covers everything about it: what happens inside, how to give the sandbox your files, and how to tell why it ended.

The shape of the command

zygo run [FLAGS] IMAGE [COMMAND ARGS...]
zygo run python:3.12-slim python3 -c 'print("hello")'

The image is a normal container image from any registry, such as Docker Hub. Everything after the image is the command, which runs inside the sandbox. With no command, the image’s own entrypoint runs, just as with Docker. Put Zygo’s flags before the image; anything after it belongs to the command.

What happens, step by step

  zygo run --mem 128M python:3.12-slim python3 app.py
     │
     ├─ 1. plan    read the flags and any sandbox.toml, check the limits,
     │             find the image in the store (pull it if missing)
     │
     ├─ 2. place   supervisor running? ── yes ─▶ hand the sandbox to it
     │                    │ no
     │                    └─▶ not in a delegated cgroup? re-exec in a systemd scope
     │
     ├─ 3. start   clone3 into new namespaces ─▶ the child builds its root,
     │             joins its cgroup, drops capabilities, adds Landlock + seccomp,
     │             and calls execve on your command
     │
     ├─ 4. run     your program runs; streams, signals and exit code pass through
     │
     └─ 5. end     the kernel removes the namespaces and the tmpfs,
                   Zygo removes the cgroup ─▶ nothing is left

These are the three phases that --outcome reports: plan (step 1), start (steps 2 and 3), and run (steps 4 and 5). Namespaces, cgroups, capabilities, Landlock and seccomp are explained in chapters 2, 3 and 4, and chapter 6 shows how Zygo puts them together.

What the sandbox has

Every sandbox starts closed. You open what you need, one flag at a time.

What it does not have: your files, your network, your other processes, or any way to reach the image store it was built from.

The first run and the second

The first run of an image pulls it, just as docker run does. After that, the image is in Zygo’s store under your home folder, and a run takes about 12 ms (median, image already pulled; the hosts are in chapter 25). The first run on a kernel older than 5.11 is also slower, because Zygo has to flatten the image’s layers once. Pull policy below explains how to control pulling.

Running your own code: mounts

The sandbox cannot see your filesystem. So a script of yours has to be mounted in first. A mount makes a file or folder from your machine appear at a path inside the sandbox. The interpreter, by contrast, comes from the image, not from your machine — chapter 6 explains why zygo ./venv/bin/python app.py is not a thing.

zygo run --mount ./hello.py:/hello.py:ro python:3.12-slim python3 /hello.py
zygo run --mount ./src:/src:ro python:3.12-slim python3 /src/main.py --flag
printf '{"n": 21}' | zygo run --mount ./src:/src:ro python:3.12-slim python3 /src/double.py

The form is --mount HOST:GUEST[:ro|rw]. HOST is the path on your machine, GUEST is the path inside the sandbox. A mount is read-only unless you add :rw. You can mount a single file or a whole folder, and you can repeat --mount as often as you need.

  your machine                          the sandbox
  ./src/main.py     ── --mount ─────▶   /src/main.py    (read-only)
  ./pkgs/           ── --mount :rw ─▶   /pkgs/          (writable, changes are real)
  everything else   ── not visible ──   (the sandbox cannot name it)

Programs from your host

You can run a program from your own machine the same way: mount it, then name it. It runs against the image’s libraries, not your host’s. So a program that is dynamically linked (it loads shared libraries such as libc when it starts) needs an image with a compatible libc. A static program carries its libraries inside it and runs anywhere.

zygo run alpine:3 pwd                                             # the image's own pwd: /
zygo run --workdir /tmp alpine:3 pwd                              # /tmp
zygo run --mount /bin/pwd:/opt/pwd:ro python:3.12-slim /opt/pwd   # your host's pwd

A glibc /bin/pwd from Ubuntu runs in python:3.12-slim, which also uses glibc. The same file fails in alpine:3 with “the program does not exist”, because Alpine uses musl and the loader the program names is not there.

Working folder, user and environment

zygo run --env GREETING=hi --workdir /tmp alpine:3 sh -c 'echo $GREETING from $(pwd)'

Do not pass secrets with --env. Environment variables are easy to leak, for example into logs or child processes. Warm functions have a safer way, described in chapter 14.

Limits on the command line

zygo run --mem 128M --pids 16 --timeout 10s alpine:3 /bin/sh
zygo run --cpu 0.5 --scratch 32M --nofile 256 python:3.12-slim python3 app.py

A limit cannot be switched off unless you also pass --allow-unlimited. Chapter 14 explains each limit.

Isolation, seccomp and network flags

zygo run --net egress --allow api.example.com:443 python:3.12-slim python3 /src/call.py

Each flag that makes the sandbox weaker has a name that says so. You cannot remove a protection by accident.

Input, output and signals

Standard input, standard output, standard error and the exit code pass straight through, so zygo run fits in a shell pipe. Zygo’s own messages, such as pulling python:3.12-slim, go to standard error. -q or --quiet hides them, which helps when a program or an AI model reads the output. It never hides what the program itself writes, and errors are still reported.

Ctrl-C works in two steps. The first Ctrl-C (or SIGTERM, or SIGHUP) is passed on to the program and every process it started, so a Python program gets its KeyboardInterrupt. The second signal of any kind sends SIGKILL to the sandbox’s first process, which ends the whole sandbox.

  Ctrl-C #1 ──▶ SIGINT to the program and its process group   ("please stop")
  Ctrl-C #2 ──▶ SIGKILL to the sandbox's pid 1 ─▶ whole sandbox gone   ("stop now")

A signal that the shell told zygo to ignore, for example under nohup, is not passed on.

A terminal of its own: --tty

Zygo has no daemon, so by default the sandbox inherits your terminal. That is why zygo run alpine:3 sh feels like a local command: colours, prompts and job control just work. The cost is that code in the sandbox holds a writable handle to your real terminal.

-t or --tty gives the sandbox a pseudo-terminal of its own instead: a fake terminal that Zygo creates. Zygo keeps the other end and copies bytes between it and your real terminal, so the sandbox never touches the real one.

  default:   your terminal ◀──────────────────────────▶ sandbox (holds your terminal)
  --tty:     your terminal ◀──▶ zygo ◀──▶ new pty ◀──▶ sandbox (never sees yours)

zygo run --tty alpine:3 /bin/sh

Why is --tty not the default? Because the default is already safe, and it is what makes a sandbox feel like a local command. The dangerous use of that handle, pushing keystrokes into your terminal with TIOCSTI, is blocked by the seccomp filter either way. --tty goes one step further and removes the handle completely. A run with --tty always stays in your shell and is never handed to a supervisor.

Coming from Docker

The sentence is the same as docker run, the defaults are the opposite, and no container object is left behind for you to remove. Chapter 5 and chapter 10 compare the two flag by flag.

One trap: in Zygo, -v means verbose, not volume. If you type zygo run -v $PWD:/src image in the folder /app, Zygo sees that the “image” looks like a mount and says so:

`/app:/src` is a mount, not an image
  → `-v` means verbose in Zygo, not volume; use: zygo run --mount /app:/src <image> …

Look before you run: --dry-run

zygo run --dry-run --json python:3.12-slim

--dry-run prints the plan and runs nothing. It shows:

• the resolved settings, after defaults, sandbox.toml and flags are merged;
• the mount plan: every path the sandbox will see, and where it comes from;
• the seccomp profile, where it came from (the flag, the spec or the default), and how many system calls it allows;
• whether Landlock applies on this host;
• the cgroup values that will be written, and how the deadline is enforced.

This is how you review a sandbox’s walls before you trust them. --json makes the plan machine-readable, so two plans can be compared with diff.

Exit codes

The exit code is the program’s own, with a few exceptions that belong to Zygo.

A program can also return 1, 2 or 125 itself. When the difference matters, use --outcome.

Knowing why a sandbox ended: --outcome

A deadline kill and an out-of-memory kill both exit 137. Both are a SIGKILL, and the exit status carries nothing more. When the difference matters, for example in an online judge or a CI step, ask for an outcome file:

zygo run --outcome /tmp/why.json --mem 64M --timeout 5s python:3.12-slim python3 big.py
cat /tmp/why.json

{"exit_code":137,"timed_out":false,"oom_killed":true,"peak_rss_kb":65780,
 "wall_ms":412.7,"plan_ms":3.1,"start_ms":9.8,"started":true,"phase":"run"}

The three times in this example are only an illustration; they depend on the host. The report goes to a file because standard output belongs to the program. Zygo writes the file whole and then renames it into place, so a reader never sees half a file.

The outcome fields

Neither timed_out nor oom_killed is a guess. The three times tell you which part was slow when one run takes much longer than usual.

When the sandbox never started

The outcome file is also written when the sandbox never started: the image is not there, the host cannot run sandboxes, or a mount does not exist. Then it says "started": false and names the phase that failed. A program that ran has "started": true and "phase": "run".

  "started": false, "phase": "plan"   ─▶ not set up (missing image, bad spec)   ─▶ unavailable
  "started": false, "phase": "start"  ─▶ the host could not build the sandbox   ─▶ unavailable
  "started": true,  "phase": "run"    ─▶ the code ran; exit_code is its own     ─▶ its own result

This is the difference between the service is unavailable and the code failed. The exit status cannot carry it, and a caller should not have to read standard error to find it.

Pull policy

--pull says when zygo run downloads the image.

never is for a caller that keeps its own clock, such as a judge with a time limit. A pull takes minutes and a run takes seconds. Without never, a run that quietly became a pull looks like a program that was too slow. With it, the answer comes in a millisecond. Pull images ahead of time with zygo pull IMAGE.

Python packages: --requirements

zygo run --requirements ./requirements.txt --mount ./src:/src:ro \
    python:3.12-slim python3 /src/main.py

--requirements FILE installs the packages in a Python venv (a folder of installed packages) and mounts it at /venv, with /venv/bin first on PATH. The venv is built once and cached. The cache key is the image’s digest plus the bytes of the file, so the next run with the same image and the same file reuses it. It is the same cache zygo serve uses, so a one-shot run and a warm function with the same requirements share one venv. Chapter 15 explains the cache.

Installing packages into a mounted folder

Many products need an “install this customer’s packages” feature. The shape is always the same: one networked one-shot installs into a mount, and then any number of sandboxes without network import from it.

mkdir -p ./pkgs
zygo run --net full --mount ./pkgs:/pkgs:rw python:3.12-slim \
    python3 -m pip install --target /pkgs python-dateutil
zygo run --mount ./pkgs:/pkgs:ro --env PYTHONPATH=/pkgs python:3.12-slim \
    python3 -c 'import dateutil, six; print(dateutil.__version__)'

  step 1 (once):   network on  ──▶ pip install ──▶ ./pkgs  (mounted rw)
  step 2 (often):  network off ──▶ import from ──▶ ./pkgs  (mounted ro)

What the package install costs

On the 2-core VM of an adoption test (a real multi-tenant product moved onto Zygo), python-dateutil and six installed in 2.8 s through Zygo, against 3.9 s through Docker.

This used to fail with [Errno 1] Operation not permitted on a RECORD file. pip install --target copies each file with Python’s shutil.copy2, which calls listxattr (a call that lists a file’s extended attributes), and the seccomp profiles did not allow it. The workaround was to point TMPDIR at the same mount. Every profile now allows the whole extended-attribute family, so neither the error nor the workaround remains, and make verify-seccomp-profiles-linux checks that it stays that way.

Using a sandbox.toml: [defaults] and -f

zygo run does not need a spec file; the flags alone are enough. But if a sandbox.toml exists, zygo run uses it. It looks in the current folder and then in each parent folder, or reads the file you name with -f.

  built-in defaults ──▶ the spec's [defaults] ──▶ your flags   (later wins)
                        ([fn.*] tables are ignored by zygo run)

zygo run -f ./ci/sandbox.toml python:3.12-slim python3 -m pytest

So a team can write its limits once in [defaults], and every zygo run in that project uses them. The [fn.*] tables describe warm functions and are ignored here. Chapter 20 describes the file.

Handing the run to the supervisor

On Linux, when a supervisor is already running (for example after zygo serve), zygo run gives the sandbox to it. The supervisor already sits in a delegated cgroup, so the run skips the systemd scope described below. Planning and pulling still happen in your process; the supervisor only builds and runs the sandbox, and your process passes its streams and signals and waits.

The hand-off does not happen with --dry-run (nothing runs), with --tty (the terminal must stay in your shell’s session), or with an isolation other than ns. zygo run -v tells you which path was taken: its timing line ends in (through the supervisor) when the supervisor ran it.

The systemd scope

On a systemd machine, a login shell lives in a cgroup that you are not allowed to split up. A sandbox needs a cgroup of its own for its limits, and Zygo refuses to run a sandbox with no limits. So zygo run, pull, serve, up and bench first check where they stand. If they are not in a delegated cgroup, they start again inside a temporary one:

  $ zygo run ...                         (login shell cgroup: not delegated)
      └─▶ systemd-run --user --scope -p Delegate=yes zygo run ...
             └─▶ zygo run ...            (own scope: delegated, can make cgroups)
                    └─▶ sandbox

This costs about 10 to 15 ms: the scope, a second process, and a cgroup tree built and removed. In a container, a systemd service, or a session that is already delegated, nothing is done. A guard stops Zygo from restarting itself forever: if the new scope is still not usable, you get Zygo’s own error.

Output for scripts

Every zygo run flag in one table

Chapter 19 lists every other command.

13. Warm functions

A warm function is a sandbox that Zygo builds once and keeps ready, so that each request costs a fork() instead of a new sandbox. This chapter shows how to serve one, call it, watch it and stop it, and what your code must look like in Python, Node, TypeScript or any other language. Chapter 6 explains the idea behind it; this chapter is about using it.

Warm functions in one picture

A handler is the function you write: it gets one request and returns one answer. zygo serve builds a sandbox, starts a small helper program inside it called the agent, and the agent loads your handler. That loaded, waiting process is the zygote. zygo exec sends a request; the zygote makes a copy of itself with fork(), and the copy runs your handler once and exits.

  zygo serve ./handler.py --name resize        (once, about 150 ms)
        │
        ▼
  ┌────────────────────── warm sandbox "resize" ───────────────────────┐
  │                                                                    │
  │   zygote: python started, imports done, handler loaded             │
  │      │                                                             │
  │      ├── fork ─▶ child: handler(event 1) ─▶ answer ─▶ exit         │
  │      ├── fork ─▶ child: handler(event 2) ─▶ answer ─▶ exit         │
  │      └── fork ─▶ child: handler(event 3) ─▶ answer ─▶ exit         │
  │                                                                    │
  └────────────────────────────────────────────────────────────────────┘
        ▲
        │
  zygo exec resize '{"url": "…"}'             (each time, about 1.4 ms)

Why this is the production shape

A one-shot sandbox (zygo run) costs about 12 ms with Python in it: about 3.6 ms is the sandbox itself, the rest is Python starting. A warm function pays the setup once and then costs about 1.4 ms a request: that is the median overhead measured on a 2-vCPU Lima VM on an Apple M1 Max (chapter 25). The gap is easy to miss. zygo run looks like the natural way to say “run this code once”, so a program that uses Zygo that way gets one sandbox per event. On that VM, zygo bench cold says 12.3 ms for that, while zygo bench warm says 1.44 ms and 1,108 requests a second — more than eight times faster, and the gap grows with every module the handler imports.

A multi-tenant application that starts with run for exactly that reason should expect to move to warm functions once it counts the events. The multi-tenant example below shows what that looks like.

  one sandbox per event (zygo run)          one warm function (zygo serve + exec)
  ────────────────────────────────          ──────────────────────────────────────
  event ─▶ build sandbox ─▶ run ─▶ clean    serve: build sandbox + load code, once
  event ─▶ build sandbox ─▶ run ─▶ clean    event ─▶ fork ─▶ run ─▶ exit
  event ─▶ build sandbox ─▶ run ─▶ clean    event ─▶ fork ─▶ run ─▶ exit
  12.3 ms each (2-vCPU Lima VM)             1.44 ms each (same VM)

Serving a function

zygo serve takes a handler file and a name. It starts the supervisor (the background process that owns every warm sandbox) if one is not running yet, warms the sandbox, and returns. From then on the function is ready.

zygo serve ./handler.py --name resize
zygo serve ./handler.py --name resize --requirements requirements.txt --mem 512M
zygo serve ./report.py  --name report --secret STRIPE_KEY --idle-timeout 5m

Every limit and sandbox flag of zygo run works here too: --mem, --cpu, --pids, --timeout, --net, --allow, --mount and the rest. They are described in chapter 14. The same settings can live in a [fn.<name>] table of sandbox.toml (chapter 20).

Calling a function

zygo exec NAME EVENT sends one request. The event is any JSON value; if you leave it out, zygo exec reads it from standard input. The handler’s answer goes to standard output. Whatever the handler itself printed goes to standard error, so you can pipe the answer into another program and still see the logs.

zygo exec resize '{"url": "https://example.com/a.png"}'
echo '{"url": "https://example.com/b.png"}' | zygo exec resize
zygo exec resize --timeout 5s '{"url": "https://example.com/c.png"}' > out.json

--timeout gives up on this request after that long; without it, the function’s own timeout applies. The exit code of zygo exec tells a script what happened:

Many events at once: --batch

--batch reads NDJSON from standard input: one JSON event per line. It sends them to the function in parallel and prints one JSON answer per line, in the same order as the input. The exit code is 0 only if every request succeeded.

zygo exec resize --batch < events.ndjson > answers.ndjson

  events.ndjson           the function (concurrency 4)          answers.ndjson
  ─────────────           ────────────────────────────          ──────────────
  line 1 ──────────────▶  fork ─▶ answer 1 ─────────────────▶  line 1
  line 2 ──────────────▶  fork ─▶ answer 2 ─────────────────▶  line 2
  line 3 ──────────────▶  fork ─▶ answer 3 ─────────────────▶  line 3
  ...                     (they run side by side,               (always in
                           and may finish in any order)          input order)

When it is full: concurrency and the queue

Each zygote runs up to concurrency requests at once (default 4). More requests wait in a queue that holds up to four times concurrency. When the queue is full too, a new request is turned away at once as busy: HTTP 429 from the API, exit code 75 from zygo exec. Turning work away quickly is better than letting every caller wait for a timeout.

                          concurrency = 4
  new request ─▶ ┌───────────────────────────────┐
                 │ running: [1] [2] [3] [4]      │  full? ─▶ wait in the queue
                 └───────────────────────────────┘
                 ┌───────────────────────────────┐
                 │ queue: up to 16 (4 × 4)       │  full? ─▶ busy: HTTP 429, exit 75
                 └───────────────────────────────┘

Looking after warm functions

These commands work on functions that are already served. Chapter 19 has every flag.

zygo shell needs one warning. It starts a new process and enters the sandbox’s namespaces (the kernel’s walls around files, processes, network and host name), so it sees what the handler sees. It holds no capabilities. But it is not under the seccomp filter, the Landlock rules or the function’s cgroup, so that a debugging shell is not killed by the memory limit. The warm zygote is not touched and keeps serving.

Warm, paused, cold

A warm function does not stay in memory forever. After idle_timeout (default 10 minutes) with no requests, Zygo pauses it: the cgroup is frozen, so its processes stop using the CPU but stay in memory. The next request wakes it with one write, in far less time than a warm-up. After cold_after (default 1 hour) the sandbox is dropped. The function still exists by name, and the next request pays a full warm-up again: about 150 ms for a Python handler with no imports, measured on a Raspberry Pi 5.

                zygo serve               no request for idle_timeout (10m)
  (nothing) ─────────────▶ ┌────────┐ ──────────────────────────────▶ ┌──────────┐
                           │  warm  │                                 │  paused  │
                           │        │ ◀────────────────────────────── │          │
                           └────────┘   a request wakes it (1 write)  └──────────┘
                               ▲                                           │
                               │ a request warms it again                  │ no request for
                               │ (about 150 ms for Python)                 │ cold_after (1h)
                           ┌───┴────┐                                      │
                           │  cold  │ ◀────────────────────────────────────┘
                           └────────┘   sandbox dropped, name kept

Which shape to use

There are three ways to give Zygo code to keep warm. Pick by two questions: is the code the same on every request, and does its runtime start slowly?

                         is your code the same on every request?
                              │
               yes ───────────┴──────────── no: each request brings a script
                │                                        │
      does its runtime start slowly?              RUNTIME POOL
      (Python, Node with many modules)            [runtime.<name>], --runtime
                │                                 one warm interpreter,
        yes ────┴──── no (Go, Rust, C, sh)        thousands of scripts
         │                    │
   AGENT FUNCTION        WARM-EXEC FUNCTION
   entry = "h.py"        cmd = ["/app/bin"]
   fork per request      new process per request
   ~1.4 ms               ~1.4 ms + your program's start

A Python handler

Write a module with a function called handler at the top level. It gets the event — whatever JSON the caller sent, already parsed — and returns something that can be turned into JSON. It may be a normal function or an async one; Zygo runs an awaitable to the end. There is no second “context” argument. Everything at module level runs once, in the zygote, before any request: put your imports, your model loading and your compiled regular expressions there.

import json, re                       # runs once, when the zygote warms
PATTERN = re.compile(r"\d+")          # also once

def handler(event):                   # runs in a fresh fork, per request
    numbers = PATTERN.findall(event["text"])
    return {"count": len(numbers)}

A larger one, which makes image thumbnails:

from PIL import Image          # imported once, in the zygote
import io, base64

MAX = (800, 800)               # module-level state is shared, copy-on-write

def handler(event: dict) -> dict:
    """Runs in a fresh fork per request. Writing to globals is safe, but the
    next request will not see it."""
    img = Image.open(io.BytesIO(base64.b64decode(event["image"])))
    img.thumbnail(MAX)
    out = io.BytesIO(); img.save(out, "WEBP")
    return {"image": base64.b64encode(out.getvalue()).decode(), "size": img.size}

What “warm” means, exactly

The sandbox is built once, and the agent inside it imports your handler. Each request is a fork() of that agent. A fork shares the parent’s memory copy-on-write: the pages are shared until one side writes to one, and only that page is copied. So nothing is copied up front and nothing is imported again. And because the child is a separate process, nothing a request writes is visible to the next one. You get the speed of a shared interpreter and the isolation of a fresh one.

That is the trade Zygo exists to make. A long-running worker is fast but leaks state from one request to the next. A container per request is clean but costs hundreds of milliseconds. A fork is both fast and clean.

What mem bounds in a warm function

mem is one request’s limit. Each request runs in a cgroup of its own with memory.max = mem, and the warm process (the agent, and for Node the workers it keeps loaded) sits in a leaf of its own with the same limit (chapter 3). A request that allocates past mem is killed by the kernel, together with everything it started and nothing else: the requests running beside it finish, and the zygote is still warm for the next one. So a function with concurrency = 4 and mem = "256M" may use up to five times mem at once, one share for the warm process and one per request; the tenant’s budget above bounds the sum. make verify-oom-linux runs three sleeping requests beside one that allocates 2 GB, for both agents, and checks that only the one dies.

  long-running worker          container per request         fork per request (Zygo)
  ───────────────────          ─────────────────────         ───────────────────────
  fast                         clean                         fast AND clean
  request 2 sees what          hundreds of ms each           each child starts as a copy
  request 1 left behind                                      of the zygote, then is gone

A fork carries three things that a fresh process would not. make fork-sweep-linux measured them across 44 popular PyPI packages. The next three sections say what Zygo does about each.

When a handler is not safe to fork

A fork copies only the thread that called it. If another thread held a lock at that moment, the lock stays locked in every child, and the child can hang. So a process that started threads cannot be forked safely. At warm-up the Python agent checks for threads that would survive a fork — Python threads and native ones, such as the four that import duckdb starts. If it finds any, it falls back to spawning a fresh interpreter per request instead of forking. That is correct, but much slower, and zygo logs says when it happened. Thread pools that stop themselves before a fork, as OpenBLAS’s does, do not trigger it. The fix is to start threads inside the handler, or lazily on first use.

Random numbers in a fork

A child starts with a copy of its parent’s random-number state, so without care every request would draw the same “random” numbers. The agent reseeds Python’s random, numpy’s global generator and torch’s generator in every child. It cannot reseed a generator that you created at import time, such as RNG = random.Random() or np.random.default_rng(): every request draws the same numbers from it. The agent names such a generator in zygo logs at warm-up. Create it inside the handler instead.

Work done lazily on first use

Every request is the first call in its own process. So a library that sets itself up on first use pays that cost on every request, and none of it is kept. Do that work at import time instead, where the zygote pays it once: build the boto3 client, compile the template, and create the pydantic model at module level.

A Node handler

Export a function: module.exports = function handler(event) {…} or module.exports.handler = …. It may return a value or a promise; undefined becomes null. One difference to know: Node is not safe to fork, so the Node agent keeps two pre-loaded worker processes ready. Each worker runs one request and exits, and a new one is started off the request path. The effect — a clean process per request — is the same, but the cost per request is a little higher than Python’s fork.

const crypto = require("crypto");          // loaded once, in each pre-loaded worker

module.exports = async function handler(event) {
  return { sha256: crypto.createHash("sha256").update(event.text).digest("hex") };
};

  Node agent (no fork)
  ┌────────────────────────────────────────────────────────────────┐
  │  agent ── keeps 2 workers loaded and parked                    │
  │    worker A: handler loaded ─▶ request 1 ─▶ exit               │
  │    worker B: handler loaded ─▶ request 2 ─▶ exit               │
  │    worker C: started in the background, ready for request 3    │
  └────────────────────────────────────────────────────────────────┘

TypeScript handlers

A .ts file is loaded as TypeScript with its types removed, with no build step and no bundler. The types are stripped as the module loads, in the worker, which is the same place a .js file is compiled. enum, namespaces and parameter properties work too. This needs Node 22.13 or later, or the amaro package in the function’s dependencies. Chapter 18 has the details.

[fn.resize]
image = "node:22-slim"
entry = "./resize.ts"     # runtime = "node", inferred from the extension

What a request sees

What a request returns

A return value must turn into JSON; in Python, bytes become base64. Output printed to stdout and stderr is not the result. Each is collected separately (the last 256 KiB of each) and returned beside it, and sent live, in pieces, when the caller asks for a stream. If the handler raises an exception, the request fails with exit code 1, and its error is the traceback without the agent’s own lines. event.progress("step 2 of 5") sends a progress line to a caller that is streaming, and does nothing otherwise.

                   ┌──────────── what comes back ────────────┐
  handler(event) ─▶│ result     the return value, as JSON    │
                   │ stdout     what it printed (≤ 256 KiB)  │
  print(...)    ──▶│ stderr     what it logged  (≤ 256 KiB)  │
                   │ exit_code  0, or 1 if it raised         │
                   │ error      the traceback, if it raised  │
                   └─────────────────────────────────────────┘

mode = "stdin"

This is for a Python handler written as a script rather than as a function. The file is run once per request with the event as JSON on standard input, and whatever it prints to standard output is parsed as the result. A non-zero exit is an error. Use it for existing scripts you do not want to change. It is slower than a handler: each request starts a new Python process for the script, inside the warm sandbox, so the script’s imports are paid every time. Because it starts a program, it cannot run under seccomp = "strict", which forbids that.

[fn.legacy]
entry = "./old_script.py"
mode  = "stdin"

Warm-exec functions

For a program that starts fast, set cmd and no entry. The sandbox is kept warm, and each request starts cmd as a new process inside it. The contract is the simplest one possible: read one JSON event from stdin, write one JSON result to stdout, exit 0. Stderr is kept as the log, and a non-zero exit is a failure. sh -c cat is the smallest program that obeys it. It works with any language and any image, and needs no agent. It costs a median of 1.4 ms per request, measured on a Lima VM, plus your program’s own start.

[fn.parse]
image  = "alpine:3"
mounts = ["./bin/parse:/app/parse:ro"]   # a static binary you built
cmd    = ["/app/parse"]

Runtime pools

A pool is a warm interpreter with no code of anyone’s in it. Each request carries a script — as source, or as the sha256: digest (a fingerprint of the file’s bytes) of a script stored earlier with PUT /scripts. The forked child loads it, runs its handler (or the function named by entry_point), and exits. The script is loaded after the child’s seccomp filter is on, so even its import-time code is filtered, and a pool’s seccomp profile defaults to strict. strict has no socket, so a pool whose scripts call out names seccomp = "default" beside its network; asking for a network under strict is refused rather than left silently unusable. This is how a platform with ten thousand user scripts keeps a handful of zygotes warm instead of ten thousand.

zygo serve --runtime py312 --image python:3.12-slim --agent python
zygo serve --runtime node22 --image node:22-slim --agent node --min-warm 2 --max-warm 8
zygo exec --runtime py312 --script report.py '{"month": "2026-09"}'
zygo exec --runtime py312 --script sha256:9f2c… --entry-point monthly '{}'

  one pool, many tenants' scripts
  ┌────────── pool py312 (min_warm 2, max_warm 8) ──────────────┐
  │  zygote  python + requirements, NO user code                │
  │    ├── fork ─▶ child loads script A (tenant acme) ─▶ exit   │
  │    ├── fork ─▶ child loads script B (tenant beta) ─▶ exit   │
  │    └── fork ─▶ child loads script A again         ─▶ exit   │
  └─────────────────────────────────────────────────────────────┘
      +0.5 ms per request, against a function with its code warmed in

A pool keeps min_warm zygotes ready whatever the load (default 1; 0 counts as 1) and may grow to max_warm under load (default the larger of min_warm and 4). --agent is python, node, or the path of your own agent inside the sandbox. The +0.5 ms was measured with a different script on every request, a thousand of them: +0.47 ms on the Lima VM and +0.46 ms on Docker Desktop’s (chapter 25). A pool with a cmd instead of an agent is a warm-exec pool: the script’s path is passed as the last argument of cmd on each request.

No listener in a pool. A pool’s requests belong to different tenants and share one network namespace, so a script that opened a TCP port on loopback would be offering a channel to every other request in the pool. Two layers refuse it: strict, the pool default, removes the socket calls, and under it Landlock refuses bind in a shared namespace (kernel 6.7+). A function has no such rule: its namespace is one tenant’s, and it may listen on its own loopback (chapter 14).

Secrets in a pool. A pool can name secrets (secrets = ["STRIPE_KEY"] in [runtime.<name>], or serve_runtime(..., secrets=[...])), but it holds no values: the zygotes are shared. On each request the calling tenant’s values are read from the tenant secret store and written as /run/secrets/<NAME> for that one request, exactly as for a function — and while they exist, the request has its zygote to itself, so no other tenant’s child is forked beside the files. A tenant that lacks one of the names is refused before anything runs. Chapter 14 has the rules; zygo stop <name> stops a pool as it does a function.

A multi-tenant consumer on the warm path

Think of a low-code platform, a workflow engine or a plugin host. It has hundreds of scripts written by its users, the tenants, and a script changes whenever somebody presses Save. Each project has its own mounts and its own egress allowlist (the list of hosts it may reach), and each run has its own secrets. The one-shot mapping is one sandbox per event. The warm mapping is one warm zygote per script version, forked per run, and it looks like this:

import hashlib
import zygo_sdk as zygo

client = zygo.connect()                      # zygo api --allow-deploy, in the VM on a Mac

def run(project, script_source, event, secrets):
    # A version is a function. The name carries the digest, so an edit is a
    # new function and the old one is reaped by --idle-timeout, not by you.
    digest = hashlib.sha256(script_source.encode()).hexdigest()[:16]
    name = f"{project.id}-{digest}"
    client.serve(
        name,
        {
            "entry": project.script_path(digest),          # the version, on disk
            "mounts": [f"{project.data_dir}:/data:rw"],    # per project
            "network": "egress",
            "allow": project.allowlist,                    # per project
            "secrets": list(secrets),                      # names; values per run
            "idle_timeout": "10m",                         # reaped when idle
            "cold_after": "1h",
            "mem": "256M", "timeout": "30s",
        },
        if_changed=True,                                   # a no-op when it is warm
    )
    return client.fn(name)(event)                          # one fork

  project acme, script v1 ─▶ function "acme-3f9a…" ─▶ warm zygote ─▶ fork per run
  project acme, script v2 ─▶ function "acme-c21e…" ─▶ warm zygote ─▶ fork per run
                             (v1 is no longer called: paused after 10m, dropped after 1h)
  project beta, script v1 ─▶ function "beta-77d0…" ─▶ warm zygote ─▶ fork per run

What each line buys

• entry and the mounts are paths on the host Zygo runs on. A function’s code is named by path, never sent, so this shape is for a worker that shares a disk with Zygo. A caller on another machine sends each script with its request instead, to a runtime pool (chapter 17).
• if_changed=True makes the serve free when the version is already warm. The caller does not have to track state: it always asks, and the supervisor does nothing when nothing changed.
• Mounts and the allowlist are per function, so two projects’ versions are two zygotes that share an interpreter’s memory pages and nothing else.
• Secrets are per run. The names are declared once; the values arrive with the request and exist as files only while the request runs.
• idle_timeout and cold_after are the eviction policy. A version nobody has called for ten minutes is paused (still in memory, one write to wake). After an hour it is dropped, and the next call pays a warm-up — about 150 ms for a Python handler, plus its imports.

Four hundred projects do not mean four hundred warm zygotes. They mean as many as were called in the last ten minutes, which is the number that matters, and zygo ps shows it. ADR 0005 has the memory per warm script, measured on a 4 GB VM, and the eviction policy written down. examples/workflow-engine/ is this shape end to end — a worker draining a job queue, one warm function per script version, and an LRU (least recently used) list over warm scripts — in Python and Node.

Other languages

There are two ways, and you pick by whether starting your runtime is slow enough to be worth doing only once.

Warm-exec is the simple one: give a function a cmd and no runtime. Each request is a fresh process in the held sandbox, with the event on standard input and JSON expected on standard output. That costs about 1.4 ms and needs no code from you beyond the program. Go, Rust, C and bash all belong here; examples/warm-exec/ has Go and shell examples.

An agent is for a runtime that is slow to start. It warms once and gives each request a process of its own. Two agents ship with Zygo, and the runtime is chosen from the entry file’s extension:

[fn.resize]
entry = "./resize.py"      # runtime = "python": a fork of the warmed interpreter
[fn.summarise]
entry = "./summarise.js"   # runtime = "node": a pool of pre-loaded workers
[fn.custom]
entry   = "./handler.rb"
runtime = { agent = "/app/zygo-agent" }

Node has no fork() in the Unix sense, so its agent keeps workers loaded and parked instead — one request each, replaced off the request path. Everything above that is the same: the same wire protocol, the same per-request cgroup, the same deadline, the same secrets. Deno and Bun have no agent and are not waiting for one (ADR 0003); they start in a few milliseconds, so a warm-exec pool serves them.

Your own agent

Any language can have a real warm path by shipping an agent: a program that talks to the supervisor over a socket, using a small documented protocol (spec/protocol.md). Messages are length-prefixed JSON: READY when warm, EXEC per request, FORKED and GO around the fork, CHUNK for streamed output, RESULT, DONE, CANCEL, PING/PONG and SHUTDOWN. Name it with runtime = { agent = "/path/in/sandbox" }. zygo agent test runs the conformance suite against it before you trust it:

zygo agent test /bin/sh -- examples/agents/sh/agent.sh examples/agents/sh/handler.sh

examples/agents/ has a complete agent in POSIX sh of about 130 lines that passes the same checks the shipped ones do. Chapter 18 is the full guide.

Handler rules, in short

1. Do slow things at module level; they run once.
2. Do not start threads at import.
3. Create random generators inside the handler, not at import.
4. Return JSON; print only for logs.
5. Read secrets from /run/secrets/, never from the environment.
6. Write only to /tmp, the workspace or a :rw mount; the rest is read-only.
7. Assume nothing survives the request — because nothing does.

14. Limits, networking and secrets

A sandbox is only as good as the walls you give it. This chapter shows how you set those walls in sandbox.toml: how much a function may use, what it may reach on the network, and how a password or API key gets to it without leaking. Every wall has a safe value even if you set nothing.

The spec file in one minute

A spec file is sandbox.toml: one file that describes every function in a project. [defaults] holds values for all functions, and each [fn.<name>] section describes one function and may change any of them. Chapter 20 lists every field; this chapter explains the ones that set limits, the network and secrets.

[defaults]
image     = "python:3.12-slim"
isolation = "ns"          # ns | gvisor | vm
mem       = "256M"
cpu       = 1.0
pids      = 64
timeout   = "30s"
network   = "none"

[fn.resize]
entry        = "./resize.py"       # defines handler(event)
requirements = "./requirements.txt"
system       = ["libwebp7"]        # apt packages, installed once as a layer
mem          = "512M"
mounts       = ["./cache:/cache:rw"]

[fn.parse]
image  = "alpine:3"                # no runtime → warm-exec
mounts = ["./bin/parse:/app/parse:ro"]
cmd    = ["/app/parse"]

[fn.fetch]
entry       = "./fetch.py"
network     = "egress"             # nothing else is reachable
allow       = ["api.stripe.com:443", "*.example.com:443", "203.0.113.0/24:5432"]
connections = 32
bandwidth   = "2M"
secrets     = ["STRIPE_KEY"]

Which value wins

A value can come from four places. The highest one that sets a field wins.

  highest ┌───────────────────────────────┐
          │ a CLI flag (or an API call)   │   --mem 512M
          ├───────────────────────────────┤
          │ [fn.<name>]                   │   mem = "512M"
          ├───────────────────────────────┤
          │ [defaults]                    │   mem = "256M"
          ├───────────────────────────────┤
  lowest  │ Zygo's built-in default       │   256M
          └───────────────────────────────┘

Lists and tables — allow, mounts, env, secrets, system — replace the one below them. They are never added together. So an allow list in [fn.fetch] is the whole list for that function, and the one in [defaults] no longer counts. Two commands show you the result:

zygo spec explain resize     # every field of `resize`, after the merge
zygo spec validate           # check the whole file, run nothing

Every limit has a value

A limit is a ceiling the kernel enforces, most of them through the function’s cgroup (chapter 3). Every limit has a default, and you cannot turn one off by leaving it out. The only way to remove one is a flag named --allow-unlimited. This is on purpose: a sandbox with no memory limit is not a sandbox, and people usually end up with one by forgetting, not by deciding.

The sections below take them one at a time.

Memory: mem

mem is the most memory the request may use, counted by the kernel for every process it starts. At 90% of it the kernel starts to push back and reclaim memory; at 100% it kills the request. There is no swap, so memory cannot quietly spill to disk and slow the host down. The smallest allowed value is 8M. The whole process tree dies together, and the warm zygote and the other requests keep running.

  0 ─────────────────────────────── 90% ───────────── 100%  (mem)
  normal use                        memory.high:      memory.max:
                                    kernel reclaims,  request killed,
                                    request slows     exit 137, oom_killed

CPU: cpu

cpu is how many CPU cores the request may use, as a number: 0.5 is half a core, 2 is two. The kernel checks it every 100 ms. A request that tries to use more is simply made to wait for the rest of that period. So a busy loop slows only itself, never the host, and it is not killed for it. This is also why a function driven past its quota shows a slow tail of tens of milliseconds; chapter 25 explains that number.

Processes: pids

pids is how many processes and threads the request may have at once. It is the defence against a fork bomb: a program that copies itself again and again until the machine can do nothing else. When the limit is reached, the next fork or new thread simply fails, and the program gets an error. 0 is not allowed.

Time: timeout

timeout is the wall-clock time a request may run, from start to end. When it runs out, Zygo kills the request through its cgroup, so every process the handler started dies with it, not just the one Zygo can see. The request exits with code 137, and the outcome says timed_out. timeout = 0 means “no limit” and needs --allow-unlimited; zygo up never accepts it.

Scratch space: scratch

A sandbox’s root file system is read-only. The one place it may write is /tmp, which lives in memory, and scratch is its size. Because it is memory, it counts against mem, so scratch must be smaller than mem, and Zygo warns you when it is more than half. The default is the smaller of 64M and half of mem. scratch is also the biggest single file a process may write, and /tmp holds 10 000 files at most. When it is full, writes fail with “No space left on device”.

Open files, connections, bandwidth and disk

nofile (default 1024) is how many files, sockets and pipes one process may have open; past it, open fails with “Too many open files”. connections (default 256) is how many TCP connections a networked function may hold at once; the firewall inside the sandbox answers the next one with a reset. It cannot be 0. bandwidth is how many bytes a second the function may send, and on hosts with an ifb device also receive. io_read and io_write limit disk reads and writes a second. These three have no default, so Zygo warns you when a served function leaves them unset.

Thread pools follow cpu

Many number-crunching libraries — OpenBLAS, OpenMP, MKL, numexpr, polars, Rayon — start one thread per CPU on the machine. In a sandbox with cpu = 2 on a big host, those threads then fight over two cores. On a 5-CPU host, eight numpy requests at once under cpu = 2 ran 24–38 a second with the host’s count and 155–173 a second with one thread each. So Zygo sets these variables to cpu, rounded up:

OMP_NUM_THREADS   OPENBLAS_NUM_THREADS   MKL_NUM_THREADS      NUMEXPR_NUM_THREADS
VECLIB_MAXIMUM_THREADS   RAYON_NUM_THREADS   POLARS_MAX_THREADS
PYTHON_CPU_COUNT   GOMAXPROCS

If the image or the spec sets any of them, that value wins.

Knowing which limit ended a request

A request that Zygo or the kernel stopped exits with 137, both for time and for memory, because both are a SIGKILL. When you need to know which, Zygo tells you: zygo run --outcome FILE writes timed_out, oom_killed, peak memory and wall time to a file, and a warm request reports the same fields. Chapter 12 shows the one-shot case in full.

Networking: four modes

A sandbox starts with no network at all. network opens it, one step at a time:

network = "none"     # the default

Egress means traffic going out. No mode lets anything connect in to a sandbox: Zygo never accepts connections for your code.

How egress works

egress and full give the sandbox’s network namespace to pasta. pasta is a small program that moves the sandbox’s packets out through the host’s normal sockets, running as your own user. Inside the namespace, Zygo installs an nftables firewall — the kernel’s packet filter — that lets only allowed traffic out. No root is needed anywhere. Under egress, Zygo also runs its own tiny DNS resolver inside the sandbox, and that resolver is what fills the firewall.

  ┌─ sandbox network namespace ─────────────────────────────────────────┐
  │                                                                     │
  │  handler                                                            │
  │    │ 1. "where is api.stripe.com?"                                  │
  │    ▼                                                                │
  │  ┌──────────────────────┐  2. on the allow list?                    │
  │  │ Zygo's DNS resolver  │     no  → "no such name" (NXDOMAIN)       │
  │  │ (on loopback)        │     yes → look it up, then …              │
  │  └──────────┬───────────┘                                           │
  │             │ 3. … add its addresses to the firewall FIRST,         │
  │             ▼    then answer the handler                            │
  │  ┌──────────────────────────────────────────────────────┐           │
  │  │ nftables firewall                                    │           │
  │  │  loopback, replies to allowed connections   → pass   │           │
  │  │  more than `connections` open               → reset  │           │
  │  │  10.x, 192.168.x, 169.254.169.254 …         → reject │           │
  │  │  addresses the resolver added, on that port → pass   │           │
  │  │  anything else                              → reject │           │
  │  └──────────┬───────────────────────────────────────────┘           │
  │             │ 4. connect to api.stripe.com:443                      │
  │             │ 5. out through tap0, the sandbox's network card       │
  └─────────────┼───────────────────────────────────────────────────────┘
                ▼
         ┌──────────────┐
         │ pasta        │──▶ normal sockets on the host ──▶ the internet
         │ (your user)  │
         └──────────────┘

Because the answer only comes back after the firewall is updated, a wildcard is checked against the name the handler really asked for. A service that changes its address keeps working, because each lookup adds the new one. The handler cannot use a resolver of its own to get around the list: under egress the only resolver it can reach is Zygo’s. The host’s search domains never enter the sandbox. A blocked connection is rejected at once, not silently dropped, so a program fails fast instead of waiting for a timeout.

A network needs a profile that can open a socket

seccomp = "strict" removes socket and connect (chapter 24). Under it an egress or full sandbox has a namespace, a firewall and an allowlist, and every connection still fails with EPERM before any of them is asked. So the combination is refused when the sandbox is declared, with the way out in the message. A runtime pool is strict unless it says otherwise, so a pool that calls out names seccomp = "default" next to its network.

The allowlist

allow is the list of places an egress function may reach. It is only valid with network = "egress". An egress function with an empty list can reach nothing, and Zygo warns you.

allow = ["api.stripe.com:443", "*.example.com:443", "203.0.113.0/24:5432"]

A CIDR is a way to write a range of addresses: 203.0.113.0/24 means every address that starts with 203.0.113..

Private ranges and the cloud metadata address

Private addresses are the ones used inside a home or company network, such as 10.x.x.x or 192.168.x.x. Link-local ones include 169.254.169.254, where cloud servers answer questions like “what are my credentials?”. That address is the first thing a compromised handler tries. So in every mode with a namespace — none, egress and full — these ranges stay closed:

10.0.0.0/8   172.16.0.0/12   192.168.0.0/16   127.0.0.0/8   169.254.0.0/16
100.64.0.0/10   0.0.0.0/8   224.0.0.0/4 (multicast)   240.0.0.0/4 (reserved)
::1   ::   fe80::/10   fc00::/7   ff00::/8 (multicast)

Multicast is on the list because a multicast group is a way to talk to the neighbours without naming any of them.

Only --allow-private-net opens them. An allow rule inside one of these ranges — a CIDR, or a single address such as 192.168.1.70:8765 — is refused unless you pass that flag. Before 0.1.4 a single address was read as a host name: serve accepted it, and the firewall then refused every connection to it as the private address it is, with nothing to say why.

Over the HTTP API a request body can never set it. Whoever starts the API can: zygo api --allow-deploy --allow-private-net lets what deploy callers serve name private addresses in allow (chapter 17).

The host’s own loopback

127.0.0.0/8 is on that list, but the sandbox has a loopback of its own, and the firewall lets it through: Zygo’s resolver lives there. So the question is whether the sandbox’s 127.0.0.1 can lead to the host’s. pasta can do exactly that, and does by default: it forwards a connection to the sandbox’s loopback on to the same port on the host’s loopback, and it answers for the gateway address itself and hands those connections to the host too. Zygo turns both off (--tcp-ns none --udp-ns none --no-map-gw). A Postgres, a Redis or zygo api bound to 127.0.0.1 on the host is out of reach from a sandbox, whatever the allowlist says and whatever the kernel’s Landlock can or cannot do.

  sandbox                              host
  ┌────────────────────────┐           ┌──────────────────────────────┐
  │ connect 127.0.0.1:5432 │──▶ own    │ 127.0.0.1:5432  postgres     │ ◀── not reachable
  │                        │  loopback │ 127.0.0.1:7700  zygo api     │ ◀── not reachable
  │ connect <gateway>:5432 │──▶ the    │                              │
  │                        │  router   │ 10.0.0.5:5432   postgres     │ ◀── allow 10.0.0.5:5432,
  └────────────────────────┘           └──────────────────────────────┘     --allow-private-net

A function that must reach a service on its own host is given that service on a second address the host has — a Docker bridge’s 172.17.0.1, or one on a dummy interface — with an allow rule naming it and --allow-private-net typed by a person. Not the host’s main address: pasta gives that same address to the sandbox’s own network card, so a sandbox dialling it reaches itself and is refused. A service on loopback only stays loopback only. Before 0.1.4 this was not so: an allow rule for any name on port 5432 also opened the host’s own 127.0.0.1:5432, and on a kernel without Landlock’s network rules (below 6.7) every port the host had bound on loopback was open.

Listening inside a sandbox

A function, and a zygo run sandbox, may open a TCP listener on its own loopback. Nothing from outside reaches it: pasta forwards no port in, and under none there is no interface but loopback. So a program that talks to itself over 127.0.0.1 — a Jupyter kernel, a Ray or Dask worker, a test suite with a local server, n8n’s runner with its health-check port — works in a sandbox the way it works on a laptop.

A runtime pool may not. Its requests belong to different tenants and share one network namespace, so a listener on loopback there would be a channel from one tenant’s request to another’s. strict, the pool default, removes the socket calls; under it, Landlock refuses bind in a shared namespace on kernels from 6.7 (chapter 13).

  function (one tenant's namespace)        runtime pool (shared between tenants)
  ┌──────────────────────────────┐         ┌──────────────────────────────┐
  │ listen 127.0.0.1:8888  ✓     │         │ listen 127.0.0.1:8888  ✗     │
  │ connect to it          ✓     │         │   strict: no socket at all   │
  │ from outside: nothing gets in│         │   Landlock: bind refused     │
  └──────────────────────────────┘         └──────────────────────────────┘

One corner under egress: the connect half of talking to yourself is subject to the allowlist’s port rule, which applies to loopback too. A function that listens on 5681 and connects to it names the port — allow = ["127.0.0.1:5681"], which needs --allow-private-net — or runs under none or full, where no port rule stands in the way. Until 0.1.3 no sandbox could listen on TCP at all on a kernel from 6.7, and a sealed one could not connect to itself either; ADR 0008 records why that changed.

If pasta or nft is missing

A networked sandbox needs both pasta and nft on the host. If either is missing, the sandbox does not start. Zygo never falls back to starting it with an open network. zygo doctor tells you what is missing, and chapter 22 covers the common problems.

Secrets: why not environment variables

A secret is a value that must not leak: an API key, a database password, a token. The usual way to pass one is an environment variable, but that is a poor fit for a sandbox. The zygote would hold it for its whole life, every forked request would inherit it, and a handler that prints its environment would print it. Zygo takes a different path: a secret is a file, and it exists only while one request runs.

Declaring a secret

You name the secret in the spec, and give its value in the shell that runs zygo serve or zygo up:

[fn.fetch]
secrets = ["STRIPE_KEY"]

export STRIPE_KEY=sk_live_...
zygo up

The value is read from your shell, not from the supervisor’s environment. If a name has no value, the command refuses and says which name is missing. The handler reads the secret as a file:

def handler(event):
    key = open("/run/secrets/STRIPE_KEY").read().strip()

A secret lives for one request

For each request, the supervisor writes the file /run/secrets/<NAME> from outside the sandbox. It is created with mode 0400 — readable only by its owner — from the first moment, not changed to that mode afterwards, so there is no instant when anyone else could read it. Inside the sandbox, only this function’s requests can read it — this one, and any other request of the same function running at the same moment, which has the same value anyway. The file is removed when the last of them ends.

  supervisor (holds the value)
      │
      │ request 7 arrives
      ├──▶ write /run/secrets/STRIPE_KEY  (0400, this function's requests only)
      │        │
      │        ▼
      │    ┌─────────────────────────────┐
      │    │ request 7: handler(event)   │  reads the file, uses the key
      │    └─────────────────────────────┘
      │        │ request 7 ends
      ├──▶ file removed
      │
      │ request 8 arrives ──▶ written again, removed again

  never in: the environment · the zygote's memory · the control socket

A request that is taken over can read its own function’s secret while it runs, and never a secret that another function uses. This matters most for AI agent tools: a model that writes the code cannot print a secret it never had in its environment.

env or secrets?

A name cannot be in both lists. Never put a secret in env.

The tenant secret store

Reading secrets from a shell works for one person at a terminal. It does not work for a platform whose customers each bring their own keys. A tenant is one such customer, with its own functions and tokens (chapter 17). For them Zygo keeps an encrypted secret store, one set of secrets per tenant, saved on disk under Zygo’s data folder.

zygo secrets keygen                              # print a new key (32 bytes, hex)
export ZYGO_SECRETS_KEY=...                      # before the supervisor starts
zygo secrets set acme STRIPE_KEY                 # prompts, echo off
echo "$KEY" | zygo secrets set acme STRIPE_KEY --stdin
zygo secrets ls acme                             # names only, never values
zygo secrets rm acme STRIPE_KEY

Over the HTTP API the same is PUT /tenants/<id>/secrets/<name>. Functions you serve from the CLI belong to the tenant named default.

How the store fills in values

When a function is served, Zygo first takes the values the client sent — for the CLI, from your shell. Then it fills in, from the tenant’s store, any name the client did not send. The client wins where both have a value. Only the names the spec lists are delivered; anything extra is dropped.

  spec: secrets = ["STRIPE_KEY", "DB_PASSWORD"]

  sent by the client:   STRIPE_KEY=sk_test_…   ─┐
                                                ├─▶  STRIPE_KEY  = sk_test_…  (client wins)
  tenant store (acme):  STRIPE_KEY=sk_live_…   ─┤    DB_PASSWORD = …          (from store)
                        DB_PASSWORD=…          ─┘
  a name with no value anywhere → the serve is refused, naming it

Secrets in a runtime pool

A runtime pool (chapter 13) is shared by every tenant, so it holds no secret values. It names them instead:

[runtime.py312]
image   = "python:3.12-slim"
agent   = "python"
secrets = ["STRIPE_KEY"]      # names a request may receive; no values here

Each request is given the calling tenant’s values — the tenant in the token or the X-Zygo-Tenant header; the default tenant for zygo exec — read from the store at that moment. Only the store: a pool never reads your shell, and a pool that names secrets on a host with no store key is refused when it is served, not on its first request. The files are written from outside, exist for that one request, and are removed when it ends.

  tenant acme calls py312             tenant beta calls py312
      │                                    │
      ├──▶ store: acme's STRIPE_KEY        ├──▶ store: beta's STRIPE_KEY
      ├──▶ /run/secrets/STRIPE_KEY         ├──▶ /run/secrets/STRIPE_KEY
      │    on a zygote acme has alone      │    on a different zygote
      └──▶ removed when acme's request ends

Two rules follow from “one directory per sandbox”:

• A request with secrets has its zygote to itself while the files exist. It takes an idle zygote; if every zygote is busy, the pool grows by one (up to max_warm), and at max_warm the caller gets busy, which the SDKs retry. Ordinary requests skip a zygote that is taken this way. Requests without secrets share zygotes as before.
• A missing name fails the request before anything runs. A tenant that has no secret under one of the pool’s names gets 400 (bad_spec) naming the secret and the route that stores it, never a value.

The store’s key

The key comes from ZYGO_SECRETS_KEY, or from a file named by ZYGO_SECRETS_KEY_FILE. It must be 32 bytes, written as hex or base64. A passphrase is refused: turning a password into a key safely needs extra machinery, and a store that quietly accepted hunter2 would be worse than one that says no. Zygo never saves the key. If you lose it, every stored secret is unreadable for good. Without a key, the supervisor still serves every function with no stored secrets; only the store refuses, and says which variable to set.

What the store protects, and what it does not

Each value is encrypted with ChaCha20-Poly1305, a modern cipher, with the tenant and the name bound to it. So a value cannot be moved to another tenant or another name by renaming files. A value may be at most 64 KiB; anything bigger is a file, not a secret. Values cannot be read back, only listed by name. The encryption protects the bytes on disk. It does not protect them from a process that can read the supervisor’s memory, and it is not a hardware security module; if you need that, use your cloud’s key service.

Loosening, by name

Three things are refused unless you type a flag that says exactly what it does:

The flags exist on zygo run, zygo serve and zygo mcp. zygo up has none of them. A spec that needs one has to be served on purpose, one function at a time, with the flag typed by a person.

15. Images and dependencies

A sandbox needs files to run: an interpreter, libraries, and your own packages. This chapter explains where those files come from — images, which Zygo pulls from normal registries, and the layers it builds on top of them — and how to keep the store tidy. There is no Dockerfile and no build step for you to run.

The big picture

A function’s file system is an image from a registry, plus up to three layers Zygo builds for you. The image is never changed. Each extra layer is built once and then shared.

   python:3.12-slim            from the registry; never changed
          │
          ├── + bytecode layer   .pyc files for the standard library     built on first pull
          ├── + system layer     apt packages: system = ["libwebp7"]     key: image + list
          └── + /venv            pip packages: requirements = "…"        key: image + file
                     │
                     ▼
   what the function sees as /   (read-only, except /tmp)
   shared by every function that names the same image and the same lists

Images, layers, tags and digests

An OCI image is the standard package format for containers, the same one Docker uses. It is a stack of layers: each layer is an archive of files, and stacking them gives one file tree. A registry is a server that stores images, such as Docker Hub or ghcr.io. A tag is a friendly name, like python:3.12-slim, that the owner can move to a new version at any time. A digest is the image’s fingerprint, like sha256:4f1b…, and it never changes: the same digest always means exactly the same bytes.

Pulling an image

To pull is to download an image into Zygo’s local store. Any OCI image from any registry works.

zygo pull python:3.12-slim                 # into the local store
zygo pull --platform linux/amd64 alpine:3  # another CPU type than this host's
zygo images                                # reference, digest, layers, size, when pulled
zygo images --json                         # the same, for a program: match `reference` exactly

--platform takes os/arch. Without it, Zygo pulls the version built for the host it runs on.

When Zygo pulls for you

zygo run pulls an image the first time you use it, like docker run. Two flags change that. --pull never refuses a missing image instead; the run exits 1, and --outcome records phase: plan. --pull always pulls again, for a tag that may have moved. zygo serve and zygo up never pull. A deploy should not quietly depend on a registry being up, so they stop and ask you to pull first.

The store

Zygo unpacks each layer once, into a content-addressed store: every layer is saved under its own digest, so the same layer is never stored twice, even when many images share it. A sandbox does not get a copy. The layers are mounted read-only into it, stacked with overlayfs (chapter 4). The store lives in Zygo’s data folder, ~/.local/share/zygo by default (chapter 21 says how to move it).

  store (on disk, once)                     sandboxes (no copies)
  ─────────────────────                     ─────────────────────
  layers/sha256:aa… ─┐
  layers/sha256:bb… ─┼── read-only mounts ─▶ sandbox 1   sandbox 2   sandbox 3
  layers/sha256:cc… ─┘                       (each stacks the same three layers)
  + the compressed download of each layer, kept beside it

Private registries: zygo login

A private registry needs a user name and a password or token. zygo login stores them:

zygo login ghcr.io -u you                                   # prompts, echo off
echo "$TOKEN" | zygo login ghcr.io -u you --password-stdin  # for CI

There is no --password flag, on purpose. A command-line argument is visible to every process on the machine through ps, and it lands in your shell history. Zygo checks the login against the registry before it saves it, so a typo fails now, not at the next deploy. The credential goes into Zygo’s own auth.json in its data folder, readable only by you (mode 0600).

Docker’s logins are read too

If you already logged in with Docker, Zygo can use that. It reads ~/.docker/config.json, or the folder named by $DOCKER_CONFIG, and never writes to it. When both have a login for the same registry, Zygo’s own auth.json wins.

  a pull from ghcr.io needs a login
      │
      ├─ 1. <data>/auth.json          (written by `zygo login`)    ← wins
      └─ 2. ~/.docker/config.json     (or $DOCKER_CONFIG; read only)

Dependencies without touching the image

What you would write in a Dockerfile goes in sandbox.toml instead. There are two kinds of dependency, and neither changes the image:

requirements = "./requirements.txt"   # Python packages, into a venv
system       = ["libwebp7"]           # apt packages, into a layer

Python packages: requirements

A venv (virtual environment) is a folder that holds a set of Python packages apart from the system’s. Zygo builds one from your requirements.txt, inside a sandbox, with the image’s own pip. That is the only way to be sure the packages match the Python that will run them. The venv is mounted read-only at /venv, with /venv/bin first on PATH. On the Lima VM, the first build of requests took about 2.5 seconds, and every later run found the built venv in a millisecond or two (chapter 25 has the numbers and the machine).

System packages: system

Some Python packages need a C library from the operating system, such as libwebp7 for WebP images. system lists Debian (apt) packages, optionally with a version: "libpq5=16.4-1". Zygo installs them inside a writable copy of the image, compares the result with the original, and saves the difference as a new OCI layer of its own. There is no Dockerfile, and nothing else is rebuilt.

  image layers (read-only) ──▶ writable copy ──▶ apt-get install libwebp7
                                                        │
                                  diff: only the files apt added or changed
                                                        ▼
                                          a new layer, "<image>+system.<key>"
                                          stacked on the image for this function

How the caches are keyed

Both kinds are built once and shared. Each is saved under a key: the image’s digest plus the list or the file’s bytes. Every function that names the same image and the same list uses the same build. Edit the list and you get a new key, so a new build. A different image is also a new key, because a package built for one Python often fails with a confusing error on another.

One-shot runs share the same cache

zygo run can use a requirements file too:

zygo run --requirements ./requirements.txt python:3.12-slim python3 -m pytest

The first job with a given image and file builds the venv. Every job after that, and every warm function with the same pair, reuses it.

The Python bytecode layer

Python turns each .py file it imports into bytecode — a faster form it can run — and normally saves it as a .pyc file for next time. The official python:*-slim images ship no .pyc files at all: python:3.12-slim has 1097 .py files in its standard library and none compiled. A sandbox’s root is read-only, so Python cannot save them either. So every run compiled every module again; import re alone took 34 ms.

How the bytecode layer works

The first time Zygo pulls or runs such an image, it compiles the standard library once, inside a sandbox, into a layer of its own. For python:3.12-slim that took about 3.2 s and made 18.6 MB. The layer is stacked on the image, so the .pyc files sit next to the sources. They are marked unchecked: a layer never changes, so there is nothing to check them against. On the Lima VM used for testing, a script that imports ten common modules (re, json, urllib.request and more) went from 165 ms to 35 ms (chapter 25 has the table).

  without the layer                         with the layer
  ─────────────────                         ──────────────
  import re                                 import re
    └─ read re.py, compile it  (every run)    └─ read re.cpython-312.pyc  (ready)
    └─ cannot save .pyc: root is read-only

When there is no bytecode layer

An image that already has bytecode, or has no Python, is used as it is. If the build fails, Zygo prints a warning and uses the original image; a run never fails because of it. ZYGO_BYTECODE=0 turns the layer off.

Nix: not built

sandbox.toml accepts a nix field, so the file format is ready for it, but Zygo does not build Nix packages yet. Serving a function that sets nix fails with a clear error. Use system or requirements for now.

zygo.lock: the same image next time

A tag can move: python:3.12-slim next month is not the same bytes as today. zygo up writes zygo.lock next to the spec. It records the digest each image resolved to, the versions apt chose for each system package, and the hash of each requirements file. Commit it to version control.

  zygo up
     │
     ├─ spec edited for this function?      yes ─▶ serve it, rewrite its lock entry
     │                                       no
     ├─ image digest same as zygo.lock?     yes ─▶ serve it
     │                                       no
     └─ refuse this function, print both digests
          → zygo up --relock   accepts the new image

Editing the spec re-locks that function without asking, because you just asked for the change. Chapter 20 has the rules in the reference.

Removing an image: zygo image rm

zygo image rm python:3.11-slim

This removes the image, the system-package images derived from it, and then anything that only they kept alive. It is refused while a warm function uses the image, because that function’s root file system is those layers, mounted. Stop the function first.

Pruning the store

Pruning deletes what is no longer needed. Start with --dry-run, which only reports.

zygo image prune --dry-run
zygo image prune                          # only what nothing can reach any more
zygo image prune --unused-for 30d --blobs

A flattened root is all of an image’s layers copied into one folder, which Zygo makes on kernels that cannot stack layers for a normal user. Every use of a cache is recorded, so a venv used yesterday is safe even at --unused-for 7d. --blobs roughly halves the store. The cost is a new download if an unpacked layer is ever lost.

16. Deploying and running in production

This chapter is about the day after the first demo: putting a project’s functions live, keeping them healthy, and changing them without dropping a request. It also covers running Zygo inside a container and on Kubernetes, choosing an isolation backend, and the things Zygo will not do for you.

From a spec file to warm functions

Deploying here means one command, zygo up. It reads sandbox.toml, asks the supervisor to warm every [fn.*] section, and writes zygo.lock. The supervisor is the long-lived Zygo process that keeps the warm functions (chapter 6).

  sandbox.toml ──▶ zygo up ──▶ supervisor ──┬──▶ fn.resize   ✓ warm
  + your shell's                            ├──▶ fn.parse    · unchanged
    secret values                           └──▶ fn.fetch    ✗ failed, reason printed
                     │
                     └──▶ zygo.lock  (image digests, apt versions; commit it)

zygo up        # every [fn.*] warm
zygo down      # stops what this spec declares, and nothing else

What zygo up prints

Functions come up in the order the file declares them, and each one gets a line of its own. ✓ means started or replaced, · means unchanged and left alone, and ✗ means it failed, with the reason. One failure does not stop the others: a file with ten functions where the sixth cannot start still brings up the other nine, and says which one failed. If any function failed, zygo up exits with 1. zygo up --json prints one document with every failure and its reason. The lines look like this (the numbers are only an example):

✓ resize — python, 38 MB, ready in 412 ms
· parse — unchanged (exec, 3 MB)
✗ fetch — fn.fetch.secrets: no value for STRIPE_KEY; set it in the environment …

Run it again: only what changed restarts

Running up after an edit is not a second deploy. For each function, Zygo compares four things with what is already running: the resolved spec, the secret values, the bytes of the handler file and the bytes of the requirements file. If all four are the same, the function is unchanged and left alone — warm memory, request counters and all. If it was paused or cold since the last deploy, up brings it back to warm. If anything differs, it is replaced.

Blue/green replacement

A changed function is replaced blue/green: the new version is started next to the old one, and traffic moves only when the new one is ready. Requests the old sandbox already accepted finish on it. Requests waiting in its queue go to the new one. No request is dropped, and none sees a half-started function.

  time ──────────────────────────────────────────────────────────▶

  old (blue)   ████████████████████████▓▓▓▓▓ finishes what it accepted, then stops
  new (green)            ░░░░░░ warming ░░░░████████████████████████████████
                                           ▲
                                  switch: the new one is warm;
                                  queued requests go to it

zygo.lock in a deploy

up records the image digest, apt versions and requirements hash for each function in zygo.lock. Commit it. If a tag has moved under a spec nobody edited, up refuses that function and prints both digests, so a deploy never quietly runs different code; zygo up --relock accepts the move. Chapter 15 explains it with a diagram. Remember too that up never pulls: pull the images first.

zygo up has no escape flags

--allow-host-net, --allow-private-net and --allow-unlimited do not exist on zygo up. A deploy from a file should never loosen a wall just because the file says so. A function that really needs one must be served by hand with zygo serve and the flag typed by a person (chapter 14). The one flag of the three on zygo api is --allow-private-net, for an embedder whose pools call back to a service of its own; it is typed by whoever starts the API, never sent in a request.

The supervisor

You rarely start the supervisor yourself. zygo serve, zygo up, zygo token, zygo secrets and zygo api start it when none is running, and give up if it is not ready within 10 seconds. It runs as your own user, never as root. When it gets SIGTERM — the normal “please stop” signal from systemd, Docker or Kubernetes — it stops taking new requests and gives the running ones up to 25 seconds to finish. That is less than the 30 seconds most process managers wait before they force a kill.

zygo supervisor status    # what it is, and where its socket lives
zygo supervisor stop      # drain, exit; the next serve or up starts a new one
zygo supervisor run       # run it in the foreground, to see why it will not start

zygo supervisor is a hidden command: it does not appear in zygo --help, because you seldom need it.

Running zygo api under systemd

On a Linux host the natural home for zygo api is a systemd unit: it starts at boot, restarts if it dies, and its logs go to the journal. One line in that unit matters more than the others. When any process inside a unit’s cgroup is killed by the kernel’s out-of-memory killer, systemd’s default (OOMPolicy=stop) stops the whole unit. A sandbox that goes over its mem limit is killed exactly that way, inside its own cgroup, which sits under the unit’s. So with the default, one request allocating too much memory takes down the API, the supervisor and every pool with it, and every later request is refused until somebody restarts the unit. OOMPolicy=continue tells systemd the kill was handled and the unit goes on.

  n8n-zygo-api.service            ◀── systemd watches memory.events here
  └─ zygo api
     └─ zygo.slice/
        └─ tenants/acme/resize/
           └─ req-0192            ◀── the kernel kills the hog here

A user unit, in ~/.config/systemd/user/zygo-api.service:

[Unit]
Description=Zygo API

[Service]
ExecStart=/usr/local/bin/zygo api --listen 127.0.0.1:7700
Environment=ZYGO_API_TOKEN=change-me
Restart=on-failure
# A sandbox over its memory limit is killed alone; the unit goes on.
OOMPolicy=continue
# The unit owns its cgroup subtree, which is where every sandbox goes.
Delegate=yes

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now zygo-api
loginctl enable-linger $USER          # keep it running when you log out

The same two settings on a transient unit, which is how a script or another service often starts it:

systemd-run --user --unit=zygo-api -p OOMPolicy=continue -p Delegate=yes -- zygo api

zygo doctor reports the unit’s policy as systemd OOM policy, and zygo api prints a warning at start when the unit it is in would stop. Chapter 22 has the journal lines this looks like when it has already happened.

Watching it

zygo ps                      # what is warm, and its counters
zygo top                     # ps on a timer, plus rates
zygo stats resize            # latencies over the log window
zygo logs resize -f          # the zygote's output and every request
zygo logs resize --failed -n 20

zygo stats keeps two kinds of number apart and labels each: counters since the function warmed up, and latencies over the log window. It refuses to report a 99th percentile from fewer than a hundred requests, rather than invent one. Over the API, GET /metrics gives counters in Prometheus format (chapter 17).

Idle functions

A warm function holds memory. So Zygo lets unused ones step down, in two stages. After idle_timeout (default ten minutes) with no request, the function is paused: frozen in place, still in memory, and thawed by the next request in a few milliseconds. After cold_after (default one hour) it is cold: the sandbox is dropped, only the spec is kept, and the next request pays the full warm-up again.

idle_timeout = "10m"
cold_after   = "1h"

  request ──▶ WARM ──(idle_timeout: 10m)──▶ PAUSED ──(cold_after: 1h)──▶ COLD
               ▲                              │                            │
               └────── next request: ~ms ─────┘                            │
               └────── next request: pays the whole warm-up again ─────────┘

zygo ps shows the state of each function. To bring one back before a real request arrives, call POST /fn/<name>/warm over the API, or client.warm(name) in the SDKs.

Capacity

Zygo’s capacity is a budget for one machine. Each function runs concurrency requests at once (default 4). Up to four times that many more wait in a queue, for at most five seconds. Past that, Zygo answers busy: HTTP 429 with the numbers, or exit code 75 from zygo exec. This is backpressure — a polite “not now” — and not a failure. The request never ran, so retrying it is safe and correct.

  concurrency = 4

  running  [■][■][■][■]                         4 at once
  queue    [·][·][·][·][·][·][·][·]…[·]         up to 16 more, 5 s at most
  more     ──▶ 429 busy (exit 75): never ran, retry later

Upgrading Zygo

A warm function cannot be carried across to a new Zygo binary. An upgrade is always: drain, exit, start the new version, warm up again. ADR 0004 is the study that explains why. To drain over the API:

curl -X POST -H "Authorization: Bearer $ZYGO_API_TOKEN" \
    "http://127.0.0.1:7700/drain?grace_ms=60000"

This stops taking new requests, finishes the running ones, and exits. The answer carries the number still running: in_flight: 0 is a clean drain, and anything else means the grace time ran out. The cost is a warm-up per function afterwards: about 150–185 ms for a Python handler, the supervisor’s own start included, measured on a Raspberry Pi 5 (chapter 25); heavy imports add to it. No request is dropped, as long as something else is ready to take them — which is what two replicas and min_warm are for.

  replica A: ███ serving ███ drain ▓▓ exit │ start new ░ warm ░ ███ serving ███
  replica B: ███████████████ serving ███████████████████████████ drain ▓▓ …
                            ▲ B takes all the traffic while A upgrades

Debugging a live function

zygo shell resize
zygo shell resize -- cat /proc/1/cgroup

zygo shell starts a new process inside the function’s namespaces. The warm zygote is not touched: it keeps its memory and keeps serving. The shell sees the sandbox’s files, processes, network and host name, and holds no capabilities. It is on purpose not under the seccomp filter, the Landlock rules or the tenant’s cgroup: a debug shell that the memory limit kills is no use to anyone.

Calling it from a program

zygo api                     # listens on 127.0.0.1:7700, bearer-token auth

import zygo_sdk as zygo
client = zygo.connect()
out = client.fn("resize")({"url": "..."}).result   # what the handler returned

The API starts call-only: a token can call the functions somebody declared in a spec file, and nothing else. --allow-deploy adds serving, stopping and one-shot runs, which together amount to a shell, not an API. ZYGO_API_TOKEN is the operator’s token. For a platform with customers, zygo token mint --tenant acme prints a token that registers scripts and calls functions for that customer only; the tenant comes from the token, never from anything the caller sends. Chapter 17 covers the API and SDKs in full.

Giving it to an agent

An AI agent can use Zygo through MCP, the Model Context Protocol, a standard way for agents to call tools. The whole installation is one entry in the agent’s configuration:

{ "mcpServers": { "zygo": { "command": "zygo", "args": ["mcp"] } } }

Chapter 17 describes the tools it offers.

Running Zygo inside a container

Zygo builds sandboxes, so a container it runs in must let it. It needs no privileges and no capabilities, and --privileged is not the answer. The container image asks for three things. A host with AppArmor needs a fourth, and sandboxes with a network need a fifth. zygo doctor names each one that is missing.

docker run --user 0:0 \
  --security-opt seccomp=unconfined \
  --security-opt systempaths=unconfined \
  --security-opt apparmor=unconfined \
  --cgroupns=host --cgroup-parent=/zygo \
  -v /sys/fs/cgroup/zygo:/sys/fs/cgroup/zygo:rw \
  --device /dev/net/tun \
  -v zygo-data:/var/lib/zygo \
  -p 7700:7700 -e ZYGO_API_TOKEN=... \
  ghcr.io/mhmtskrc2/zygo

--user 0:0 is there because the cgroup folder belongs to root. As the image’s own user, 65532, Zygo could not write it. No sandbox runs as that root: each one is in a user namespace of its own.

With Docker’s systemd cgroup driver — the default on Ubuntu and Debian — the parent must be a slice: --cgroup-parent=zygo.slice and /sys/fs/cgroup/zygo.slice.

The three things, and two more

  ┌─ the container Zygo runs in ──────────────────────────────────────────┐
  │                                                                       │
  │  1. seccomp=unconfined      ─▶ may call unshare(CLONE_NEWUSER)        │
  │  2. systempaths=unconfined  ─▶ /proc not masked, so a fresh /proc     │
  │                                can be mounted in a user namespace     │
  │  3. its own cgroup subtree  ─▶ somewhere to put each sandbox          │
  │  4. apparmor=unconfined     ─▶ (AppArmor hosts) mounts not refused    │
  │  5. /dev/net/tun            ─▶ (egress / full) pasta's network card   │
  │                                                                       │
  │   ┌──────────┐ ┌──────────┐ ┌──────────┐                              │
  │   │ sandbox  │ │ sandbox  │ │ sandbox  │  ◀── the real boundary       │
  │   └──────────┘ └──────────┘ └──────────┘                              │
  └───────────────────────────────────────────────────────────────────────┘

Give it its own cgroup, not the host’s

The usual advice is to mount all of /sys/fs/cgroup read-write. That works, but it hands the container the host’s whole cgroup tree: it could then change limits on any cgroup on the machine, including other containers’. That is worse than the privilege you were trying to avoid. Give Zygo a subtree of its own instead, as the command above does with /zygo. Sealed sandboxes — network = "none", the default — do not need /dev/net/tun.

The host’s AppArmor still applies

A host’s own AppArmor rules still reach into a container. Ubuntu’s passt profile attaches to the path /usr/bin/pasta and refuses the pid file Zygo asks for. An image that installs pasta somewhere else on PATH is not affected, and the Zygo image puts nothing at /usr/bin/pasta for exactly this reason (chapter 22 has the error and the fix).

Why this is safe enough

A container set up this way is no easier to escape than the host it runs on. That is the point: the wall Zygo enforces is the one it builds inside — the sandboxes — not the container around it. The test scripts tests/linux/verify_supervisor.sh and tests/linux/verify_api.sh run in exactly this shape in CI, unprivileged, on every change. make verify-oci builds the image and runs a sandbox inside it the same way.

The container image

make oci-image      # build it from the binary make already checked
make verify-oci     # build it and run a sandbox inside it, unprivileged

The published image is ghcr.io/mhmtskrc2/zygo:<version>, for linux/amd64 and linux/arm64. It is signed with cosign and listed in the release’s SHA256SUMS. It is Alpine plus a few programs:

The data folder is /var/lib/zygo (ZYGO_DATA_HOME), a volume. The image has a health check on /healthz.

No Python in the Zygo image

That last row surprises people. The Zygo image does not need Python, because your function never runs in it: it runs in python:3.12-slim or whatever image the spec names, pulled into Zygo’s store. If you want those images already on the node, so a cold start is not a trip to the registry, build a worker image: Dockerfile.worker runs zygo pull at build time and is a few lines long. It trades a bigger image for no waiting; pull only the images you really serve.

The image is call-only by default

The image’s default command is api --listen 0.0.0.0:7700, with no --allow-deploy. A token can call the functions somebody declared and nothing else. Deploy rights over HTTP are a shell, and a default that hands them out is one nobody reads the flag for. Pass --allow-deploy when the thing in front of the API is your own control plane. The worker image does.

Verifying a published image

cosign verify ghcr.io/mhmtskrc2/zygo:0.1.5 \
  --certificate-identity-regexp '^https://github\.com/.*/\.github/workflows/release\.yml@refs/tags/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

This is keyless signing: the signature is tied to the GitHub workflow that built the image and the tag it was built from. There is no private key to keep safe, and none to leak.

Zygo on Kubernetes

examples/kubernetes/ has a working manifest. A CI job named kubernetes applies exactly this file to a fresh kind cluster on every change and runs a sandbox in it, so the file is tested, not just an illustration.

kubectl apply -f examples/kubernetes/deployment.yaml
kubectl -n zygo port-forward svc/zygo 7700:7700

  namespace zygo
  ┌─────────────────────────────────────────────────────────────────────┐
  │ Secret zygo: api-token, secrets-key                                 │
  │ Service zygo :7700 ─────────┬──────────────────────┐                │
  │                             ▼                      ▼                │
  │  ┌─ pod 1 ─────────────────────────┐  ┌─ pod 2 ─────────────────┐   │
  │  │ zygo api --allow-deploy         │  │ (same)                  │   │
  │  │ securityContext: see below      │  │                         │   │
  │  │ /var/lib/zygo  emptyDir 20Gi    │  │                         │   │
  │  │ /run/zygo      emptyDir, memory │  │                         │   │
  │  │ readiness: /healthz             │  │                         │   │
  │  │ liveness:  zygo doctor --json   │  │                         │   │
  │  │ preStop:   POST /drain          │  │                         │   │
  │  └─────────────────────────────────┘  └─────────────────────────┘   │
  └─────────────────────────────────────────────────────────────────────┘
     replicas: 2 · maxUnavailable: 0 · maxSurge: 1 · no hostPath anywhere

The Secret holds two values: the operator’s API token, from which every customer token is minted, and the key that seals per-tenant secrets (chapter 14). The pod asks for 1 CPU and 2 GiB, and sets a memory limit of 8 GiB but no CPU limit: each tenant’s CPU is already capped by a cgroup Zygo sets per request, and a pod-wide CPU ceiling would also slow the supervisor itself.

What securityContext is for, and privileged: true

A pod needs the same things as the container above. One has a field of its own, two come with privileged: true, and one is a node setting:

privileged: true is there for the third row, together with runAsUser: 0: the pod’s cgroup belongs to root, and privileged gives capabilities to root only. It also brings an unmasked /proc, /dev/net/tun and no AppArmor profile, which a pod without it would have to ask for as the table above describes. The field for the first, procMount: Unmasked, is refused by Kubernetes unless the pod also sets hostUsers: false, so the example does not use it. The cost is smaller than it looks, because the wall Zygo enforces is the sandbox it builds inside the pod — namespaces, seccomp, Landlock, a cgroup per request — not the pod itself. Still, run it on nodes of its own, and read chapter 23. The day Kubernetes can say “give this pod its own cgroup subtree”, that line goes, and nothing else changes.

Rolling out without dropping a request

Three settings together: maxUnavailable: 0, so no pod stops before its replacement is ready; a preStop hook that calls POST /drain; and a terminationGracePeriodSeconds (120 in the example) longer than your longest request. Kubernetes does these in the right order: it takes the pod out of the Service before preStop runs, so the drain only finishes work already accepted. /healthz answers 503 from the moment a drain starts, which also takes the pod out of any load balancer that watches readiness.

  kubectl rollout
     │
     ├─ start new pod ─▶ startupProbe /healthz passes (warm) ─▶ ready
     ├─ old pod removed from the Service's endpoints
     ├─ preStop: POST /drain?grace_ms=60000
     │      stop admitting · finish in-flight · reply {in_flight: 0} · exit
     └─ old pod gone          (terminationGracePeriodSeconds: 120)

Log the drain’s answer: in_flight: 0 is clean, and anything else means the grace time ran out.

Three probes, three questions

The image store on Kubernetes

The store is an emptyDir, because it is a cache: a pod that moves to another node pulls again what it needs. The things that are not a cache — tenants, tokens, sealed secrets — belong in your control plane’s database, with the API as the way in. Two alternatives are both fine: a PVC (a persistent volume), if re-pulling costs you more than a volume to manage, or a worker image with the images baked in, which costs nothing at run time. No hostPath appears anywhere, so pods can be scheduled freely.

Trying it on kind

kind.yaml creates a one-node kind cluster (Kubernetes in Docker) that can run the manifest. It needs cgroup v2 on the host. A real cluster does not need this file, but it needs cgroup v2 too.

What the Kubernetes example leaves out

• An Ingress. The API belongs to your control plane, not the internet. Put a service of your own in front of it.
• Autoscaling. A warm pool costs memory, not CPU. The signal that matters is how many requests are waiting, and GET /metrics does not publish that yet: it has requests, failures, memory and state per function (chapter 17). Until it does, scale on memory (zygo_function_rss_bytes) and on the 429 answers your callers see.
• A PodSecurityPolicy or Gatekeeper rule. A cluster that enforces one will need an exception for this namespace, and that exception is yours to write.

Choosing an isolation backend

isolation = "ns"     # ns | gvisor | vm

A backend is the kind of wall between the sandbox and the host. You change it with one field; nothing else in the spec changes.

  ns       your code ─▶ seccomp, Landlock, namespaces ─▶ host kernel
  gvisor   your code ─▶ gVisor (a kernel in user space) ─▶ host kernel
  vm       your code ─▶ guest kernel ─▶ KVM (hardware) ─▶ host kernel
           ─────────────────────────────────────────────────────────▶
           cheaper, and warm functions               stronger wall

The three backends in detail

ns uses one kernel, shared with the host, with every lock the kernel offers turned on. Every number in this book is measured on it. gvisor puts a kernel written in user space between the sandbox and yours: a smaller attack surface, at a cost on every syscall. Warm functions and networked sandboxes on it are refused with a reason, never weakened. vm is a hardware wall: libkrun and KVM, with a guest kernel of its own. On the Raspberry Pi 5 a one-shot run took about 420 ms, against about 73 ms for ns on the same machine. The guest writes to a private layer, bounded by scratch, and nothing it writes reaches the shared image or the next sandbox. It has no networking and no warm functions yet.

Installing a backend

zygo backend list        # what this host can actually use
zygo backend install gvisor
zygo run --isolation gvisor python:3.12-slim python3 -c 'import platform; print(platform.release())'

zygo backend install gvisor downloads gVisor’s runsc from its official release bucket on storage.googleapis.com and checks its sha512 before unpacking it. The vm backend needs a Zygo binary built with --features vm (or make vm-build) and a guest kernel file at <data>/backends/krun/Image. There is no published kernel to download yet: zygo backend install vm tells you to build one with make vm-kernel and where to copy it. zygo doctor reports it once it is in place.

Which backend to pick

Use ns for code you chose, or code you half trust. For code you did not choose, use the strongest wall you can get. Read chapter 23 before you trust any of them with something that matters — in particular the part about where the wall is weaker than it looks. ADR 0002 explains why warm functions stay on ns.

What Zygo will not do

This list is plain on purpose: a tool that is vague about its limits is worse than one that lacks a feature.

• Run on macOS or Windows natively. Sandboxes are a Linux feature. On a Mac, Zygo manages a Linux VM for you.
• Accept connections. No mode of Zygo serves your traffic. A function is called through the CLI, the SDKs or Zygo’s own HTTP API; you put your own ingress in front of that.
• Scale past one machine. Capacity is a budget per host, and a 429 past it.
• Replace Docker. Zygo uses OCI images and none of Docker’s runtime. If you need docker compose, long-running services or published ports, you need Docker.
• Analyse the code it runs. Zygo confines hostile code. It does not tell you the code was hostile: there is no audit mode, no network log and no verdict.
• Hide the kernel. The ns backend shares one kernel with the host, and this book says so everywhere.

17. The HTTP API, the SDKs and MCP

The CLI is one way in. Programs use the HTTP API — directly, or through the Python, Node and Elixir clients — and agent hosts use the MCP server. All of them end at the same supervisor, the long-lived Zygo process that owns the warm sandboxes.

  your program ──▶ Python/Node/Elixir ─┐
  curl ────────────────────────────────┤  HTTP  ┌──────────┐ unix socket ┌────────────┐
                                       └───────▶│ zygo api │────────────▶│            │
                                                └──────────┘             │            │
  CLI (zygo exec, zygo up, …) ──────────────────────────────────────────▶│ supervisor │
                                                                         │            │
  agent host ──▶ zygo mcp (stdin/stdout) ───────────────────────────────▶│            │
                                                                         └────────────┘

This chapter has three parts. First the HTTP API itself: how to start it, who may call what, the routes and the answers. Then the three SDKs, and the features they reach: tenants, secrets, files, streaming, cancelling, runtime pools and dependency sets. Last, the MCP server for agent hosts.

Part one: the HTTP API

zygo api is a small web server. It turns HTTP requests into messages for the supervisor, and turns the supervisor’s answers back into HTTP. It does not run sandboxes itself. Every boundary a sandbox has is built by the zygo binary and enforced by the kernel, so nothing a caller sends over HTTP can do more than the rules below allow.

Starting the API

zygo api runs the API in the foreground; run it under systemd or in a container for production. It listens on 127.0.0.1:7700, or where [api] listen or --listen says — an IP:PORT or unix:///path (created with mode 0600, so only your user can open it). By default every caller needs a bearer token — a secret string sent in the Authorization header — on every kind of listener, unix sockets included. --no-auth (or [api] auth = "none") turns that off, and is refused anywhere but a unix socket or a loopback address. So a unix socket or loopback only permits turning auth off; it never skips it by itself. zygo api --openapi prints the OpenAPI 3.1 document for this build; the same file is committed as spec/openapi.json.

Who is calling: tokens and tenants

  ZYGO_API_TOKEN (bootstrap) ───────┐
                                    ├──▶ OPERATOR: the host's owner. May act for
  operator token (zygo token mint) ─┘    any tenant with the X-Zygo-Tenant header.

  tenant token (zygo token mint --tenant acme) ──▶ TENANT acme: sees and calls
                                                   only its own things.

A tenant is one customer of whoever embeds Zygo. A token proves who is calling, and the caller cannot choose it. The bootstrap token comes from ZYGO_API_TOKEN; more are minted with zygo token mint (secrets look like zygo_ plus 64 hex characters). Only a hash is stored, so a lost token cannot be shown again — revoke it and mint another. A revoke works from the next request. An operator acting for a tenant sends X-Zygo-Tenant: acme; a tenant token that sends a different tenant gets 403. The sections Tenants and Tokens below show both from the SDKs.

The deploy gate

Calling a function that exists is one thing; creating sandboxes is another, because whoever can create one can run any image with any mount as your user. That is a shell, not an API. So zygo api starts call-only: a token reaches the functions somebody declared in a spec file that was reviewed, and nothing else. The routes that create, change or destroy things need deploy rights. An operator token minted with zygo token mint always has them, because minting it already needed them. The bootstrap token has them only when zygo api was started with --allow-deploy. A tenant token never does, whatever the flag says: one customer does not get to name an image because another is trusted.

  request with a valid token
        │
        ▼
  route needs deploy rights? ──no──▶ allowed (a tenant sees only its own things)
        │ yes
        ▼
  tenant token? ──yes──▶ 403
        │ no
        ▼
  operator token minted with `zygo token mint` ──▶ allowed
  bootstrap ZYGO_API_TOKEN ──▶ allowed only if `zygo api --allow-deploy`, else 403

Which routes are gated

These need deploy rights: PUT and DELETE /fn/{name}, POST /run, POST and DELETE /runtimes, PATCH /tenants/{id}/limits, PUT and DELETE /tenants/{id}/secrets/{name}, DELETE /tenants/{id}, DELETE /scripts/{digest}, DELETE /blobs/{digest}, DELETE /deps/{id}, POST /drain, and every token route. POST /tenants and GET /tenants are for the operator only, but do not need deploy rights.

These are not gated, and any valid token may use them: PUT /scripts, GET /scripts/{digest}, PUT /blobs, POST /deps, GET /runtimes, POST /runtimes/{name}/call and DELETE /requests/{id}. Registering code that still needs a pool to run, and calling a pool somebody else declared, is exactly what a call-only token is for.

Why each one is gated

Serving a function, a one-shot run and creating a pool all name an image and mounts, which is running code as the user Zygo runs as. Deleting a script, a blob or a dependency set is gated because each store is shared by digest or id: forgetting one forgets it for every tenant that sent the same bytes. Limits, secrets and tokens are the operator’s relationship with a customer, and draining stops the host serving anybody. Without the rights, the SDK raises AuthError; the message names the flag for an operator, and says what a tenant token is for instead.

What stays off for everybody

A request body can never remove a guarantee. Host networking and unlimited limits are refused over HTTP whatever the caller and whatever the body says. Set those where the sandbox is declared, in sandbox.toml (see chapter 20).

Private-range egress has one more place: the command that starts the API. zygo api --allow-deploy --allow-private-net lets what deploy callers serve — PUT /fn/{name}, POST /runtimes, POST /run — name private and link-local addresses in allow. It opens nothing by itself: a rule still names the address and the port, and everything else in those ranges stays shut. It is for an embedder whose scripts must call back to a service of its own, which chapter 14 says to put on an address the host really has. Without --allow-deploy it does nothing, because only a deploy call declares a sandbox, and GET /version reports it as private_net for callers who can use it.

  zygo api --allow-deploy                       zygo api --allow-deploy --allow-private-net
  POST /runtimes  allow=["10.0.0.5:4010"]       POST /runtimes  allow=["10.0.0.5:4010"]
    → 400 bad_spec: targets a private range       → served; 10.0.0.5:4010 and nothing else

Request and response headers

The routes

“any” means any valid token; “op” means operator; “deploy” means deploy rights. Bodies are JSON unless marked raw; the limit is 16 MiB.

A layer in a body is one function’s table from sandbox.toml, sent as JSON: the same fields [fn.NAME] takes — image, entry or cmd, mem, network, allow, secrets, and so on — with every field optional, exactly as chapter 20 lists them. It is called a layer because it is merged over [defaults] the way a [fn.NAME] table is. base_dir is the folder on the server that its relative paths (entry, mounts, requirements) are relative to.

A batch runs at most 16 of its events at the same time. ?workspace= on a function call takes only a blob digest, never an inline tar (see Files in and out).

Status codes

Errors always have the shape {"error": "…", "code": "…"}.

A successful call

{
  "result":     {"count": 3},
  "request_id": "req-01f3…",
  "stdout":     "",
  "stderr":     "",
  "metrics":    {"wall_ms": 4.1, "cpu_ms": 2.3, "peak_rss_kb": 18420}
}

A streamed call (?stream=1) answers application/x-ndjson: one line per event, {"stream": "stdout" | "stderr" | "progress", "data": …}, then a last line with the body above and a status. Watching a request explains it.

The answer of a one-shot run

POST /run answers with more than an exit code, because the exit code alone cannot carry what a caller needs:

{"exit_code": 137, "timed_out": false, "oom_killed": true,
 "peak_rss_kb": 65536, "wall_ms": 412.7, "stdout": "", "stderr": "",
 "started": true, "phase": "run"}

A deadline kill and an out-of-memory kill are both SIGKILL, so both show exit code 137. timed_out comes from the launcher, which enforced the deadline; oom_killed comes from the kernel’s own counter in the sandbox’s cgroup (the kernel group that holds its limits). Neither is a guess. A caller deciding between “too slow” and “too much memory” — an online judge, a CI step — has nothing else to go on.

started, and when POST /run says 408

started is whether the program ran at all. false, with phase naming what failed (plan or start), is Zygo failing to build the sandbox. A caller reports that as unavailable, not as the code’s failure, and ok is false for it. An API one release behind does not send the field, and only answered once the program had run, so the SDKs default it to true.

The answer is a 200 whenever the sandbox ran, whatever ended it: a non-zero exit, its own timeout, an out-of-memory kill. The body says which. It is a 408 only when the API’s own outer deadline fired, the child was killed, and what it would have said is unknown — Zygo failing to finish, not a sandbox doing its job.

Metrics, OTLP and usage events

/metrics gives Prometheus series: zygo_api_requests_total, zygo_api_errors_total, and per function zygo_function_requests_total, zygo_function_failures_total, zygo_function_rss_bytes and zygo_function_state. --otlp-endpoint URL pushes the same numbers, plus per-tenant requests, outcomes, CPU and wall time, to an OpenTelemetry collector. --usage-webhook URL POSTs batches of usage events for billing — {tenant, function, script, request_id, wall_ms, cpu_ms, peak_rss_kb, outcome, finished_ms} — at least once, so key on request_id. Both count only requests that went through this API process.

Who sees what on /metrics. It needs a token, and the answer depends on the token, the same way GET /fn does. A tenant token gets zygo_api_requests_total and zygo_api_errors_total, which carry no names, and the per-function series for its own functions only. An operator token gets every function on the host. There are still no per-tenant series; the OTLP push and the usage webhook are where per-tenant numbers live.

Usage, for billing has the detail on the usage events.

Part two: the SDKs

There are three clients over the HTTP API: Python, Node and Elixir. Python and Node have no dependencies: the standard library has an HTTP client in each language, and a unix socket is a few lines on top of it. Erlang’s HTTP client keeps its connections to itself, so the Elixir client has two small dependencies instead (see The Elixir client). An SDK for a runtime whose point is a small, auditable boundary should not arrive with a dependency tree of its own. None is a second implementation of Zygo, and nothing in any client can widen a sandbox.

  your process ──HTTP──▶ zygo api ──control socket──▶ supervisor ──▶ warm sandboxes

On one machine the HTTP hop can be a unix socket at mode 0600, so there is no port. Across machines it is TCP. Either way a bearer token is needed, unless the API was started with auth turned off on a unix socket or loopback address.

The Python client

pip install zygo-sdk        # or: pip install -e sdk/python; then `import zygo_sdk`

The module is zygo_sdk, not zygo: a different, older project owns the name zygo on PyPI, and two packages that install the same module name overwrite each other. import zygo_sdk as zygo keeps the short name in your code.

No dependencies. zygo.connect() finds the API from its url argument, then ZYGO_API_URL, then http://127.0.0.1:7700, and its token from token= or ZYGO_API_TOKEN.

import zygo_sdk as zygo
client = zygo.connect()
res = client.fn("resize")({"url": "…"})               # a call; ~1.4 ms of overhead
print(res.result)                                      # what the handler returned
res = client.call("resize", {"url": "…"}, timeout=5)   # the same call, with a timeout
print(res.result, res.stdout, res.metrics.wall_ms)     # a Result: value, output, timings
for ev in client.stream("resize", {"url": "…"}):       # live output
    print(ev.kind, ev.data)

The client covers every route but GET /metrics, which is for a scraper: call, batch, stream, cancel, logs, stats, warm, serve, stop, run, functions; pools with serve_runtime, run_script, stream_script, put_script; put_deps, put_blob; tenants, secrets, limits and tokens; for_tenant(id); health, version, drain. What the clients can do lists every method.

The async Python client

zygo.aio is an asynchronous client, which is what an agent framework needs:

import asyncio
import zygo_sdk as zygo

async def main(events):
    async with zygo.aio.connect() as client:
        return await asyncio.gather(*(client.call("resize", e) for e in events))

asyncio.run(main(events))

It is the whole API, not a subset: every method of the plain client, with the same name and arguments, as a coroutine — tenants, tokens, secrets, limits, blobs, drain, for_tenant, and workspace/out on call and run_script included. A test holds the two surfaces equal. zygo.aio is reachable after import zygo_sdk as zygo; import zygo_sdk.aio works too. Cancelling the task cancels the request on the server.

The Node client

npm install zygo-sdk

No dependencies, Node 18 or newer, ES modules. The same methods in camelCase — runScript, serveRuntime, putSecret — with timeouts in seconds and an AbortSignal that cancels the request for you.

import { connect } from 'zygo-sdk';
const client = connect();                         // ZYGO_API_URL, ZYGO_API_TOKEN
const resize = client.fn('resize');
const out = await resize({ url: 'https://example.com/a.png' });
console.log(out.result, out.metrics.wallMs);

The TypeScript types

TypeScript types ship beside the JavaScript in index.d.ts. They are written by hand rather than compiled, so the package has no build step and what you read in the repository is what runs. A test (sdk/node/test/types.test.js) reads the declaration file against the module as it runs — every export, every Client method, every option a call reads — so a method added without a declaration fails the suite rather than shipping untyped.

The Elixir client

# mix.exs
{:zygo_sdk, "~> 0.1"}

Elixir 1.18 or newer, for the JSON module in the standard library. The same methods as Python, with the same snake_case names, as functions in the Zygo module that take the client first. Every call to the API returns {:ok, value} or {:error, %Zygo.Error{}}, and has a ! twin that returns the value or raises. The two streams yield an error as an element instead, and their ! twins raise it. Durations — timeout:, backoff:, retry_after — are milliseconds, as is usual in Elixir.

client = Zygo.connect()                                  # ZYGO_API_URL, ZYGO_API_TOKEN
{:ok, out} = Zygo.call(client, "resize", %{"url" => "…"}, timeout: 5_000)
IO.inspect({out.result, out.stdout, out.metrics.wall_ms})

There is no fn(name) handle: fn is a keyword in Elixir, and &Zygo.call(client, "resize", &1) is the same thing in one line.

Processes, and the pool

The client is a plain struct, so any process may use it. Its connections live in a pool process, which Zygo.connect/2 starts linked to the caller. A client that should live as long as your application goes under your own supervisor instead, and is fetched by name:

children = [{Zygo, name: MyApp.Zygo, url: "unix:///run/zygo/api.sock", retries: 3}]
Zygo.functions!(Zygo.client(MyApp.Zygo))

  caller process ──checkout──▶ ┌── Zygo pool ─────────────┐
    (sends, reads the answer)  │ idle connection  ·  12 s │
  ◀──────────────connection─── │ idle connection  ·   3 s │
  caller process ──checkin───▶ │ (up to pool_size: 32)    │
                               └──────────────────────────┘

A caller borrows one connection for one request and gives it back. The socket moves with it, so a caller that crashes mid-request takes its socket down rather than leaving a half-read answer in the pool. More callers than pool_size wait for a connection to come free. The pool is NimblePool, and the connections are Mint, which is HTTP as a data structure rather than a process and opens a unix socket. Those two are the package’s only dependencies.

Errors, and streams

The other clients raise one class per kind of failure. The Elixir client has one exception, Zygo.Error, and its kind field says which failure it was: :busy, :handler, :timeout and so on (see Errors in the clients). A case on the kind is how a caller branches:

case Zygo.call(client, "resize", event) do
  {:ok, out} -> out.result
  {:error, %Zygo.Error{kind: :busy, retry_after: ms}} -> {:later, ms}
  {:error, %Zygo.Error{kind: :handler, stderr: stderr}} -> {:bug, stderr}
end

Zygo.stream/4 returns a lazy Stream, and nothing is sent until it is read. Its items are {:stdout, text}, {:stderr, text} and {:progress, text}, then one last item: {:result, %Zygo.Result{}}, or {:error, %Zygo.Error{}}. A refusal before anything ran is that error alone. Zygo.stream!/4 raises the error instead, after the output before it.

What the clients can do

Who is the token the call is made with: a tenant token is one customer’s, an operator token is the host’s. Deploy marks the calls that need deploy rights (see The deploy gate).

A listing is scoped to the caller: functions() and runtimes() through a tenant token show that tenant’s names, and nobody else’s. (/metrics is scoped the same way; see Metrics, OTLP and usage events.)

Connecting

All three clients find the address in the same order: the argument, then ZYGO_API_URL, then http://127.0.0.1:7700. These forms are accepted:

unix:///run/user/1000/zygo/api.sock     a local API over a unix socket
http://127.0.0.1:7700                   the default
https://zygo.internal:8443              across a network
box:9000                                bare host and port

The token comes from ZYGO_API_TOKEN unless one is passed. That is the same variable the server reads, so a shell that can start the API can talk to it. A unix socket still needs the token, unless the API was started with --no-auth. Every client pools connections and is safe to share between threads, tasks or processes. That matters: one connection would line concurrent callers up behind a single socket, and the warm path is measured in milliseconds.

The API hangs up on a connection that has waited 30 seconds for its next request. A pooled connection older than 20 seconds is therefore dropped, not reused; and if a reused one turns out to be closed — the send fails, or the socket ends where the status line should begin — the request goes once more on a fresh connection. That is safe because nothing came back, so the request never ran. A connection that fails after the first byte of an answer, or a new connection that fails at all, is reported as a TransportError (in Elixir, kind: :transport), never retried.

Retries

Every client can resend a refused request. retries (default 0) is how many times; backoff (default 1 s) is the base wait. Only Busy and Unavailable qualify — both mean the request never ran, which is what makes sending it again safe. A handler that raised, a request the deadline killed, a missing function: sent once, never again. Each wait is the longer of the server’s Retry-After and backoff doubled per attempt (1 s, 2 s, 4 s …), and the last refusal is the error you see.

client = zygo.connect(retries=3)                 # Busy or Unavailable → wait, send again

const client = connect(undefined, { retries: 3, backoff: 0.5 });

client = Zygo.connect(retries: 3, backoff: 500)   # milliseconds

Tenants

An embedder — a product that runs its customers’ code on Zygo — has customers. A tenant is one of them, and it is what lets the API answer “whose script is this?”:

client.create_tenant("acme")                    # POST /tenants, idempotent
acme = client.for_tenant("acme")                # a view; the same connection

script = acme.put_script(source)                # registered against acme
acme.run_script("py312", script.sha256, event)  # and only acme may run it

await client.createTenant('acme');
const acme = client.forTenant('acme');

Zygo.create_tenant!(client, "acme")
acme = Zygo.for_tenant(client, "acme")         # a new struct; the same pool

Listing or creating tenants is the operator’s: a customer that could list the other customers is a leak, whatever the limits say. A tenant may read its own record, which is how a client finds out what it registered.

What a tenant gets

• Its own scripts. A digest (the SHA-256 hash that names a script) is not a key to it — anyone holding the bytes can compute one. So a tenant naming a digest it did not register is told the script does not exist. That is the same answer an unregistered digest gets, on purpose: “it exists but is not yours” is a fact about another customer.
• Its own cgroup. tenants/<tenant>/<function>/…, so everything one customer runs is in one place, can be killed in one write, and counted in one read. Limiting a tenant narrows it.
• Deletion that means it. client.delete_tenant(id) stops their functions and pools and removes the scripts nothing else refers to. It answers with both lists, because neither can be rebuilt afterwards. It also takes the tenant’s tokens.

Tokens

A tenant is only worth having if the server can tell whose request this is without being told. That is what a token is: the one part of a request the caller cannot choose.

import os
import zygo_sdk as zygo

operator = zygo.connect(url, token=os.environ["ZYGO_API_TOKEN"])
minted = operator.mint_token("acme")        # POST /tenants/acme/tokens
print(minted.secret)                        # the only time this exists

acme = zygo.connect(url, token=minted.secret)
acme.put_script(source)                     # registered against acme, no header

const minted = await operator.mintToken('acme');
const acme = connect(url, { token: minted.secret });

There are two kinds, on purpose, and no finer ladder of scopes. An operator token (mint_token(), no tenant) belongs to whoever runs this Zygo: tenants, functions, pools, and more tokens. A tenant token (mint_token(id)) belongs to one customer. It registers scripts for itself, calls the pools and functions the operator declared, reads its own record — and cannot see that any other tenant exists.

What follows from that

• The secret exists once. The server keeps only a SHA-256 of it, so client.tokens() can list every token on the host without being a way to steal one. Nothing can print a secret again: lose one, revoke it, mint another.
• X-Zygo-Tenant is the operator’s. for_tenant(id) says which of your customers you are acting for. A tenant token already names its tenant, and a header that disagrees with it is refused, not ignored.
• Revoking is immediate. The next request with a revoked token is a 401. The record stays, marked, so an id in a log line still points at something.
• Deleting a tenant takes their tokens along with the scripts only they referred to.

ZYGO_API_TOKEN is the bootstrap operator token: the same variable an existing deployment already sets, with the rights it already had. On the host, zygo token mint, zygo token ls and zygo token revoke <id> do the same three things without an HTTP round trip.

Health and draining

GET /healthz needs no token, so a load balancer can probe it. It answers one of three things:

degraded is a 200 on purpose. A host that can serve should be served to. A probe that took hosts out of rotation for being slow would take every host out at once after a restart. stopping is the one answer that is not a 200, because a balancer that keeps sending to a draining host is the reason draining fails.

Draining

client.drain(grace=30)    # {"drained": true, "in_flight": 0}

Draining stops taking new requests, lets the running ones finish, answers, and then exits. The default grace is 30 seconds. in_flight: 0 is a clean drain; anything else means the grace ran out. A deploy script needs to know which of those happened. SIGTERM does the same, so a container stop or a systemctl restart needs no call at all. The grace there is 25 seconds, chosen against the 30 that systemd and Docker give a process before SIGKILL: a drain that outlived its own kill would never finish. Chapter 16 covers running Zygo in production.

Usage, for billing

Every finished request produces one usage event, from the supervisor — whether it came over HTTP, from zygo exec at a terminal, or from an MCP tool:

{"tenant": "acme", "function": "py312", "script": "sha256:…",
 "request_id": "00000042", "wall_ms": 812.4, "cpu_ms": 740.1,
 "peak_rss_kb": 48200, "outcome": "ok", "finished_ms": 1790000000000}

outcome is one word — ok, error, timeout, cancelled, stuck — so a dashboard groups by it instead of working it out again from four true/false fields. A caller who cancelled their own request reads cancelled, even if the deadline happened to pass while the kill landed.

Three ways to collect it

The OTLP series and the webhook are counted in the memory of the zygo api process. Requests from zygo exec or from MCP never pass through it, so they appear only in the supervisor’s log. If you bill from the webhook, make sure every billable request goes through the API.

The webhook is at least once

A batch that fails goes back on the front of the queue, in order, and is retried. So a receiver may see an event twice and should key on request_id. The queue holds at most 10 000 events. When a webhook has been down long enough to fill it, the oldest events are dropped and the count is logged. The alternative is the API process growing until it takes the serving path down to protect the billing path, which is the wrong way round. The events are in the supervisor’s log either way.

Limiting a tenant

A pool is declared once by the operator and called by every customer. One customer should not be able to take the whole of it:

client.set_limits("acme", mem="256M", cpu=0.5, pids=64, timeout="30s")

The keys are mem, cpu, pids, timeout, scratch, network and allow. It is a PATCH, so the keys you pass are set and the rest are left alone. timeout lands on the supervisor’s deadline rather than the cgroup, because a cgroup cannot enforce a wall clock. The rest are cgroup settings.

Limits only narrow

A tenant’s limits are applied as the minimum of themselves and whatever the function or pool was declared with. They are written on the request’s own cgroup before the handler is let go. So the worst a wrong value can do is give a customer less than they were promised, never more. No value and no key can widen anything, which is what makes the route safe to expose.

A value above every ceiling the tenant can reach — their own functions, and every pool on the host — is refused with 422 naming the key. It could never take effect, and storing it would let you believe you had tightened something you had not. Above one ceiling and below another is fine: it narrows the larger and does nothing to the smaller. Chapter 14 explains the limits themselves.

Secrets

A function’s secrets are delivered as files at /run/secrets/<name>, written from outside the sandbox and removed when the last request in flight finishes. An operator at a terminal supplies them from their own environment. An embedder’s customers cannot: they have their own keys and nobody to restart a supervisor. So Zygo can store secrets per tenant.

export ZYGO_SECRETS_KEY=$(zygo secrets keygen)   # before the supervisor starts
zygo secrets set acme STRIPE_KEY                 # reads it with echo off

client.put_secret("acme", "STRIPE_KEY", value)   # PUT, needs deploy rights
client.secrets("acme")                           # ['STRIPE_KEY'] — names only

A stored secret fills in what the shell did not supply, per tenant, when a function is served. Where both have a value, the shell wins: zygo serve at a terminal is somebody saying what they want now. A tenant cannot set its own secrets; the operator holds that relationship.

No way to read a value back

This is not a missing route. There is no answer shape that could carry a value, because a store that answered with values would make every route that reaches it a way to read every customer’s keys. Values are sealed with ChaCha20-Poly1305 (an authenticated cipher) under a key Zygo never stores. Each is bound to its own tenant/name, so it cannot be moved to another by anything that can only rename files. A passphrase is refused rather than stretched: turning one into a key needs a password key function, and a store that accepted hunter2 and stretched it badly would be worse than one that said no.

What that protects, and what not

It protects the bytes at rest: a backup, a stray tar, anything that can read one user’s files. It does not protect them from a process that can read the supervisor’s memory, and it is not a hardware root of trust. An operator who needs those has a KMS (a key management service).

Secrets for a runtime pool

A pool’s sandbox is shared by several tenants, and /run/secrets is one directory in it — so a pool holds no values. It names them, and each call is given the calling tenant’s values from the store, for that call only:

client.serve_runtime("py312", {"image": "python:3.12-slim", "agent": "python"},
                     secrets=["STRIPE_KEY"])            # names, not values
client.put_secret("acme", "STRIPE_KEY", value)          # acme's value
client.for_tenant("acme").run_script("py312", digest)   # reads acme's, as a file

await client.serveRuntime('py312', { image: 'python:3.12-slim', agent: 'python' },
                          { secrets: ['STRIPE_KEY'] });

The script reads /run/secrets/STRIPE_KEY as a function would. While the files exist the call has its zygote to itself, so no other tenant’s child is forked beside them; if every zygote is busy the pool grows, and at max_warm the call gets busy, which the SDKs retry. A call from a tenant that has no secret under one of the names is refused with 400 before anything runs, naming the secret; a pool that names secrets on a host with no store key is refused when it is served. The serve_runtime argument is a list of names — the difference from serve, whose secrets are values. Chapter 14 has the rules.

Files in and out

A handler that converts a document needs the document, and the caller needs what comes back. Neither belongs in a JSON event. So a request can carry a workspace: a tar archive (one file that packs many files) that Zygo unpacks into the request’s own directory. With out, the directory comes back as a tar with the answer.

  caller                                  one request's sandbox
  ──────                                  ─────────────────────
  tar of files ──inline or blob──▶ unpacked into a fresh directory under /work
                                   the handler starts in it: reads in.pdf,
                                   writes out.png
  answer + tar ◀──── out=1 ──────  the directory, packed on the way out
                                   then removed, whatever happened

tar = make_tar({"in.pdf": pdf_bytes})

out = client.run_script("convert", script, {"to": "png"},
                        workspace={"inline": base64.b64encode(tar).decode()},
                        out=True)

open("result.tar", "wb").write(out.workspace)   # already decoded

const out = await client.runScript('convert', script, { to: 'png' }, {
  workspace: { inline: tar.toString('base64') },
  out: true,
});

Inside the handler

The handler is started in its own directory and told where it is:

def handler(event):
    with open("in.pdf", "rb") as f:        # the caller's files are just here
        ...
    open("out.png", "wb").write(rendered)  # and this comes back with `out=1`
    return {"pages": 3}

Send a fixture once: blobs

An embedder often sends the same fixture across a thousand calls. Store it once as a blob and name it by digest:

blob = client.put_blob(tar)                       # PUT /blobs, idempotent
client.run_script("convert", script, event, workspace={"blob": blob.sha256})

A warm function’s body is the event itself, with nowhere to put a workspace. So there it goes in the query string: client.call(name, event, workspace=blob.sha256, out=True), which is ?workspace=sha256:…&out=1. It takes a blob only, because an inline tar in a URL would be a megabyte of base64 in a request line. The async Python client takes both as well.

What keeps one request’s files from another’s

Not a mount namespace, and the reason was measured rather than assumed. A forked child runs at an unprivileged uid with no CAP_SYS_ADMIN in the sandbox’s user namespace, so unshare(CLONE_NEWNS) fails with EPERM. That holds even with the most permissive seccomp profile, so it is the namespace and not a filter. One path cannot mean a different directory to each request. What is there instead is listed plainly, because the first two are weaker than a namespace would be:

• /work cannot be listed (mode 0311), so a request cannot see its neighbours’ names. The test suite checks this by trying.
• The directory’s name is 128 random bits, not the request id, which is a counter.
• It is removed when the request ends, whatever the request did, so the window is one request long. This is also checked.

Archives are unpacked, not trusted

Every archive is unpacked by Zygo under strict rules. Only files and directories are allowed: no symlinks, no hard links, because that is how an archive writes outside the directory it was unpacked into. No .., no absolute paths. File modes are Zygo’s, not the archive’s. Entries and bytes are capped, and counted as they are written, not read from a header an archive is free to lie in.

The way out is careful too. ?out=1 packs what the handler left, and the handler may leave a process running that keeps changing it. So Zygo never follows a path into the folder. It opens each folder it has already opened, one name at a time, and never follows a link. A name that has become a link, a pipe or a socket since the folder was read is skipped. A workspace folder swapped for a link is not packed at all. Only files and folders come back, and never more bytes than the cap.

Watching a request

A call that takes a minute has something to say before it finishes:

for event in client.stream("render", {"pages": 400}):
    if event.is_result:
        print(event.result.result)
    else:
        print(event.kind, event.data, end="")      # stdout, stderr, progress

for await (const event of client.stream('render', { pages: 400 })) {
  if (event.kind === 'result') console.log(event.result.result);
  else process.stdout.write(event.data);
}

Each item is a piece of the request’s output, and the last one is the result: exactly what the non-streaming call would have returned, or raised. A handler that printed and then failed produced both. So the output is delivered first, and the exception comes when you iterate past the result.

Three kinds of item

stdout and stderr are what the request’s process wrote, kept apart as everywhere else. progress is its own kind, not a line of stdout. A long request has two things to say — what it printed, and how far it has got — and a caller should not have to parse a handler’s log messages to find the second. The handler calls event.progress(...). It exists whether or not anybody is listening, so a handler does not break depending on who called it:

def handler(event):
    for n, page in enumerate(event["pages"]):
        event.progress(f"{n} of {len(event['pages'])}")
    return {"done": True}

The result still carries the whole of stdout and stderr, bounded as always. A caller that streamed and one that did not see the same text; streaming only changes when.

On the wire

  POST /fn/render?stream=1          Content-Type: application/x-ndjson
  ──────────────────────────────────────────────────────────────────────────
  {"stream": "stdout",   "data": "loading\n"}
  {"stream": "progress", "data": "1 of 400"}
  {"stream": "stderr",   "data": "warning: …\n"}
  …
  {"status": 200, "result": {…}, "request_id": "…", "stdout": "…", …}   ◀── last line

Streaming is per request, not per function. Sending each print() as it happens costs a system call per print() on a path measured in milliseconds. So a caller that wants to watch pays for it, and everybody else keeps the fast shape. The answer is newline-delimited JSON (one JSON object per line) rather than server-sent events: every language can read a line and parse JSON. The streaming connection is held for the whole request and is not pooled. Abandoning the iterator closes it, which does not cancel the request — pass a key and use cancel for that.

Cancelling a request

A request that is running can be stopped:

out = client.call("render", event, key="job-4711")   # name it on the way in
...
client.cancel("job-4711")                            # from anywhere

const controller = new AbortController();
const call = client.call('render', event, { signal: controller.signal });
controller.abort();                                  // cancels it on the server too

Asynchronous Python needs no key at all:

task = asyncio.ensure_future(client.call("render", event))
task.cancel()          # sends the cancel before CancelledError propagates

The caller of the cancelled request gets Cancelled (HTTP 499), which is on purpose not Timeout. A timeout says the work is too slow or the limit too tight; a cancel says the answer stopped being wanted. Both arrive as exit 137 from the kernel, and only the side that sent the signal can tell them apart, so the supervisor records which it was.

  caller A ── POST /fn/render  (X-Zygo-Request-Key: job-4711) ──▶ running …
  caller B ── DELETE /requests/job-4711 ──▶ supervisor
                                              └──▶ writes cgroup.kill on that
                                                   request's own cgroup
  caller A ◀── 499 Cancelled

A key, not the request id

The id is assigned by the host and arrives with the answer, which is too late to stop the call it belongs to. X-Zygo-Request-Id comes back on every response and in the body as request_id; it joins a log line to its request. The key is what a caller uses to name a request it is still waiting for. Reusing a key is allowed, and one cancel then stops every call under it.

What actually stops the work

The supervisor writes cgroup.kill on the request’s own cgroup, from outside the sandbox. That reaches everything the handler started, and does not depend on the tenant’s code being in a state where a signal helps. The agent inside is told, so it can mark the answer, but an agent that ignores the message changes nothing. A cancel that arrives before the request was let go is the best case: nothing of the handler has run, and started: false in the answer says so. Cancelling something already finished, or another tenant’s request, is the same NotFound. Request ids are a counter, not a secret, so ownership is what keeps a cancel honest.

Runtime pools

A warm function is one script in one zygote (the parked, ready process that Zygo forks per request; see chapter 13). A runtime pool is the other shape: an image, a dependency set and an agent, with no code in it. The script arrives with the call.

  WARM FUNCTION                          RUNTIME POOL
  ┌───────────────────────────┐          ┌───────────────────────────┐
  │ image + deps + agent      │          │ image + deps + agent      │
  │ + one handler, imported   │          │ no code                   │
  └───────────────────────────┘          └───────────────────────────┘
  event ──▶ handler(event)               script digest + event ──▶ the forked
                                         child loads that script, runs it

client.serve_runtime("py312", {                 # POST /runtimes
    "image": "python:3.12-slim",
    "agent": "python",
    "min_warm": 2, "max_warm": 8,
    "mem": "512M", "timeout": "60s",
})

script = client.put_script(source)              # once, PUT /scripts
out = client.run_script("py312", script.sha256, {"month": "2026-09"})
print(out.result, out.metrics.wall_ms)

await client.serveRuntime('py312', { image: 'python:3.12-slim', agent: 'python' });
const { sha256 } = await client.putScript(source);
const out = await client.runScript('py312', sha256, { month: '2026-09' });

Zygo.serve_runtime!(client, "py312", %{"image" => "python:3.12-slim", "agent" => "python"})
%Zygo.Script{sha256: sha256} = Zygo.put_script!(client, source)
out = Zygo.run_script!(client, "py312", sha256, %{"month" => "2026-09"})

How a script gets in

client.runtimes() lists the pools with their zygote counts, and client.stop_runtime(name) stops one (so does zygo stop <name>). A pool can name secrets; each call then gets the calling tenant’s values — see Secrets for a runtime pool. run_script also takes the source directly, for a one-off not worth registering. The request never reaches the zygote: the supervisor writes the script into the sandbox, and the forked child loads it after the child’s seccomp filter (its list of allowed system calls) is installed. That is what makes it safe for two tenants to share a pool. It is checked, not just claimed: tests/linux/verify_api.sh asks a script where it was loaded from and whether it can list what else is in flight.

A pool with no agent

A pool usually holds an agent — an interpreter, warm, forking per request. It does not have to. Give it a cmd and no agent and you get the warm-exec shape. Zygo holds the sandbox, writes each request’s script into it, and runs cmd with that path as its last argument and the event on stdin.

client.serve_runtime("sh", {"image": "alpine:3", "cmd": ["/bin/sh"]})
script = client.put_script(open("wordcount.sh").read())
out = client.run_script("sh", script.sha256, {"text": "warm exec in sh"})

That runs sh /run/script/<digest> inside the sandbox. It is the right shape for bash, for a static binary that takes a script as an argument, and for any language that starts in under a millisecond. There is nothing for an agent to save there, and the warm protocol would only be one more moving part.

What warm-exec does not have

There is no protocol to carry them, so warm-exec has no streaming, no progress(), no workspaces, and no per-tenant limits narrowed per request. The sandbox itself is the same: same namespaces, same seccomp profile, same cgroup per request, same deadline. examples/warm-exec/ has both shapes side by side.

Dependency sets

A pool’s requirements names a file on the Zygo host, which is the one thing an embedder does not have. POST /deps is the other half: send the lock file itself, and get back an id a pool can be built on.

deps = client.put_deps("python:3.12-slim", {        # POST /deps
    "requirements.txt": open("requirements.txt").read(),
})
deps.id        # 'deps_3f1c…' — its name from now on
deps.state     # 'building'

while client.deps(deps.id).building:                # GET /deps/<id>
    time.sleep(2)

client.serve_runtime(
    "py312",
    {"image": "python:3.12-slim", "agent": "python"},
    deps=deps.id,
)

const deps = await client.putDeps('node:22-slim', {
  'package.json': manifest,
  'package-lock.json': lockfile,      // `npm ci` needs it, so this does too
});
await client.serveRuntime('node22', { image: 'node:22-slim', agent: 'node' },
                          { deps: deps.id });

Python takes a requirements.txt and gets a venv (a private folder of installed packages). Node takes a package.json and its lock file and gets npm ci. Either way the result is mounted read-only at /venv and the environment points at it — PATH for Python, NODE_PATH for Node — so a script just imports what the lock file named. Chapter 15 covers dependencies in general.

  POST /deps ──▶ 202 building ──┬──▶ ready ──▶ serve_runtime(…, deps=id)
                                │              └─▶ mounted read-only at /venv
                                └──▶ failed ──▶ deps(id).log has the reason

Five things worth knowing

• It answers before the build finishes. A pip install takes minutes, and an HTTP request that waited would time out in every proxy on the way. Poll deps(id), or send the serve_runtime and retry the Unavailable it raises, which carries the host’s Retry-After as retry_after — a client opened with retries= does that itself.
• A pool named against a build still running is refused, not queued. Nothing is started. A zygote warmed without the dependencies it was promised would serve requests that fail at import.
• The id is a hash of the files and the image. The same lock file on the same image is one build however many customers send it, and all of them see it in deps(). A different image is a different id, because a wheel built for one interpreter fails in another.
• The build reaches the package registries and nothing else. Installing a package runs its code — a setup.py, an npm lifecycle script — and the lock file came from whoever holds a token. The build sandbox has network = "egress" with only the registries allowed. A host that cannot enforce that (no passt, no nftables) refuses the build rather than falling back to host networking.
• A failed build keeps its log, on the same object as the state: client.deps(id).log is the resolver’s own words.

client.delete_deps(id) forgets one, and is refused while a pool is built on it. The pool holds a read-only mount of that directory, and removing it under a warm zygote would make its imports fail one at a time.

The script store

An embedder’s scripts live in the embedder’s database, not on the Zygo host. PUT /scripts is how one gets to a sandbox without a file on the host or a line in sandbox.toml:

script = client.put_script(source)      # PUT /scripts, body is the script
script.sha256                           # 'sha256:71e2b5…' — its name from now on
script.existed                          # the store already had exactly these bytes

The body is the script itself, not JSON around it: a script is a file, and wrapping its bytes only to unwrap them again helps nobody. The name is the SHA-256 of those bytes. That makes the call idempotent (safe to repeat) in the strongest sense: the same script from two tenants is one file on disk, and neither can put different bytes under a digest the other is running.

Looking up and forgetting

client.script(digest) says whether the host holds it and how big it is. It never returns the bytes: a digest is not a key, so a store that answered with the script would make every tenant’s code readable by anyone who could guess it. client.delete_script(digest) forgets one, and needs deploy rights, because forgetting it forgets it for every tenant that sent the same bytes.

Why PUT /scripts is not gated

Registering a script does not make it runnable on its own. A request still has to name something to run it in, which is a runtime pool the operator declared. So any authenticated caller may register a script, tenant tokens included. A tenant registering its own code runs nothing by doing so, and the digest is theirs from then on.

Errors in the clients

Each kind of failure is its own type, because each one means something different about what to do next.

Unavailable is the 503: the host cannot do this yet, and nothing about the request needs changing. Unlike Busy it is not backpressure — the wait is for something the host is doing. health() raises it once the API is stopping. A 422 (a limit above every ceiling) is still a plain ZygoError. Every error type is a subclass of ZygoError, so catching it catches everything. In Elixir every one is a Zygo.Error, returned as {:error, _} or raised by the ! functions, with the same fields as the class it stands for — retry_after, code, stdout, stderr, exit_code, request_id.

Busy or HandlerError: the one that matters

Busy means the request was refused before anything happened, so retrying is correct. A handler that raised will raise again. Timeout is not a guess either. The supervisor records that it killed the request, because a deadline kill and an out-of-memory kill both arrive as exit 137. In all, exit 137 has four readings — a deadline, an out-of-memory kill, a cancel, and a sandbox that went quiet — and the supervisor is the only side that knows which, so it says.

Long requests

A function’s timeout may be hours, and a caller may wait up to a day (X-Zygo-Timeout-Ms, at most 24 hours; the default wait is 60 s). Two things make that safe rather than a way to hold a slot for ever:

• A heartbeat. The agent says every second or two that a request is still alive. A request the supervisor has heard nothing about for a minute is killed and raises Stuck, whatever its budget said. So a wedged request costs a minute, not its whole timeout, and “your code is slow” stays different from “the sandbox stopped answering”.
• The idle policy leaves working zygotes alone. A zygote nobody has called for idle_timeout is frozen. One with a request in flight is not, whatever the clock says, because freezing it would stop the request. A request that runs for an hour under a two-second idle_timeout still finishes.

A batch is the exception to raising: each element is a result or an error, returned rather than raised, because one refused event must not hide the answers to the others.

A worked example

examples/plugin-host/ is a plugin host in about a hundred lines, built on this API alone: no sandbox.toml, no file on the Zygo machine, no shelling out to zygo. It onboards customers and gives each their own token, limits and secrets. It declares one runtime they all share, installs their code by digest, runs it with files in and out, streams the long ones, stops them, and offboards. make verify-plugin-host runs it against a real kernel. That includes the two checks this whole layer exists for: one customer cannot run another’s plugin by naming its digest, and a customer is held to their own memory limit rather than the runtime’s.

The same host from Node is examples/plugin-host-node/: a node:http server that onboards customers with an operator token, registers and runs their scripts through forTenant, streams one route as NDJSON, and turns HandlerError, Timeout, Cancelled, NotFound, Busy and Unavailable into its own status codes. node --test runs it against a fake Zygo; make verify-plugin-host-node against a real one.

The OpenAPI document