11. Getting started

This chapter takes you from nothing to three warm functions behind an HTTP API. You install Zygo, ask it whether your machine can run sandboxes, run one program in a sandbox, and then keep a function warm and call it. Each step is short, and each one links to the chapter that explains it in full.

The road through this chapter

  install ──▶ zygo doctor ──▶ zygo run ──▶ zygo serve ──▶ sandbox.toml ──▶ zygo api
  (a file)    (can this       (one        (one warm      (three          (call them
               host do it?)    sandbox)    function)      functions)       over HTTP)

You can stop after any step. A CI job that only needs zygo run never has to learn about warm functions, and a Mac user can do all of it without setting up Linux by hand.

The five-minute version

If you would rather type first and read later, this is the whole road in one block. Every line is explained further down the page.

# install: Linux, x86_64 or aarch64 (a Mac: brew install mhmtskrc2/zygo/zygo)
url=https://github.com/mhmtskrc2/zygo/releases/latest/download
curl -fsSLO "$url/zygo-$(uname -m)-unknown-linux-musl.tar.gz"
curl -fsSL "$url/SHA256SUMS" | sha256sum -c --ignore-missing
tar xzf zygo-*-unknown-linux-musl.tar.gz && sudo install -m 0755 zygo-*/zygo /usr/local/bin/

zygo doctor                                   # can this host run sandboxes? --fix if not
zygo run python:3.12-slim python3 -c 'print("hello")'   # one sandbox, thrown away

echo 'def handler(event): return {"got": event}' > handler.py
zygo serve ./handler.py --name echo           # warm it once: about 150 ms
zygo exec echo '{"n": 1}'                     # a fresh fork of it: about 1.4 ms
zygo stop echo

If zygo doctor reports something red, its section below says what each line means, and zygo doctor --fix applies the usual fixes for you.

What you need

Zygo is one static binary: a single file that carries everything it needs, with no libraries or runtimes to install beside it. It builds sandboxes out of Linux kernel features, so the sandboxes themselves always run on Linux.

On Linux you also need unprivileged user namespaces (a normal user may make the private views from chapter 2) and cgroup v2 delegation (a normal user may own part of the cgroup tree from chapter 3). Most modern distributions have the first; the second often needs one small fix, which zygo doctor can apply for you.

Installing on Linux

Release builds exist for x86_64 and aarch64 (64-bit ARM, such as a Raspberry Pi 5 or a Graviton server). Both are static musl builds: musl is a small C library that is linked into the binary, so it does not depend on the host’s own libraries.

url=https://github.com/mhmtskrc2/zygo/releases/latest/download
curl -fsSLO "$url/zygo-$(uname -m)-unknown-linux-musl.tar.gz"
curl -fsSL "$url/SHA256SUMS" | sha256sum -c --ignore-missing   # prints "OK"
tar xzf zygo-*-unknown-linux-musl.tar.gz
sudo install -m 0755 zygo-*/zygo /usr/local/bin/zygo

uname -m prints x86_64 or aarch64, which picks the right file. The second line checks the archive against the SHA256SUMS file that every release carries, so you know it is the one that was published. The sudo is only for copying the file into /usr/local/bin; Zygo itself never runs as root.

SHA256SUMS is itself signed, in every release after 0.1.1. With cosign installed, this proves the checksums came from this project’s release workflow, not only from the same download page:

curl -fsSLO "$url/SHA256SUMS"
curl -fsSLO "$url/SHA256SUMS.sigstore.json"
cosign verify-blob SHA256SUMS --bundle SHA256SUMS.sigstore.json \
  --certificate-identity-regexp '^https://github\.com/.*/\.github/workflows/release\.yml@refs/tags/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Each release also carries an SBOM — a list of every library compiled into the binary, with its version — as zygo-<version>.cdx.json, in the CycloneDX format that security scanners read.

Installing on macOS

On a Mac, Homebrew installs three things: a small zygo shim for macOS (a thin program that passes your commands on), the Linux build of Zygo that the shim forwards into, and Lima, the tool that starts the Linux VM.

brew install mhmtskrc2/zygo/zygo

The formula lives in a tap (a small GitHub repository of Homebrew formulas), mhmtskrc2/homebrew-zygo; Homebrew finds it from the name. The release build writes the formula there, with checksums computed from the same archives, not typed in by hand. The section How Zygo runs on a Mac below explains what the VM is and what it costs.

Other ways to install

# from crates.io, anywhere with a Rust toolchain
cargo install zygo-cli

# from a checkout of the repository
cargo build --release
sudo install -m 0755 target/release/zygo /usr/local/bin/zygo

cargo install zygo-cli builds the default set of features. The vm backend is not in it. That backend links a virtual machine monitor from a pinned git tag, and a crate published on crates.io may not point at a git tag. So make vm-build from a checkout is the way to get it, and zygo doctor tells you this on a host where vm would otherwise work. Chapter 16 explains the backends.

The container image

Zygo also ships as a container image, ghcr.io/mhmtskrc2/zygo. It is Alpine Linux plus the static zygo binary and the few helpers networking needs. It does not need --privileged, but it does need a few specific things from Docker, because Zygo builds sandboxes inside it:

docker run --user 0:0 --security-opt seccomp=unconfined \
    --security-opt systempaths=unconfined --security-opt apparmor=unconfined \
    --cgroupns=host --cgroup-parent=/zygo -v /sys/fs/cgroup/zygo:/sys/fs/cgroup/zygo:rw \
    -p 7700:7700 -e ZYGO_API_TOKEN=... ghcr.io/mhmtskrc2/zygo

Sandboxes with a network need --device /dev/net/tun as well, and a volume keeps pulled images across restarts; chapter 16 has the full command. zygo doctor names anything that is missing, inside the container just as on a host. packaging/oci/ has the details, and chapter 16 covers running it in production.

On Windows, through WSL2

WSL2 is a real Linux kernel in a small VM that Windows manages, so Zygo runs there as on any Linux host. The project has not tested it yet — no CI runner or development machine here is Windows — so treat this as the route that should work, and tell us if it does not. Three settings matter:

1. A distribution with systemd. In Ubuntu 24.04 under WSL2, put this in /etc/wsl.conf, then run wsl --shutdown from Windows:

   [boot]
   systemd=true
   

   systemd is what gives your user a delegated cgroup, as on a desktop Linux.

2. cgroup v2 only. WSL2 can mount the old cgroup v1 hierarchy beside v2, and Zygo needs v2 on its own. In %UserProfile%\.wslconfig on the Windows side, then wsl --shutdown again:

   [wsl2]
   kernelCommandLine = cgroup_no_v1=all
   

3. Install as on Linux, with the commands above, then run zygo doctor. It checks the kernel version, user namespaces and cgroup delegation there, and zygo doctor --fix applies what it can.

WSL2’s kernel is 5.15 or newer, well above Zygo’s 5.3 floor; below 6.1 doctor calls it degraded, as on any host, and it reports whether the kernel was built with Landlock. There is no /dev/kvm by default, so the vm backend is off. Troubleshooting covers what doctor reports.

In a dev container or a Codespace

The repository has a .devcontainer for VS Code’s Reopen in Container and for GitHub Codespaces. It builds Zygo, installs the tools the tests and networked sandboxes need, and runs zygo doctor. Sandboxes work inside it: one-shot runs, warm functions and egress networking were checked in it.

It needs two things a default container does not give. The container is started --privileged, so Zygo may create user namespaces and mount inside them. And a small script arranges the container’s own cgroup tree at every start, because cgroup v2 hands controllers only to a cgroup with no processes in it. zygo in that container is a wrapper that starts the real binary in the empty cgroup the script left for it.

  /sys/fs/cgroup   (the container's own root: memory, pids, cpu for children)
  ├── init/        every process the container started, and each new shell
  └── launch/      empty until zygo starts in it
      └── zygo.slice/ …  the sandboxes, as on a host

What does not work there: the vm backend (no /dev/kvm), and on a host whose kernel is older than 5.13, Landlock; zygo doctor names both. It is a development machine, not a production shape — chapter 16 covers running Zygo in a container for real, without --privileged.

How Zygo runs on a Mac

macOS has no namespaces or cgroups, so a Mac cannot build a Linux sandbox itself. Zygo solves this by keeping a small Linux virtual machine (VM): a whole computer simulated in software, with its own Linux kernel. You never log into it. The zygo command on your Mac passes every sandbox command to a Linux zygo inside the VM, with the same arguments, the same working folder and the same input and output. The exit status comes back out to your shell.

  your Mac                                     Lima VM "zygo" (Ubuntu 24.04)
  ┌───────────────────────────────┐            ┌─────────────────────────────────┐
  │ $ zygo run python:3.12 ...    │   SSH      │ zygo run python:3.12 ...        │
  │   zygo (macOS shim) ──────────┼───────────▶│   └─▶ sandbox (namespaces,      │
  │                               │  ~22 ms    │        cgroup, seccomp ...)     │
  │ output + exit status ◀────────┼────────────┼── output + exit status          │
  │                               │            │                                 │
  │ /Users/you/project  ◀─────────┼── same ────┼─▶ /Users/you/project            │
  │                               │   path     │   (your home, mounted inside)   │
  └───────────────────────────────┘            └─────────────────────────────────┘

The Mac VM in detail

Your home folder is mounted inside the VM at the same path, and it is writable. So ./handler.py is one file, seen from two sides. That is also the limit, and Zygo enforces it: every host path must be under $HOME. A command run from outside your home folder is refused, and the message names both folders instead of quietly running somewhere else.

A few commands never need Linux, so they run on the Mac itself: zygo completion, zygo doctor, zygo agent test and zygo api --openapi. If the VM cannot be reached, a command exits with status 111; chapter 22 says what to do.

What the Mac VM costs

A one-shot zygo run python:3.12-slim true typed in a Mac shell takes about 29 ms end to end, when a supervisor is running in the VM. About 6 ms of that is the sandbox, and about 22 ms is the trip into the VM. true starts no interpreter; the 12 ms quoted elsewhere for a one-shot run is python3 -c pass, with Python’s own start inside it. These are medians of nine runs on the Mac that chapter 25 names. The same run through Docker Desktop on the same Mac took 397 ms.

Why does a one-shot run care about a supervisor? For its cgroup. Without one, zygo run on a systemd login must make a transient scope of its own and re-execute itself inside it, which costs about 12 ms more (25.7 against 13.5 ms inside the Lima VM, chapter 25). With a supervisor running, the run is handed to its cgroup tree instead.

The trip is paid per command, not per request. The millisecond warm path is still there when you call functions through the HTTP API or the SDKs, because then the calls happen inside the VM and the connection pays the trip once. zygo api running inside the VM is the answer for anything that must be fast on a Mac.

The Mac VM from a checkout

Without Homebrew you need the same two parts: Lima, and a Linux build of Zygo to put inside the VM.

brew install lima            # what starts the VM
make guest-build             # the Linux build that runs inside it, compiled in the VM

make guest-build needs nothing but the VM. It installs a Rust toolchain inside the VM on first use. On a Mac that already has Docker, make tests/linux/bin/zygo-linux-musl builds the same binary in a Docker container instead.

Checking the host: zygo doctor

zygo doctor is the first thing to run on any new machine. It checks everything a sandbox needs, prints one line per check, and prints the fix for anything that is missing. It tries each thing rather than reading a setting. For example, it really builds a user namespace and mounts inside it. So a kernel that has a feature switched on but ignores it is caught here, not later inside a sandbox.

zygo doctor           # can this host run sandboxes?
zygo doctor --json    # the same report, for a script or a health check

On a Mac, doctor checks both sides: the Mac’s side (can it reach the VM?) and the VM’s kernel. You get one report with one verdict.

What doctor checks

The last lines of the report list the backends this host can use right now: the host has what each one needs and this build of Zygo includes it.

Reading the report

Each line ends in one of four words. The exit status of zygo doctor is 0 exactly when no line says FAIL, and the JSON field ok says the same thing.

network = "none", the default, needs no networking tools at all. So a missing pasta is absent, not FAIL, and only matters when you turn a sandbox’s network on.

Which kernel version gives what

Zygo needs kernel 5.3 or newer. Later kernels unlock extra features rather than block Zygo. 6.1 or newer is recommended, because from there cgroup.kill and memory.peak are both present. Landlock’s network rules come later, in 6.7: below it, bind and connect are limited by the network namespace and the firewall alone.

  5.3 ─────────── 5.11 ─────────── 5.13 ─────────── 6.1 ─────────── 6.7 ─────────▶
  minimum         overlayfs in a   Landlock         recommended:    Landlock
  to run          user namespace   (file access     cgroup.kill,    network rules
                  (below: layers   rules)           memory.peak
                  are flattened,
                  more disk, slower
                  first run)

Letting doctor fix things: --fix

zygo doctor --fix applies the fixes it knows. It first prints a plan: each change, why it is needed, what it costs, and the exact commands. Then it asks apply these? [y/N]. Nothing happens unless you type y. In a script, where nobody can answer, add --yes.

zygo doctor --fix           # show the plan, ask, then apply
zygo doctor --fix --yes     # the same, without asking

Ubuntu and Debian: two AppArmor rules

AppArmor is a Linux security module that limits what programs may do. Ubuntu and Debian ship two AppArmor rules that get in Zygo’s way. Both come from the distribution, not from Zygo.

The first is kernel.apparmor_restrict_unprivileged_userns=1. It lets a normal process create a user namespace, and then refuses the first mount inside it. That mount is the first thing every sandbox does. zygo doctor finds this by trying the mount, and zygo doctor --fix installs an AppArmor profile that lets the zygo binary alone past it. The restriction stays on for every other program. Chapter 23 says what the rule protects.

The second is an AppArmor profile for pasta, the program Zygo uses to give a sandbox a network. Where that profile is enforced, it keeps pasta out of the sandbox’s user namespace. Then network = "egress" and "full" cannot start, even though /dev/net/tun works. The error says so and names aa-complain. The default, network = "none", does not use pasta at all.

cgroup delegation, the usual gap

On a machine with systemd, delegation is the most common missing piece, and it has two halves. First, your user manager (the systemd process that looks after your login) has to pass the controllers down to you. This is the fix doctor --fix writes, and you can also do it by hand:

mkdir -p ~/.config/systemd/user/user@.service.d
printf '[Service]\nDelegate=cpu cpuset io memory pids\n' \
    > ~/.config/systemd/user/user@.service.d/delegate.conf
systemctl --user daemon-reexec

Second, the process that runs Zygo has to sit in a delegated cgroup, and a login shell over SSH does not. Zygo handles this half itself, the way Podman does. A command that builds a sandbox restarts itself inside a temporary systemd scope of its own, which is a delegated cgroup made for one command. Chapter 12 has the details.

  your SSH login shell's cgroup      (not delegated: cannot hold a sandbox)
     │
     └─ zygo run ...  ──re-exec──▶  systemd-run --user --scope -p Delegate=yes
                                        └─ zygo run ...   (delegated: builds the sandbox)

One result surprises people. zygo doctor run from a plain SSH session can report cgroup v2 … FAIL, because that is true of the cgroup it stands in. zygo run from the same shell still works, because it moves into a scope first. Chapter 22 has the rest.

Your first one-shot sandbox

A one-shot sandbox is built for one program and thrown away when the program ends. This is zygo run, and it reads like docker run:

zygo run python:3.12-slim python3 -c 'print("hello")'

The first run downloads the image, just as docker run does. After that, a run takes about 12 ms (median, with the image already pulled, measured on the hosts in chapter 25). The sandbox has a read-only root, no network, no capabilities, and memory, CPU and process limits. A few more to try:

zygo run --mem 128M --pids 16 --timeout 10s alpine:3 /bin/sh   # tighter limits
zygo run --tty alpine:3 /bin/sh                                # with a terminal of its own
zygo run --dry-run --json python:3.12-slim                     # the plan, without running it

Chapter 12 covers zygo run in full: mounts, environment, exit codes and more.

Your first warm function

A warm function is a sandbox that is built once and kept ready, so each request skips the start-up. You write a handler: a function that takes one event and returns a result.

mkdir demo && cd demo
cat > handler.py <<'EOF'
def handler(event):
    return {"doubled": event.get("n", 0) * 2}
EOF

zygo serve handler.py --name double      # python:3.12-slim is in the store from the run above
zygo exec double '{"n": 21}'             # {"doubled": 42}
zygo ps                                  # what is running

serve starts a supervisor in your session if none is running: the background process that keeps warm functions alive. Then it warms a zygote: the Python interpreter with handler.py already imported, waiting. Every exec is a fork() of it, which means a copy made in a moment. So there is no import cost per request and no state left over between requests. Chapter 6 explains the idea.

What the warm function costs

  zygo serve                         zygo exec          zygo exec          zygo exec
  ├─ build sandbox                   ├─ fork            ├─ fork            ├─ fork
  ├─ start Python, import handler    └─ reply           └─ reply           └─ reply
  └─ park the zygote                   ~1.4 ms            ~1.4 ms            ~1.4 ms
     ~150 ms, paid once

On a Raspberry Pi 5, a Python handler that imports nothing is warm in about 150 ms, including starting the supervisor. The median overhead per request is about 1.4 ms, measured on a Lima VM on an Apple M1 Max. Both numbers and their machines are in chapter 25. The same trick works from a one-line file:

echo 'def handler(event): return {"got": event}' > handler.py
zygo serve ./handler.py --name echo
zygo exec echo '{"n": 1}'
zygo bench all            # every published number, measured again on this host

A project with three functions

A real project describes its functions in a file called sandbox.toml, so you do not repeat flags. Here are three functions that each show a different shape: a Python function with system packages, a Go binary, and a function that may call one outside API.

# sandbox.toml
[defaults]
image = "python:3.12-slim"
mem   = "256M"

[fn.resize]
entry        = "./resize.py"
requirements = "./requirements.txt"    # a venv, built once inside the image
system       = ["libwebp7"]            # apt packages, once, as a layer
mem          = "512M"

[fn.parse]
image  = "alpine:3"                     # no runtime → warm-exec
mounts = ["./bin/parse:/app/parse:ro"]  # a static Go binary
cmd    = ["/app/parse"]

[fn.fetch]
entry   = "./fetch.py"
network = "egress"
allow   = ["api.example.com:443", "*.cdn.example.com:443"]
secrets = ["API_KEY"]

[defaults] applies to every function; each [fn.NAME] table adds to it or overrides it. Chapter 20 lists every field.

What each function in the file does

  sandbox.toml
     │
     ├─ [fn.resize] ──▶ python:3.12-slim + libwebp7 layer + /venv ──▶ zygote: fork per request
     ├─ [fn.parse]  ──▶ alpine:3 + your Go binary, mounted ─────────▶ sandbox: exec per request
     └─ [fn.fetch]  ──▶ python:3.12-slim, network: only the allow list,
                        /run/secrets/API_KEY per request ───────────▶ zygote: fork per request

Chapter 13 covers the shapes, chapter 14 the network and secrets, and chapter 15 the venv and the apt layer.

Bringing the project up

zygo pull alpine:3                     # up never pulls; python:3.12-slim is already there
export API_KEY=…                       # read from this shell, delivered as a file
zygo up                                # every [fn.*] warm
zygo exec fetch '{"path": "/v1/ping"}'
zygo logs fetch -f                     # follow the function's log
zygo down                              # stop them all

zygo up warms every function in the file. It never pulls an image — a deploy should not quietly depend on a registry — so pull first, as above; zygo run is the one command that pulls for you (chapter 15). Run up again after an edit, and only the functions that changed are restarted.

Calling the functions over HTTP

zygo api puts an HTTP server in front of the functions. By default it listens on 127.0.0.1:7700, and every caller needs a bearer token: a secret string sent in the Authorization header.

export ZYGO_API_TOKEN=$(openssl rand -hex 16)
zygo api                               # 127.0.0.1:7700

curl -H "Authorization: Bearer $ZYGO_API_TOKEN" \
     -d '{"n": 4}' http://127.0.0.1:7700/fn/double

The token is needed on every kind of listener, unix sockets included. --no-auth turns it off, and Zygo refuses --no-auth anywhere except a unix socket or a loopback address such as 127.0.0.1.

  curl / your app ──HTTP──▶ zygo api ──▶ supervisor ──▶ double  (zygote)
      Authorization:        127.0.0.1:7700              resize  (zygote)
      Bearer <token>                                    parse   (warm sandbox)
                                                        fetch   (zygote)

What the API answers

Chapter 17 has every route, status code and header, and the Python, Node and Elixir clients.

Sending metrics to OpenTelemetry

Instead of waiting to be scraped at /metrics, Zygo can push the same numbers to an OpenTelemetry collector. OpenTelemetry (OTel) is a common standard for sending metrics and traces to monitoring tools.

zygo api --otlp-endpoint http://localhost:4318

OTEL_EXPORTER_OTLP_ENDPOINT does the same as the flag. The numbers go as OTLP/HTTP JSON once a minute, and OTEL_EXPORTER_OTLP_HEADERS adds a header, for example for authentication. Only metrics are sent. There are no per-request traces (spans) yet.

When something is wrong

zygo logs fetch --failed                    # the requests that failed, with their stderr
zygo shell fetch                            # a shell inside the warm sandbox
zygo spec explain fetch                     # every limit and mount, fully resolved
zygo run --dry-run --json python:3.12-slim  # the mount plan, without running it
zygo doctor                                 # is the host still able to run sandboxes?

Chapter 22 lists the errors people actually meet, with the fix for each.

Where to go next

The examples/ folder has complete projects: a small web API, a webhook, a CI job, an LLM tool, a Go program and a shell script as warm-exec functions, a plugin host in Python and in Node, a workflow-engine worker, an n8n Code-node runner, a Kubernetes deployment, and an agent in POSIX sh. Its README says what each one shows.