Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

17. The HTTP API, the SDKs and MCP

The CLI is one way in. Programs use the HTTP API — directly, or through the Python, Node and Elixir clients — and agent hosts use the MCP server. All of them end at the same supervisor, the long-lived Zygo process that owns the warm sandboxes.

  your program ──▶ Python/Node/Elixir ─┐
  curl ────────────────────────────────┤  HTTP  ┌──────────┐ unix socket ┌────────────┐
                                       └───────▶│ zygo api │────────────▶│            │
                                                └──────────┘             │            │
  CLI (zygo exec, zygo up, …) ──────────────────────────────────────────▶│ supervisor │
                                                                         │            │
  agent host ──▶ zygo mcp (stdin/stdout) ───────────────────────────────▶│            │
                                                                         └────────────┘

This chapter has three parts. First the HTTP API itself: how to start it, who may call what, the routes and the answers. Then the three SDKs, and the features they reach: tenants, secrets, files, streaming, cancelling, runtime pools and dependency sets. Last, the MCP server for agent hosts.

Part one: the HTTP API

zygo api is a small web server. It turns HTTP requests into messages for the supervisor, and turns the supervisor’s answers back into HTTP. It does not run sandboxes itself. Every boundary a sandbox has is built by the zygo binary and enforced by the kernel, so nothing a caller sends over HTTP can do more than the rules below allow.

Starting the API

zygo api runs the API in the foreground; run it under systemd or in a container for production. It listens on 127.0.0.1:7700, or where [api] listen or --listen says — an IP:PORT or unix:///path (created with mode 0600, so only your user can open it). By default every caller needs a bearer token — a secret string sent in the Authorization header — on every kind of listener, unix sockets included. --no-auth (or [api] auth = "none") turns that off, and is refused anywhere but a unix socket or a loopback address. So a unix socket or loopback only permits turning auth off; it never skips it by itself. zygo api --openapi prints the OpenAPI 3.1 document for this build; the same file is committed as spec/openapi.json.

Who is calling: tokens and tenants

  ZYGO_API_TOKEN (bootstrap) ───────┐
                                    ├──▶ OPERATOR: the host's owner. May act for
  operator token (zygo token mint) ─┘    any tenant with the X-Zygo-Tenant header.

  tenant token (zygo token mint --tenant acme) ──▶ TENANT acme: sees and calls
                                                   only its own things.

A tenant is one customer of whoever embeds Zygo. A token proves who is calling, and the caller cannot choose it. The bootstrap token comes from ZYGO_API_TOKEN; more are minted with zygo token mint (secrets look like zygo_ plus 64 hex characters). Only a hash is stored, so a lost token cannot be shown again — revoke it and mint another. A revoke works from the next request. An operator acting for a tenant sends X-Zygo-Tenant: acme; a tenant token that sends a different tenant gets 403. The sections Tenants and Tokens below show both from the SDKs.

The deploy gate

Calling a function that exists is one thing; creating sandboxes is another, because whoever can create one can run any image with any mount as your user. That is a shell, not an API. So zygo api starts call-only: a token reaches the functions somebody declared in a spec file that was reviewed, and nothing else. The routes that create, change or destroy things need deploy rights. An operator token minted with zygo token mint always has them, because minting it already needed them. The bootstrap token has them only when zygo api was started with --allow-deploy. A tenant token never does, whatever the flag says: one customer does not get to name an image because another is trusted.

  request with a valid token
        │
        ▼
  route needs deploy rights? ──no──▶ allowed (a tenant sees only its own things)
        │ yes
        ▼
  tenant token? ──yes──▶ 403
        │ no
        ▼
  operator token minted with `zygo token mint` ──▶ allowed
  bootstrap ZYGO_API_TOKEN ──▶ allowed only if `zygo api --allow-deploy`, else 403

Which routes are gated

These need deploy rights: PUT and DELETE /fn/{name}, POST /run, POST and DELETE /runtimes, PATCH /tenants/{id}/limits, PUT and DELETE /tenants/{id}/secrets/{name}, DELETE /tenants/{id}, DELETE /scripts/{digest}, DELETE /blobs/{digest}, DELETE /deps/{id}, POST /drain, and every token route. POST /tenants and GET /tenants are for the operator only, but do not need deploy rights.

These are not gated, and any valid token may use them: PUT /scripts, GET /scripts/{digest}, PUT /blobs, POST /deps, GET /runtimes, POST /runtimes/{name}/call and DELETE /requests/{id}. Registering code that still needs a pool to run, and calling a pool somebody else declared, is exactly what a call-only token is for.

Why each one is gated

Serving a function, a one-shot run and creating a pool all name an image and mounts, which is running code as the user Zygo runs as. Deleting a script, a blob or a dependency set is gated because each store is shared by digest or id: forgetting one forgets it for every tenant that sent the same bytes. Limits, secrets and tokens are the operator’s relationship with a customer, and draining stops the host serving anybody. Without the rights, the SDK raises AuthError; the message names the flag for an operator, and says what a tenant token is for instead.

What stays off for everybody

A request body can never remove a guarantee. Host networking and unlimited limits are refused over HTTP whatever the caller and whatever the body says. Set those where the sandbox is declared, in sandbox.toml (see chapter 20).

Private-range egress has one more place: the command that starts the API. zygo api --allow-deploy --allow-private-net lets what deploy callers serve — PUT /fn/{name}, POST /runtimes, POST /run — name private and link-local addresses in allow. It opens nothing by itself: a rule still names the address and the port, and everything else in those ranges stays shut. It is for an embedder whose scripts must call back to a service of its own, which chapter 14 says to put on an address the host really has. Without --allow-deploy it does nothing, because only a deploy call declares a sandbox, and GET /version reports it as private_net for callers who can use it.

  zygo api --allow-deploy                       zygo api --allow-deploy --allow-private-net
  POST /runtimes  allow=["10.0.0.5:4010"]       POST /runtimes  allow=["10.0.0.5:4010"]
    → 400 bad_spec: targets a private range       → served; 10.0.0.5:4010 and nothing else

Request and response headers

HeaderDirectionMeaning
Authorization: Bearer …inThe token.
X-Zygo-TenantinWhich tenant an operator is acting for.
X-Zygo-Timeout-MsinHow long the caller will wait; up to 24 hours. The default wait is 60 s.
X-Zygo-Request-KeyinThe caller’s own name for a request (1–128 printable characters), to cancel it by.
X-Zygo-Request-IdoutZygo’s id for the request.
Retry-AfteroutOn 429 (1 s), and on 503 while dependencies build (5 s).

The routes

“any” means any valid token; “op” means operator; “deploy” means deploy rights. Bodies are JSON unless marked raw; the limit is 16 MiB.

RouteWhoWhat it does
GET /healthznobody needs a tokenok, degraded (a pool below min_warm) or stopping (503).
GET /versionanyZygo version, API version, and whether you have deploy rights.
GET /metricsanyPrometheus text (below); scoped to the token.
POST /drain?grace_ms=deployStop taking requests, finish the running ones, exit.
GET /fnanyFunctions you can see: state, image, memory, request counts.
PUT /fn/{name}deployServe or replace a function. Body: {layer, base_dir, secrets?, if_changed?}.
DELETE /fn/{name}deployStop it.
POST /fn/{name}anyCall it. Body: the event. ?stream=1 for live output; ?out=1 to get /work back as a tar; ?workspace=sha256:… to send one in.
POST /fn/{name}/batchanyCall it with an array of events (up to 1024); an array of answers comes back.
GET /fn/{name}/logsanyRecent log entries: ?after=, ?limit=, ?failed=.
GET /fn/{name}/statsanyOne function’s status.
POST /fn/{name}/warmanyWake a paused or cold function now.
GET /runtimesanyPools and their load.
POST /runtimesdeployStart a pool: {name, layer, base_dir?, deps?}. layer.secrets names the secrets a call may receive; values are the calling tenant’s.
DELETE /runtimes/{name}deployStop a pool.
POST /runtimes/{name}/callanyRun a script in a pool: {script, event?, entry_point?, workspace?}.
PUT /scriptsanyStore a script (raw body); returns its sha256.
GET / DELETE /scripts/{digest}any / deployCheck or remove one.
PUT /blobsanyStore a tar (raw body), for workspaces.
GET / DELETE /blobs/{digest}any / deployCheck or remove one.
POST /depsanyBuild a dependency set from lock files: {image, files}. 202 while building.
GET /deps, GET /deps/{id}anyTheir state, and the build log.
DELETE /deps/{id}deployRemove one no pool uses.
POST /tenants · GET /tenantsopCreate or list tenants. 400 on a host whose user has no subordinate uid range, unless the supervisor runs with ZYGO_ALLOW_SHARED_UID=1 (chapter 23).
GET /tenants/{id}that tenant, or opOne tenant: scripts, limits.
DELETE /tenants/{id}deployRemove it, its scripts and its functions.
PATCH /tenants/{id}/limitsdeployNarrow its limits: mem, cpu, pids, timeout, scratch, network, allow.
GET /tenants/{id}/secretsthat tenant, or opSecret names — never values.
PUT / DELETE /tenants/{id}/secrets/{name}deploySet (raw body) or remove one.
POST /tokens · POST /tenants/{id}/tokensdeployMint an operator or tenant token.
GET /tokens · DELETE /tokens/{id}deployList or revoke.
DELETE /requests/{id}anyCancel your own request, by id or by key.
POST /rundeployA one-shot sandbox: {layer, stdin?}; returns exit code, output, and why it ended.

A layer in a body is one function’s table from sandbox.toml, sent as JSON: the same fields [fn.NAME] takes — image, entry or cmd, mem, network, allow, secrets, and so on — with every field optional, exactly as chapter 20 lists them. It is called a layer because it is merged over [defaults] the way a [fn.NAME] table is. base_dir is the folder on the server that its relative paths (entry, mounts, requirements) are relative to.

A batch runs at most 16 of its events at the same time. ?workspace= on a function call takes only a blob digest, never an inline tar (see Files in and out).

Status codes

CodeMeaning
200 / 201 / 202Done / created / still building.
400A bad spec, header or body.
401No token, or a wrong or revoked one.
403Your token may not do this: not the operator, no deploy rights, or the wrong tenant.
404 · 405No such thing · wrong method.
408The function’s own timeout killed the request. On POST /run, only when the API’s own outer deadline fired (see below).
413Body over 16 MiB, or a batch over 1024.
422A tenant limit above what could ever apply.
429Busy: every slot and the queue are full. Retry after the header.
499The request was cancelled.
500The handler raised: the body has error, stdout, stderr, exit_code.
503Warming failed, dependencies still building, or the API is stopping.
504The request stopped answering heartbeats: stuck.

Errors always have the shape {"error": "…", "code": "…"}.

A successful call

{
  "result":     {"count": 3},
  "request_id": "req-01f3…",
  "stdout":     "",
  "stderr":     "",
  "metrics":    {"wall_ms": 4.1, "cpu_ms": 2.3, "peak_rss_kb": 18420}
}

A streamed call (?stream=1) answers application/x-ndjson: one line per event, {"stream": "stdout" | "stderr" | "progress", "data": …}, then a last line with the body above and a status. Watching a request explains it.

The answer of a one-shot run

POST /run answers with more than an exit code, because the exit code alone cannot carry what a caller needs:

{"exit_code": 137, "timed_out": false, "oom_killed": true,
 "peak_rss_kb": 65536, "wall_ms": 412.7, "stdout": "", "stderr": "",
 "started": true, "phase": "run"}

A deadline kill and an out-of-memory kill are both SIGKILL, so both show exit code 137. timed_out comes from the launcher, which enforced the deadline; oom_killed comes from the kernel’s own counter in the sandbox’s cgroup (the kernel group that holds its limits). Neither is a guess. A caller deciding between “too slow” and “too much memory” — an online judge, a CI step — has nothing else to go on.

started, and when POST /run says 408

started is whether the program ran at all. false, with phase naming what failed (plan or start), is Zygo failing to build the sandbox. A caller reports that as unavailable, not as the code’s failure, and ok is false for it. An API one release behind does not send the field, and only answered once the program had run, so the SDKs default it to true.

The answer is a 200 whenever the sandbox ran, whatever ended it: a non-zero exit, its own timeout, an out-of-memory kill. The body says which. It is a 408 only when the API’s own outer deadline fired, the child was killed, and what it would have said is unknown — Zygo failing to finish, not a sandbox doing its job.

Metrics, OTLP and usage events

/metrics gives Prometheus series: zygo_api_requests_total, zygo_api_errors_total, and per function zygo_function_requests_total, zygo_function_failures_total, zygo_function_rss_bytes and zygo_function_state. --otlp-endpoint URL pushes the same numbers, plus per-tenant requests, outcomes, CPU and wall time, to an OpenTelemetry collector. --usage-webhook URL POSTs batches of usage events for billing — {tenant, function, script, request_id, wall_ms, cpu_ms, peak_rss_kb, outcome, finished_ms} — at least once, so key on request_id. Both count only requests that went through this API process.

Who sees what on /metrics. It needs a token, and the answer depends on the token, the same way GET /fn does. A tenant token gets zygo_api_requests_total and zygo_api_errors_total, which carry no names, and the per-function series for its own functions only. An operator token gets every function on the host. There are still no per-tenant series; the OTLP push and the usage webhook are where per-tenant numbers live.

Usage, for billing has the detail on the usage events.

Part two: the SDKs

There are three clients over the HTTP API: Python, Node and Elixir. Python and Node have no dependencies: the standard library has an HTTP client in each language, and a unix socket is a few lines on top of it. Erlang’s HTTP client keeps its connections to itself, so the Elixir client has two small dependencies instead (see The Elixir client). An SDK for a runtime whose point is a small, auditable boundary should not arrive with a dependency tree of its own. None is a second implementation of Zygo, and nothing in any client can widen a sandbox.

  your process ──HTTP──▶ zygo api ──control socket──▶ supervisor ──▶ warm sandboxes

On one machine the HTTP hop can be a unix socket at mode 0600, so there is no port. Across machines it is TCP. Either way a bearer token is needed, unless the API was started with auth turned off on a unix socket or loopback address.

The Python client

pip install zygo-sdk        # or: pip install -e sdk/python; then `import zygo_sdk`

The module is zygo_sdk, not zygo: a different, older project owns the name zygo on PyPI, and two packages that install the same module name overwrite each other. import zygo_sdk as zygo keeps the short name in your code.

No dependencies. zygo.connect() finds the API from its url argument, then ZYGO_API_URL, then http://127.0.0.1:7700, and its token from token= or ZYGO_API_TOKEN.

import zygo_sdk as zygo
client = zygo.connect()
res = client.fn("resize")({"url": "…"})               # a call; ~1.4 ms of overhead
print(res.result)                                      # what the handler returned
res = client.call("resize", {"url": "…"}, timeout=5)   # the same call, with a timeout
print(res.result, res.stdout, res.metrics.wall_ms)     # a Result: value, output, timings
for ev in client.stream("resize", {"url": "…"}):       # live output
    print(ev.kind, ev.data)

The client covers every route but GET /metrics, which is for a scraper: call, batch, stream, cancel, logs, stats, warm, serve, stop, run, functions; pools with serve_runtime, run_script, stream_script, put_script; put_deps, put_blob; tenants, secrets, limits and tokens; for_tenant(id); health, version, drain. What the clients can do lists every method.

The async Python client

zygo.aio is an asynchronous client, which is what an agent framework needs:

import asyncio
import zygo_sdk as zygo

async def main(events):
    async with zygo.aio.connect() as client:
        return await asyncio.gather(*(client.call("resize", e) for e in events))

asyncio.run(main(events))

It is the whole API, not a subset: every method of the plain client, with the same name and arguments, as a coroutine — tenants, tokens, secrets, limits, blobs, drain, for_tenant, and workspace/out on call and run_script included. A test holds the two surfaces equal. zygo.aio is reachable after import zygo_sdk as zygo; import zygo_sdk.aio works too. Cancelling the task cancels the request on the server.

The Node client

npm install zygo-sdk

No dependencies, Node 18 or newer, ES modules. The same methods in camelCase — runScript, serveRuntime, putSecret — with timeouts in seconds and an AbortSignal that cancels the request for you.

import { connect } from 'zygo-sdk';
const client = connect();                         // ZYGO_API_URL, ZYGO_API_TOKEN
const resize = client.fn('resize');
const out = await resize({ url: 'https://example.com/a.png' });
console.log(out.result, out.metrics.wallMs);

The TypeScript types

TypeScript types ship beside the JavaScript in index.d.ts. They are written by hand rather than compiled, so the package has no build step and what you read in the repository is what runs. A test (sdk/node/test/types.test.js) reads the declaration file against the module as it runs — every export, every Client method, every option a call reads — so a method added without a declaration fails the suite rather than shipping untyped.

The Elixir client

# mix.exs
{:zygo_sdk, "~> 0.1"}

Elixir 1.18 or newer, for the JSON module in the standard library. The same methods as Python, with the same snake_case names, as functions in the Zygo module that take the client first. Every call to the API returns {:ok, value} or {:error, %Zygo.Error{}}, and has a ! twin that returns the value or raises. The two streams yield an error as an element instead, and their ! twins raise it. Durations — timeout:, backoff:, retry_after — are milliseconds, as is usual in Elixir.

client = Zygo.connect()                                  # ZYGO_API_URL, ZYGO_API_TOKEN
{:ok, out} = Zygo.call(client, "resize", %{"url" => "…"}, timeout: 5_000)
IO.inspect({out.result, out.stdout, out.metrics.wall_ms})

There is no fn(name) handle: fn is a keyword in Elixir, and &Zygo.call(client, "resize", &1) is the same thing in one line.

Processes, and the pool

The client is a plain struct, so any process may use it. Its connections live in a pool process, which Zygo.connect/2 starts linked to the caller. A client that should live as long as your application goes under your own supervisor instead, and is fetched by name:

children = [{Zygo, name: MyApp.Zygo, url: "unix:///run/zygo/api.sock", retries: 3}]
Zygo.functions!(Zygo.client(MyApp.Zygo))
  caller process ──checkout──▶ ┌── Zygo pool ─────────────┐
    (sends, reads the answer)  │ idle connection  ·  12 s │
  ◀──────────────connection─── │ idle connection  ·   3 s │
  caller process ──checkin───▶ │ (up to pool_size: 32)    │
                               └──────────────────────────┘

A caller borrows one connection for one request and gives it back. The socket moves with it, so a caller that crashes mid-request takes its socket down rather than leaving a half-read answer in the pool. More callers than pool_size wait for a connection to come free. The pool is NimblePool, and the connections are Mint, which is HTTP as a data structure rather than a process and opens a unix socket. Those two are the package’s only dependencies.

Errors, and streams

The other clients raise one class per kind of failure. The Elixir client has one exception, Zygo.Error, and its kind field says which failure it was: :busy, :handler, :timeout and so on (see Errors in the clients). A case on the kind is how a caller branches:

case Zygo.call(client, "resize", event) do
  {:ok, out} -> out.result
  {:error, %Zygo.Error{kind: :busy, retry_after: ms}} -> {:later, ms}
  {:error, %Zygo.Error{kind: :handler, stderr: stderr}} -> {:bug, stderr}
end

Zygo.stream/4 returns a lazy Stream, and nothing is sent until it is read. Its items are {:stdout, text}, {:stderr, text} and {:progress, text}, then one last item: {:result, %Zygo.Result{}}, or {:error, %Zygo.Error{}}. A refusal before anything ran is that error alone. Zygo.stream!/4 raises the error instead, after the output before it.

What the clients can do

Who is the token the call is made with: a tenant token is one customer’s, an operator token is the host’s. Deploy marks the calls that need deploy rights (see The deploy gate).

PythonNodeElixirWhoDeploy
Call a warm functionclient.call(name, event)client.call(name, event)Zygo.call(client, name, event)eitherno
A callable for one functionclient.fn(name)client.fn(name)&Zygo.call(client, name, &1)eitherno
Several events at onceclient.batch(name, events)client.batch(name, events)Zygo.batch(client, name, events)eitherno
List functionsclient.functions()client.functions()Zygo.functions(client)eitherno
Countersclient.stats(name)client.stats(name)Zygo.stats(client, name)eitherno
Warm one nowclient.warm(name)client.warm(name)Zygo.warm(client, name)eitherno
Recent logclient.logs(name)client.logs(name)Zygo.logs(client, name)eitherno
Versionclient.version()client.version()Zygo.version(client)eitherno
Healthclient.health()client.health()Zygo.health(client)anyoneno
Drain the hostclient.drain(grace)client.drain(grace)Zygo.drain(client, grace: ms)operatoryes
Register a scriptclient.put_script(source)client.putScript(source)Zygo.put_script(client, source)eitherno
Build a dependency setclient.put_deps(image, files)client.putDeps(image, files)Zygo.put_deps(client, image, files)eitherno
How a build wentclient.deps(id)client.deps(id)Zygo.deps(client, id)own, or operatorno
Stop a running requestclient.cancel(id)client.cancel(id)Zygo.cancel(client, id)own, or operatorno
Limit a tenantclient.set_limits(id, **keys)client.setLimits(id, keys)Zygo.set_limits(client, id, keys)operatoryes
A tenant’s secret namesclient.secrets(id)client.secrets(id)Zygo.secrets(client, id)own, or operatorno
Set oneclient.put_secret(id, name, v)client.putSecret(id, name, v)Zygo.put_secret(client, id, name, v)operatoryes
Forget oneclient.delete_secret(id, name)client.deleteSecret(id, name)Zygo.delete_secret(client, id, name)operatoryes
Store a blobclient.put_blob(tar)client.putBlob(tar)Zygo.put_blob(client, tar)eitherno
Look one upclient.blob(digest)client.blob(digest)Zygo.blob(client, digest)eitherno
Forget oneclient.delete_blob(digest)client.deleteBlob(digest)Zygo.delete_blob(client, digest)operatoryes
Watch a call’s outputclient.stream(name, event)client.stream(name, event)Zygo.stream(client, name, event)eitherno
The same, for a poolclient.stream_script(rt, script)client.streamScript(rt, script)Zygo.stream_script(client, rt, script)eitherno
Look a script upclient.script(digest)client.script(digest)Zygo.script(client, digest)eitherno
Run a script in a poolclient.run_script(runtime, script)client.runScript(runtime, script)Zygo.run_script(client, runtime, script)eitherno
List runtime poolsclient.runtimes()client.runtimes()Zygo.runtimes(client)eitherno
Read a tenantclient.tenant(id)client.tenant(id)Zygo.tenant(client, id)own, or operatorno
Act for a tenantclient.for_tenant(id)client.forTenant(id)Zygo.for_tenant(client, id)operatorno
List tenantsclient.tenants()client.tenants()Zygo.tenants(client)operatorno
Create a tenantclient.create_tenant(id)client.createTenant(id)Zygo.create_tenant(client, id)operatorno
Serve a functionclient.serve(name, layer)client.serve(name, layer)Zygo.serve(client, name, layer)operatoryes
Stop oneclient.stop(name)client.stop(name)Zygo.stop(client, name)operatoryes
One-shot sandboxclient.run(image, cmd)client.run(image, cmd)Zygo.run(client, image, cmd)operatoryes
Forget a scriptclient.delete_script(digest)client.deleteScript(digest)Zygo.delete_script(client, digest)operatoryes
Forget a dependency setclient.delete_deps(id)client.deleteDeps(id)Zygo.delete_deps(client, id)operatoryes
Delete a tenantclient.delete_tenant(id)client.deleteTenant(id)Zygo.delete_tenant(client, id)operatoryes
Serve a runtime poolclient.serve_runtime(name, layer, secrets=[…])client.serveRuntime(name, layer, { secrets })Zygo.serve_runtime(client, name, layer, secrets: […])operatoryes
Stop oneclient.stop_runtime(name)client.stopRuntime(name)Zygo.stop_runtime(client, name)operatoryes
Mint a tokenclient.mint_token(tenant)client.mintToken(tenant)Zygo.mint_token(client, tenant)operatoryes
List tokensclient.tokens()client.tokens()Zygo.tokens(client)operatoryes
Revoke oneclient.revoke_token(id)client.revokeToken(id)Zygo.revoke_token(client, id)operatoryes

A listing is scoped to the caller: functions() and runtimes() through a tenant token show that tenant’s names, and nobody else’s. (/metrics is scoped the same way; see Metrics, OTLP and usage events.)

Connecting

All three clients find the address in the same order: the argument, then ZYGO_API_URL, then http://127.0.0.1:7700. These forms are accepted:

unix:///run/user/1000/zygo/api.sock     a local API over a unix socket
http://127.0.0.1:7700                   the default
https://zygo.internal:8443              across a network
box:9000                                bare host and port

The token comes from ZYGO_API_TOKEN unless one is passed. That is the same variable the server reads, so a shell that can start the API can talk to it. A unix socket still needs the token, unless the API was started with --no-auth. Every client pools connections and is safe to share between threads, tasks or processes. That matters: one connection would line concurrent callers up behind a single socket, and the warm path is measured in milliseconds.

The API hangs up on a connection that has waited 30 seconds for its next request. A pooled connection older than 20 seconds is therefore dropped, not reused; and if a reused one turns out to be closed — the send fails, or the socket ends where the status line should begin — the request goes once more on a fresh connection. That is safe because nothing came back, so the request never ran. A connection that fails after the first byte of an answer, or a new connection that fails at all, is reported as a TransportError (in Elixir, kind: :transport), never retried.

Retries

Every client can resend a refused request. retries (default 0) is how many times; backoff (default 1 s) is the base wait. Only Busy and Unavailable qualify — both mean the request never ran, which is what makes sending it again safe. A handler that raised, a request the deadline killed, a missing function: sent once, never again. Each wait is the longer of the server’s Retry-After and backoff doubled per attempt (1 s, 2 s, 4 s …), and the last refusal is the error you see.

client = zygo.connect(retries=3)                 # Busy or Unavailable → wait, send again
const client = connect(undefined, { retries: 3, backoff: 0.5 });
client = Zygo.connect(retries: 3, backoff: 500)   # milliseconds

Tenants

An embedder — a product that runs its customers’ code on Zygo — has customers. A tenant is one of them, and it is what lets the API answer “whose script is this?”:

client.create_tenant("acme")                    # POST /tenants, idempotent
acme = client.for_tenant("acme")                # a view; the same connection

script = acme.put_script(source)                # registered against acme
acme.run_script("py312", script.sha256, event)  # and only acme may run it
await client.createTenant('acme');
const acme = client.forTenant('acme');
Zygo.create_tenant!(client, "acme")
acme = Zygo.for_tenant(client, "acme")         # a new struct; the same pool

Listing or creating tenants is the operator’s: a customer that could list the other customers is a leak, whatever the limits say. A tenant may read its own record, which is how a client finds out what it registered.

What a tenant gets

  • Its own scripts. A digest (the SHA-256 hash that names a script) is not a key to it — anyone holding the bytes can compute one. So a tenant naming a digest it did not register is told the script does not exist. That is the same answer an unregistered digest gets, on purpose: “it exists but is not yours” is a fact about another customer.
  • Its own cgroup. tenants/<tenant>/<function>/…, so everything one customer runs is in one place, can be killed in one write, and counted in one read. Limiting a tenant narrows it.
  • Deletion that means it. client.delete_tenant(id) stops their functions and pools and removes the scripts nothing else refers to. It answers with both lists, because neither can be rebuilt afterwards. It also takes the tenant’s tokens.

Tokens

A tenant is only worth having if the server can tell whose request this is without being told. That is what a token is: the one part of a request the caller cannot choose.

import os
import zygo_sdk as zygo

operator = zygo.connect(url, token=os.environ["ZYGO_API_TOKEN"])
minted = operator.mint_token("acme")        # POST /tenants/acme/tokens
print(minted.secret)                        # the only time this exists

acme = zygo.connect(url, token=minted.secret)
acme.put_script(source)                     # registered against acme, no header
const minted = await operator.mintToken('acme');
const acme = connect(url, { token: minted.secret });

There are two kinds, on purpose, and no finer ladder of scopes. An operator token (mint_token(), no tenant) belongs to whoever runs this Zygo: tenants, functions, pools, and more tokens. A tenant token (mint_token(id)) belongs to one customer. It registers scripts for itself, calls the pools and functions the operator declared, reads its own record — and cannot see that any other tenant exists.

What follows from that

  • The secret exists once. The server keeps only a SHA-256 of it, so client.tokens() can list every token on the host without being a way to steal one. Nothing can print a secret again: lose one, revoke it, mint another.
  • X-Zygo-Tenant is the operator’s. for_tenant(id) says which of your customers you are acting for. A tenant token already names its tenant, and a header that disagrees with it is refused, not ignored.
  • Revoking is immediate. The next request with a revoked token is a 401. The record stays, marked, so an id in a log line still points at something.
  • Deleting a tenant takes their tokens along with the scripts only they referred to.

ZYGO_API_TOKEN is the bootstrap operator token: the same variable an existing deployment already sets, with the rights it already had. On the host, zygo token mint, zygo token ls and zygo token revoke <id> do the same three things without an HTTP round trip.

Health and draining

GET /healthz needs no token, so a load balancer can probe it. It answers one of three things:

StatusCodeMeans
ok200every pool is at its floor
degraded200a pool is below min_warm; requests work, the first pay a cold start
stopping503the supervisor is draining

degraded is a 200 on purpose. A host that can serve should be served to. A probe that took hosts out of rotation for being slow would take every host out at once after a restart. stopping is the one answer that is not a 200, because a balancer that keeps sending to a draining host is the reason draining fails.

Draining

client.drain(grace=30)    # {"drained": true, "in_flight": 0}

Draining stops taking new requests, lets the running ones finish, answers, and then exits. The default grace is 30 seconds. in_flight: 0 is a clean drain; anything else means the grace ran out. A deploy script needs to know which of those happened. SIGTERM does the same, so a container stop or a systemctl restart needs no call at all. The grace there is 25 seconds, chosen against the 30 that systemd and Docker give a process before SIGKILL: a drain that outlived its own kill would never finish. Chapter 16 covers running Zygo in production.

Usage, for billing

Every finished request produces one usage event, from the supervisor — whether it came over HTTP, from zygo exec at a terminal, or from an MCP tool:

{"tenant": "acme", "function": "py312", "script": "sha256:…",
 "request_id": "00000042", "wall_ms": 812.4, "cpu_ms": 740.1,
 "peak_rss_kb": 48200, "outcome": "ok", "finished_ms": 1790000000000}

outcome is one word — ok, error, timeout, cancelled, stuck — so a dashboard groups by it instead of working it out again from four true/false fields. A caller who cancelled their own request reads cancelled, even if the deadline happened to pass while the kill landed.

Three ways to collect it

WhereHowWhich requests
The supervisor’s log, target zygo::usagenothing to set upall of them: HTTP, zygo exec, MCP
OTLPzygo api --otlp-endpoint URL: zygo.tenant.requests, .outcomes, .cpu, .wall, one series per tenantonly those that went through this API process
A webhookzygo api --usage-webhook URL: batches of up to 256, posted as {"events": [...]}only those that went through this API process

The OTLP series and the webhook are counted in the memory of the zygo api process. Requests from zygo exec or from MCP never pass through it, so they appear only in the supervisor’s log. If you bill from the webhook, make sure every billable request goes through the API.

The webhook is at least once

A batch that fails goes back on the front of the queue, in order, and is retried. So a receiver may see an event twice and should key on request_id. The queue holds at most 10 000 events. When a webhook has been down long enough to fill it, the oldest events are dropped and the count is logged. The alternative is the API process growing until it takes the serving path down to protect the billing path, which is the wrong way round. The events are in the supervisor’s log either way.

Limiting a tenant

A pool is declared once by the operator and called by every customer. One customer should not be able to take the whole of it:

client.set_limits("acme", mem="256M", cpu=0.5, pids=64, timeout="30s")

The keys are mem, cpu, pids, timeout, scratch, network and allow. It is a PATCH, so the keys you pass are set and the rest are left alone. timeout lands on the supervisor’s deadline rather than the cgroup, because a cgroup cannot enforce a wall clock. The rest are cgroup settings.

Limits only narrow

A tenant’s limits are applied as the minimum of themselves and whatever the function or pool was declared with. They are written on the request’s own cgroup before the handler is let go. So the worst a wrong value can do is give a customer less than they were promised, never more. No value and no key can widen anything, which is what makes the route safe to expose.

A value above every ceiling the tenant can reach — their own functions, and every pool on the host — is refused with 422 naming the key. It could never take effect, and storing it would let you believe you had tightened something you had not. Above one ceiling and below another is fine: it narrows the larger and does nothing to the smaller. Chapter 14 explains the limits themselves.

Secrets

A function’s secrets are delivered as files at /run/secrets/<name>, written from outside the sandbox and removed when the last request in flight finishes. An operator at a terminal supplies them from their own environment. An embedder’s customers cannot: they have their own keys and nobody to restart a supervisor. So Zygo can store secrets per tenant.

export ZYGO_SECRETS_KEY=$(zygo secrets keygen)   # before the supervisor starts
zygo secrets set acme STRIPE_KEY                 # reads it with echo off
client.put_secret("acme", "STRIPE_KEY", value)   # PUT, needs deploy rights
client.secrets("acme")                           # ['STRIPE_KEY'] — names only

A stored secret fills in what the shell did not supply, per tenant, when a function is served. Where both have a value, the shell wins: zygo serve at a terminal is somebody saying what they want now. A tenant cannot set its own secrets; the operator holds that relationship.

No way to read a value back

This is not a missing route. There is no answer shape that could carry a value, because a store that answered with values would make every route that reaches it a way to read every customer’s keys. Values are sealed with ChaCha20-Poly1305 (an authenticated cipher) under a key Zygo never stores. Each is bound to its own tenant/name, so it cannot be moved to another by anything that can only rename files. A passphrase is refused rather than stretched: turning one into a key needs a password key function, and a store that accepted hunter2 and stretched it badly would be worse than one that said no.

What that protects, and what not

It protects the bytes at rest: a backup, a stray tar, anything that can read one user’s files. It does not protect them from a process that can read the supervisor’s memory, and it is not a hardware root of trust. An operator who needs those has a KMS (a key management service).

Secrets for a runtime pool

A pool’s sandbox is shared by several tenants, and /run/secrets is one directory in it — so a pool holds no values. It names them, and each call is given the calling tenant’s values from the store, for that call only:

client.serve_runtime("py312", {"image": "python:3.12-slim", "agent": "python"},
                     secrets=["STRIPE_KEY"])            # names, not values
client.put_secret("acme", "STRIPE_KEY", value)          # acme's value
client.for_tenant("acme").run_script("py312", digest)   # reads acme's, as a file
await client.serveRuntime('py312', { image: 'python:3.12-slim', agent: 'python' },
                          { secrets: ['STRIPE_KEY'] });

The script reads /run/secrets/STRIPE_KEY as a function would. While the files exist the call has its zygote to itself, so no other tenant’s child is forked beside them; if every zygote is busy the pool grows, and at max_warm the call gets busy, which the SDKs retry. A call from a tenant that has no secret under one of the names is refused with 400 before anything runs, naming the secret; a pool that names secrets on a host with no store key is refused when it is served. The serve_runtime argument is a list of names — the difference from serve, whose secrets are values. Chapter 14 has the rules.

Files in and out

A handler that converts a document needs the document, and the caller needs what comes back. Neither belongs in a JSON event. So a request can carry a workspace: a tar archive (one file that packs many files) that Zygo unpacks into the request’s own directory. With out, the directory comes back as a tar with the answer.

  caller                                  one request's sandbox
  ──────                                  ─────────────────────
  tar of files ──inline or blob──▶ unpacked into a fresh directory under /work
                                   the handler starts in it: reads in.pdf,
                                   writes out.png
  answer + tar ◀──── out=1 ──────  the directory, packed on the way out
                                   then removed, whatever happened
tar = make_tar({"in.pdf": pdf_bytes})

out = client.run_script("convert", script, {"to": "png"},
                        workspace={"inline": base64.b64encode(tar).decode()},
                        out=True)

open("result.tar", "wb").write(out.workspace)   # already decoded
const out = await client.runScript('convert', script, { to: 'png' }, {
  workspace: { inline: tar.toString('base64') },
  out: true,
});

Inside the handler

The handler is started in its own directory and told where it is:

def handler(event):
    with open("in.pdf", "rb") as f:        # the caller's files are just here
        ...
    open("out.png", "wb").write(rendered)  # and this comes back with `out=1`
    return {"pages": 3}

Send a fixture once: blobs

An embedder often sends the same fixture across a thousand calls. Store it once as a blob and name it by digest:

blob = client.put_blob(tar)                       # PUT /blobs, idempotent
client.run_script("convert", script, event, workspace={"blob": blob.sha256})

A warm function’s body is the event itself, with nowhere to put a workspace. So there it goes in the query string: client.call(name, event, workspace=blob.sha256, out=True), which is ?workspace=sha256:…&out=1. It takes a blob only, because an inline tar in a URL would be a megabyte of base64 in a request line. The async Python client takes both as well.

What keeps one request’s files from another’s

Not a mount namespace, and the reason was measured rather than assumed. A forked child runs at an unprivileged uid with no CAP_SYS_ADMIN in the sandbox’s user namespace, so unshare(CLONE_NEWNS) fails with EPERM. That holds even with the most permissive seccomp profile, so it is the namespace and not a filter. One path cannot mean a different directory to each request. What is there instead is listed plainly, because the first two are weaker than a namespace would be:

  • /work cannot be listed (mode 0311), so a request cannot see its neighbours’ names. The test suite checks this by trying.
  • The directory’s name is 128 random bits, not the request id, which is a counter.
  • It is removed when the request ends, whatever the request did, so the window is one request long. This is also checked.

Archives are unpacked, not trusted

Every archive is unpacked by Zygo under strict rules. Only files and directories are allowed: no symlinks, no hard links, because that is how an archive writes outside the directory it was unpacked into. No .., no absolute paths. File modes are Zygo’s, not the archive’s. Entries and bytes are capped, and counted as they are written, not read from a header an archive is free to lie in.

The way out is careful too. ?out=1 packs what the handler left, and the handler may leave a process running that keeps changing it. So Zygo never follows a path into the folder. It opens each folder it has already opened, one name at a time, and never follows a link. A name that has become a link, a pipe or a socket since the folder was read is skipped. A workspace folder swapped for a link is not packed at all. Only files and folders come back, and never more bytes than the cap.

Watching a request

A call that takes a minute has something to say before it finishes:

for event in client.stream("render", {"pages": 400}):
    if event.is_result:
        print(event.result.result)
    else:
        print(event.kind, event.data, end="")      # stdout, stderr, progress
for await (const event of client.stream('render', { pages: 400 })) {
  if (event.kind === 'result') console.log(event.result.result);
  else process.stdout.write(event.data);
}

Each item is a piece of the request’s output, and the last one is the result: exactly what the non-streaming call would have returned, or raised. A handler that printed and then failed produced both. So the output is delivered first, and the exception comes when you iterate past the result.

Three kinds of item

stdout and stderr are what the request’s process wrote, kept apart as everywhere else. progress is its own kind, not a line of stdout. A long request has two things to say — what it printed, and how far it has got — and a caller should not have to parse a handler’s log messages to find the second. The handler calls event.progress(...). It exists whether or not anybody is listening, so a handler does not break depending on who called it:

def handler(event):
    for n, page in enumerate(event["pages"]):
        event.progress(f"{n} of {len(event['pages'])}")
    return {"done": True}

The result still carries the whole of stdout and stderr, bounded as always. A caller that streamed and one that did not see the same text; streaming only changes when.

On the wire

  POST /fn/render?stream=1          Content-Type: application/x-ndjson
  ──────────────────────────────────────────────────────────────────────────
  {"stream": "stdout",   "data": "loading\n"}
  {"stream": "progress", "data": "1 of 400"}
  {"stream": "stderr",   "data": "warning: …\n"}
  …
  {"status": 200, "result": {…}, "request_id": "…", "stdout": "…", …}   ◀── last line

Streaming is per request, not per function. Sending each print() as it happens costs a system call per print() on a path measured in milliseconds. So a caller that wants to watch pays for it, and everybody else keeps the fast shape. The answer is newline-delimited JSON (one JSON object per line) rather than server-sent events: every language can read a line and parse JSON. The streaming connection is held for the whole request and is not pooled. Abandoning the iterator closes it, which does not cancel the request — pass a key and use cancel for that.

Cancelling a request

A request that is running can be stopped:

out = client.call("render", event, key="job-4711")   # name it on the way in
...
client.cancel("job-4711")                            # from anywhere
const controller = new AbortController();
const call = client.call('render', event, { signal: controller.signal });
controller.abort();                                  // cancels it on the server too

Asynchronous Python needs no key at all:

task = asyncio.ensure_future(client.call("render", event))
task.cancel()          # sends the cancel before CancelledError propagates

The caller of the cancelled request gets Cancelled (HTTP 499), which is on purpose not Timeout. A timeout says the work is too slow or the limit too tight; a cancel says the answer stopped being wanted. Both arrive as exit 137 from the kernel, and only the side that sent the signal can tell them apart, so the supervisor records which it was.

  caller A ── POST /fn/render  (X-Zygo-Request-Key: job-4711) ──▶ running …
  caller B ── DELETE /requests/job-4711 ──▶ supervisor
                                              └──▶ writes cgroup.kill on that
                                                   request's own cgroup
  caller A ◀── 499 Cancelled

A key, not the request id

The id is assigned by the host and arrives with the answer, which is too late to stop the call it belongs to. X-Zygo-Request-Id comes back on every response and in the body as request_id; it joins a log line to its request. The key is what a caller uses to name a request it is still waiting for. Reusing a key is allowed, and one cancel then stops every call under it.

What actually stops the work

The supervisor writes cgroup.kill on the request’s own cgroup, from outside the sandbox. That reaches everything the handler started, and does not depend on the tenant’s code being in a state where a signal helps. The agent inside is told, so it can mark the answer, but an agent that ignores the message changes nothing. A cancel that arrives before the request was let go is the best case: nothing of the handler has run, and started: false in the answer says so. Cancelling something already finished, or another tenant’s request, is the same NotFound. Request ids are a counter, not a secret, so ownership is what keeps a cancel honest.

Runtime pools

A warm function is one script in one zygote (the parked, ready process that Zygo forks per request; see chapter 13). A runtime pool is the other shape: an image, a dependency set and an agent, with no code in it. The script arrives with the call.

  WARM FUNCTION                          RUNTIME POOL
  ┌───────────────────────────┐          ┌───────────────────────────┐
  │ image + deps + agent      │          │ image + deps + agent      │
  │ + one handler, imported   │          │ no code                   │
  └───────────────────────────┘          └───────────────────────────┘
  event ──▶ handler(event)               script digest + event ──▶ the forked
                                         child loads that script, runs it
client.serve_runtime("py312", {                 # POST /runtimes
    "image": "python:3.12-slim",
    "agent": "python",
    "min_warm": 2, "max_warm": 8,
    "mem": "512M", "timeout": "60s",
})

script = client.put_script(source)              # once, PUT /scripts
out = client.run_script("py312", script.sha256, {"month": "2026-09"})
print(out.result, out.metrics.wall_ms)
await client.serveRuntime('py312', { image: 'python:3.12-slim', agent: 'python' });
const { sha256 } = await client.putScript(source);
const out = await client.runScript('py312', sha256, { month: '2026-09' });
Zygo.serve_runtime!(client, "py312", %{"image" => "python:3.12-slim", "agent" => "python"})
%Zygo.Script{sha256: sha256} = Zygo.put_script!(client, source)
out = Zygo.run_script!(client, "py312", sha256, %{"month" => "2026-09"})

How a script gets in

client.runtimes() lists the pools with their zygote counts, and client.stop_runtime(name) stops one (so does zygo stop <name>). A pool can name secrets; each call then gets the calling tenant’s values — see Secrets for a runtime pool. run_script also takes the source directly, for a one-off not worth registering. The request never reaches the zygote: the supervisor writes the script into the sandbox, and the forked child loads it after the child’s seccomp filter (its list of allowed system calls) is installed. That is what makes it safe for two tenants to share a pool. It is checked, not just claimed: tests/linux/verify_api.sh asks a script where it was loaded from and whether it can list what else is in flight.

A pool with no agent

A pool usually holds an agent — an interpreter, warm, forking per request. It does not have to. Give it a cmd and no agent and you get the warm-exec shape. Zygo holds the sandbox, writes each request’s script into it, and runs cmd with that path as its last argument and the event on stdin.

client.serve_runtime("sh", {"image": "alpine:3", "cmd": ["/bin/sh"]})
script = client.put_script(open("wordcount.sh").read())
out = client.run_script("sh", script.sha256, {"text": "warm exec in sh"})

That runs sh /run/script/<digest> inside the sandbox. It is the right shape for bash, for a static binary that takes a script as an argument, and for any language that starts in under a millisecond. There is nothing for an agent to save there, and the warm protocol would only be one more moving part.

What warm-exec does not have

There is no protocol to carry them, so warm-exec has no streaming, no progress(), no workspaces, and no per-tenant limits narrowed per request. The sandbox itself is the same: same namespaces, same seccomp profile, same cgroup per request, same deadline. examples/warm-exec/ has both shapes side by side.

Dependency sets

A pool’s requirements names a file on the Zygo host, which is the one thing an embedder does not have. POST /deps is the other half: send the lock file itself, and get back an id a pool can be built on.

deps = client.put_deps("python:3.12-slim", {        # POST /deps
    "requirements.txt": open("requirements.txt").read(),
})
deps.id        # 'deps_3f1c…' — its name from now on
deps.state     # 'building'

while client.deps(deps.id).building:                # GET /deps/<id>
    time.sleep(2)

client.serve_runtime(
    "py312",
    {"image": "python:3.12-slim", "agent": "python"},
    deps=deps.id,
)
const deps = await client.putDeps('node:22-slim', {
  'package.json': manifest,
  'package-lock.json': lockfile,      // `npm ci` needs it, so this does too
});
await client.serveRuntime('node22', { image: 'node:22-slim', agent: 'node' },
                          { deps: deps.id });

Python takes a requirements.txt and gets a venv (a private folder of installed packages). Node takes a package.json and its lock file and gets npm ci. Either way the result is mounted read-only at /venv and the environment points at it — PATH for Python, NODE_PATH for Node — so a script just imports what the lock file named. Chapter 15 covers dependencies in general.

  POST /deps ──▶ 202 building ──┬──▶ ready ──▶ serve_runtime(…, deps=id)
                                │              └─▶ mounted read-only at /venv
                                └──▶ failed ──▶ deps(id).log has the reason

Five things worth knowing

  • It answers before the build finishes. A pip install takes minutes, and an HTTP request that waited would time out in every proxy on the way. Poll deps(id), or send the serve_runtime and retry the Unavailable it raises, which carries the host’s Retry-After as retry_after — a client opened with retries= does that itself.
  • A pool named against a build still running is refused, not queued. Nothing is started. A zygote warmed without the dependencies it was promised would serve requests that fail at import.
  • The id is a hash of the files and the image. The same lock file on the same image is one build however many customers send it, and all of them see it in deps(). A different image is a different id, because a wheel built for one interpreter fails in another.
  • The build reaches the package registries and nothing else. Installing a package runs its code — a setup.py, an npm lifecycle script — and the lock file came from whoever holds a token. The build sandbox has network = "egress" with only the registries allowed. A host that cannot enforce that (no passt, no nftables) refuses the build rather than falling back to host networking.
  • A failed build keeps its log, on the same object as the state: client.deps(id).log is the resolver’s own words.

client.delete_deps(id) forgets one, and is refused while a pool is built on it. The pool holds a read-only mount of that directory, and removing it under a warm zygote would make its imports fail one at a time.

The script store

An embedder’s scripts live in the embedder’s database, not on the Zygo host. PUT /scripts is how one gets to a sandbox without a file on the host or a line in sandbox.toml:

script = client.put_script(source)      # PUT /scripts, body is the script
script.sha256                           # 'sha256:71e2b5…' — its name from now on
script.existed                          # the store already had exactly these bytes

The body is the script itself, not JSON around it: a script is a file, and wrapping its bytes only to unwrap them again helps nobody. The name is the SHA-256 of those bytes. That makes the call idempotent (safe to repeat) in the strongest sense: the same script from two tenants is one file on disk, and neither can put different bytes under a digest the other is running.

Looking up and forgetting

client.script(digest) says whether the host holds it and how big it is. It never returns the bytes: a digest is not a key, so a store that answered with the script would make every tenant’s code readable by anyone who could guess it. client.delete_script(digest) forgets one, and needs deploy rights, because forgetting it forgets it for every tenant that sent the same bytes.

Why PUT /scripts is not gated

Registering a script does not make it runnable on its own. A request still has to name something to run it in, which is a runtime pool the operator declared. So any authenticated caller may register a script, tenant tokens included. A tenant registering its own code runs nothing by doing so, and the digest is theirs from then on.

Errors in the clients

Each kind of failure is its own type, because each one means something different about what to do next.

HTTPPython and Node errorElixir kindMeansWhat to do
429Busy, with retry_after:busythe pool is full; the request never ranretry after retry_after
408Timeout:timeoutthe deadline killed the requestthe work is too slow, or the limit too tight
499Cancelled:cancelledsomebody stopped the requestnothing: this is what was asked for
504Stuck:stuckthe sandbox went quiet with budget leftlook at the function, not its timeout
500 from a handlerHandlerError, with stdout, stderr, exit_code:handlerthe handler raisedfix the function
404NotFound:not_foundno function (or script, or request) by that nameserve it, or check the name
401, 403AuthError:authwrong token, or a deploy call without deploy rightscheck the token or the flag
400SpecError:specthe sandbox as described cannot be resolvedfix the request
no connectionTransportError:transportthe API could not be reachednothing ran
503Unavailable, with code and retry_after:unavailabledependencies still building (deps_building), a zygote that failed to warm (warm_failed), or the API stopping; the request never ranretry after retry_after, or let retries do it
anything elseZygoError:othere.g. 413, 422; the message has the statusread the message

Unavailable is the 503: the host cannot do this yet, and nothing about the request needs changing. Unlike Busy it is not backpressure — the wait is for something the host is doing. health() raises it once the API is stopping. A 422 (a limit above every ceiling) is still a plain ZygoError. Every error type is a subclass of ZygoError, so catching it catches everything. In Elixir every one is a Zygo.Error, returned as {:error, _} or raised by the ! functions, with the same fields as the class it stands for — retry_after, code, stdout, stderr, exit_code, request_id.

Busy or HandlerError: the one that matters

Busy means the request was refused before anything happened, so retrying is correct. A handler that raised will raise again. Timeout is not a guess either. The supervisor records that it killed the request, because a deadline kill and an out-of-memory kill both arrive as exit 137. In all, exit 137 has four readings — a deadline, an out-of-memory kill, a cancel, and a sandbox that went quiet — and the supervisor is the only side that knows which, so it says.

Long requests

A function’s timeout may be hours, and a caller may wait up to a day (X-Zygo-Timeout-Ms, at most 24 hours; the default wait is 60 s). Two things make that safe rather than a way to hold a slot for ever:

  • A heartbeat. The agent says every second or two that a request is still alive. A request the supervisor has heard nothing about for a minute is killed and raises Stuck, whatever its budget said. So a wedged request costs a minute, not its whole timeout, and “your code is slow” stays different from “the sandbox stopped answering”.
  • The idle policy leaves working zygotes alone. A zygote nobody has called for idle_timeout is frozen. One with a request in flight is not, whatever the clock says, because freezing it would stop the request. A request that runs for an hour under a two-second idle_timeout still finishes.

A batch is the exception to raising: each element is a result or an error, returned rather than raised, because one refused event must not hide the answers to the others.

A worked example

examples/plugin-host/ is a plugin host in about a hundred lines, built on this API alone: no sandbox.toml, no file on the Zygo machine, no shelling out to zygo. It onboards customers and gives each their own token, limits and secrets. It declares one runtime they all share, installs their code by digest, runs it with files in and out, streams the long ones, stops them, and offboards. make verify-plugin-host runs it against a real kernel. That includes the two checks this whole layer exists for: one customer cannot run another’s plugin by naming its digest, and a customer is held to their own memory limit rather than the runtime’s.

The same host from Node is examples/plugin-host-node/: a node:http server that onboards customers with an operator token, registers and runs their scripts through forTenant, streams one route as NDJSON, and turns HandlerError, Timeout, Cancelled, NotFound, Busy and Unavailable into its own status codes. node --test runs it against a fake Zygo; make verify-plugin-host-node against a real one.

The OpenAPI document

zygo api --openapi > openapi.json

This prints OpenAPI 3.1 for the build that printed it; spec/openapi.json in the repository is the committed copy. info.version is Zygo’s release. x-zygo-api is the surface version — the one a client checks. It moves only when an operation is removed or renamed. Adding a route does not move it, because an older client does not call a route it has never heard of.

What it covers, and what it does not. It lists every route with its path parameters, who may call it (x-zygo-who), whether it needs deploy rights (x-zygo-deploy), the bearer scheme (none for GET /healthz) and the shape of an error. It does not describe request bodies, query parameters, headers, or any status but 200. Those are in the tables at the top of this chapter. A client generated from the document gets one method per route and no types for what goes in, so write the bodies from this chapter, or use one of the SDKs.

The document is hand-written, and three tests read the router’s own source to keep it true: one fails when a route is missing from the document, one when the document names a route that is gone, and one when a route answered without a token is not marked so. Each SDK has a test of its own, over every operation but GET /metrics, which is for a scraper rather than a client. A route added without a client method is a test failure rather than something an embedder finds later.

Versioning

GET /version reports:

FieldMeaning
versionThe Zygo release.
apiThe HTTP surface. Bumped only on an incompatible change to a route, so it stays put across releases that change what happens behind them. This is what a client checks.
controlThe CLI-to-supervisor protocol, which no SDK speaks. Reported because a mismatch there explains an API that is up but answering errors.
deployWhether this caller has deploy rights — the truth about your own token, not just the flag.
private_netWhether what this caller deploys may allow private addresses: zygo api --allow-private-net, and deploy rights.

All three packages are 0.1.5 and follow the repository’s 0.x policy: the shape may change with a release note. After 1.0 it will not change without a major version.

Running against a real Zygo

export ZYGO_API_TOKEN=$(head -c 32 /dev/urandom | base64)
zygo up                            # warm what sandbox.toml declares
zygo api --allow-deploy &          # 127.0.0.1:7700

python -c "import zygo_sdk as zygo; print(zygo.connect().functions())"
elixir -e 'Mix.install([{:zygo_sdk, "~> 0.1"}]); IO.inspect(Zygo.functions!(Zygo.connect()))'

For one customer rather than the whole host:

zygo token mint --tenant acme      # prints the secret, once

The SDK test suites do not need any of that. All three run against a stand-in API and check the client: the transport, the error mapping, the connection pool. A test that needs a real sandbox belongs in the Rust suites, against a real kernel. The Elixir suite also has one read-only check against a real API, which runs only when asked: ZYGO_LIVE=1 mix test --only live in sdk/elixir, against the API that ZYGO_API_URL names.

make test-sdk

Part three: the MCP server

zygo mcp gives an agent host a code interpreter whose boundary is a file somebody reviewed. It speaks the Model Context Protocol (MCP), the standard way an AI agent host talks to tool servers, over standard input and output. That is what Claude Code, Claude Desktop, Cursor and the rest start as a child process. There is no port, no token and no network: the transport is a pipe between two processes running as the same user.

{ "mcpServers": { "zygo": { "command": "zygo", "args": ["mcp"] } } }

That is the whole installation. On macOS it is forwarded into the Linux VM like every other sandbox command, and the VM hop is paid once, when the host starts the server, rather than once per tool call.

Two hosts have a command for it instead of a file:

# Claude Code, for this project; add --scope user for every project
claude mcp add zygo -- zygo mcp --mem 512M --timeout 60s

# Codex: in ~/.codex/config.toml
#   [mcp_servers.zygo]
#   command = "zygo"
#   args = ["mcp", "--mem", "512M", "--timeout", "60s"]

The flags after mcp are the ceiling the model works under — the next section explains why they belong to whoever installs the server.

  ┌────────────┐  stdin/stdout, JSON-RPC  ┌──────────┐      ┌────────────────────────────┐
  │ agent host │◀────────────────────────▶│ zygo mcp │─────▶│ run_code: a fresh one-shot │
  └────────────┘                          └────┬─────┘      │ sandbox for every call     │
                                               │            └────────────────────────────┘
                                               │ list_functions, call_function,
                                               │ function_logs
                                               ▼
                                     an existing supervisor (warm functions)

The rule that shapes it

A model reads untrusted text — a web page, a file, an error message — and that text can ask it for things. So the tools expose a program and nothing else: no image, no mounts, no network mode, no limits. Those are set once, on the command line, by the person who installed the server.

zygo mcp --mem 512M --timeout 60s --workspace ./agent-scratch
zygo mcp --net egress --allow api.github.com:443
zygo mcp -f ./sandbox.toml              # [defaults] becomes the ceiling

A model that needs more than this does not get to ask for it. Somebody declares a function in sandbox.toml — with its dependencies, its egress allow list and its secrets — and the model calls that by name. The boundary is then in a file that was reviewed, which is where it belongs. A test enforces this: no_tool_can_widen_the_sandbox fails if any tool schema ever grows an image, mount, network, mem or similar field. Adding one looks harmless on its own, which is exactly why it is checked.

The flags

FlagDefaultWhat it sets
--workspace DIRa scratch folder, removed on exitthe host folder mounted at /work
--python-imagepython:3.12-slimthe image for language: python
--node-imagenode:22-slimthe image for language: node
--sh-imagealpine:3the image for language: sh
--mem, --cpu, --pids, --timeout, …the resolver’s defaultsthe limits, as for zygo run
--net, --allow, and the other sandbox flagsno networkthe sandbox, as for zygo run
-f FILE—a spec file; its [defaults] becomes the ceiling

The tools

ToolParametersWhat it does
run_codelanguage (python, node, sh), code, stdin?Runs the code in a fresh sandbox, with /work as its folder; returns output and, in words, why it failed.
list_functions—The warm functions and their state.
call_functionname, event?Calls one by name with a JSON event (fixed 60 s limit).
function_logsname, limit? (1–200), failed?Its recent log, or only the failures.

There are no tools for pools, scripts or tenants. call_function runs with no tenant, as the host’s own call. A limit outside 1–200 is clamped into that range rather than refused. The server’s instructions tell the model to prefer a warm function over run_code where one exists: a millisecond against tens of them, with dependencies already imported.

How run_code runs a program

The code is written to a file, /zygo/main.py, /zygo/main.js or /zygo/main.sh, in a folder mounted read-only. It is then run with python3, node or /bin/sh. So the program can be of any length, tracebacks name a real file, and the code cannot rewrite itself mid-run. stdin, if given, is fed to the program and then closed. The outer deadline is the sandbox’s timeout (30 s by default) plus 300 s, which leaves room for an image pull on the first run; the sandbox’s own timeout is still enforced on the whole process tree.

/work persists between calls

/work is a writable directory and the program’s working directory. It persists between calls, so a model can write a file in one call and read it in the next. Everything else written is thrown away when the call ends. Without --workspace it is a scratch directory removed when the server exits; naming one makes it real, and is how an agent is given a project to work on.

Failures in words

When a sandbox is killed, the tool result says why in words: “ran out of memory” and “exceeded its time limit” are different sentences. Both are exit 137, and a model told only “exit 137” cannot know which of its two problems to fix.

What a sandbox gets

Whatever zygo run gives. On the ns backend that is: no capabilities, a read-only root with pivot_root and a masked /proc, the default seccomp allowlist of about 215 system calls (190 of them exist on aarch64), Landlock where the kernel has it, required memory, CPU and process limits, and no network at all unless the command line said otherwise. Chapter 23 says where that boundary is weaker than it looks. run_code is not sandboxed from the model — running the model’s code is the point. It is sandboxed from the machine.

Concurrency and failures

The server speaks JSON-RPC 2.0, a simple format where each request is a JSON object with a method name and an id. It knows four methods: initialize, ping, tools/list and tools/call. Notifications (messages with no id) are ignored. Each request is handled on a thread of its own, so a run_code that takes thirty seconds does not block a list_functions behind it. Answers are written one line at a time, so two cannot mix.

ProblemAnswer
A line that is not JSONJSON-RPC error -32700 (parse error)
An unknown method-32601
Bad parameters for a method-32602
A tool that ran and faileda normal result marked isError

A tool that fails answers with a result marked isError, not a JSON-RPC error. The difference matters: an error at the protocol layer is handled by the host and never reaches the model, and the model is the one that could fix a traceback. The three tools that read warm functions connect to an existing supervisor and do not start one. There is nothing to read or call unless somebody has already served something.

Protocol revisions

The server speaks 2025-06-18 (its preferred one), 2025-03-26 and 2024-11-05. It answers initialize with the revision the client asked for when it is one of those, and with its preferred one otherwise. So the client decides whether it can live with the answer, rather than the server agreeing to a dialect it does not know.

Trying it without a host

The transport is a pipe, so a shell is enough:

printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18"}}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
  | zygo mcp

Two lines of JSON come back, one per request. Standard output belongs to the protocol, so anything a human should read — including the workspace path — goes to standard error, which is where a host shows a server’s log. Chapter 18 goes further with agents.

What is not built

To say it plainly in one place:

  • Streaming, progress, workspaces and per-request tenant limits in warm-exec pools (a pool with cmd and no agent).
  • Per-tenant series in /metrics, and billing counts for zygo exec and MCP requests outside the supervisor log.
  • Pool, script and tenant tools in MCP.
  • A warm function’s code sent by value. PUT /fn/{name} names its entry, requirements and mounts as paths under base_dir, on the host the API runs on. One warm zygote per script version (chapter 13) is therefore for a caller that shares a disk with Zygo. A caller on another machine sends its code with each request, to a runtime pool.
  • Request bodies, query parameters and status codes in the OpenAPI document (above).