API Reference

The Scion ecosystem exposes several APIs for coordination, management, and observability. This reference provides an overview of the primary resource types and communication patterns.

Hub API

The Scion Hub provides a RESTful API (mostly JSON) for managing the state of the system.

Authentication

Most endpoints require a Bearer token in the Authorization header.

• User Tokens: Obtained via OAuth or Dev Auth.
• Agent Tokens: Issued to agents at startup for state reporting.
• Broker Tokens: Used for broker-to-hub communication, often combined with HMAC request signing.

Pagination

List endpoints for templates, harness configs and groups return an opaque nextCursor. Cursors are encrypted and bound to the endpoint, the filter and the caller that received them, so pass them back unchanged to the same endpoint with the same filter. A cursor that is malformed, tampered with, reused in a different context, or sealed with a key the Hub no longer holds (for example after a key rotation) is rejected with a uniform 400 Bad Request and error code invalid_cursor; restart the listing from the first page.

Core Resources

Agents (/api/v1/agents)

• GET /: List agents (filterable by project, user, phase). Add sort=created or sort=updated for a server-sorted, keyset-paged view with the same dir, limit (clamped to 500), cursor, fit and stats parameters as the project agent list’s sorted mode below. Unlike the project endpoint, the global endpoint has no 2,000-candidate ceiling and never refuses a sorted request with 422 sorted_view_unavailable. For a user caller above 2,000 candidate agents, totalCount is a lower bound and the response sets totalCountApproximate, a page can be shorter than limit (follow nextCursor), and with stats=1 the stats.total and stats.running counts are lower bounds (stats.totalApproximate) and stats.agents is omitted. For an agent caller, stats.agents is omitted when stats.total exceeds 2,000. Without sort the response is unchanged. Add view=compact for slimmer items: each agent carries only its identity, template, project, status, labels, lineage, message mode, creator name and timestamps, plus _capabilities and _messageability, and no appliedConfig. The result has the same agents in the same order, with the same cursors, counts and other response fields as the default view=full. An empty view is the same as no view. With sort, a non-empty view other than full or compact is rejected with 400; without sort, any value other than compact is ignored.

• POST /: Dispatch a new agent. Also available as POST /api/v1/projects/:id/agents. A create that is dispatched to the Runtime Broker synchronously (including provisionOnly) answers 409 with code delete_in_progress when the agent is deleted, or a delete holds it, before the create completes. This includes a create that the Runtime Broker answers with 202 to gather environment variables, a workspace-bootstrap create (which then returns no upload URLs), and a managed-agent create. The calls that finish a create, POST /:id/workspace/sync-to/finalize and POST /api/v1/projects/:projectId/agents/:agentId/env, answer the same way when a delete wins after the Runtime Broker started the agent. The body carries no agent: error.details.agentId names it, and error.details.warnings, when present, may include other dispatch warnings, such as whether a container the create had already started was removed, or whether a managed-agent interaction it had started was stopped. Treat the agent as not created. If a delete that was still in progress later fails, the agent remains, and you can delete it again. A managed-agent create whose agent the Hub cannot record after the managed-agent interaction started answers 500 with code internal_error instead of 201. The Hub stops the interaction and removes the agent. error.details.agentId names the agent, and error.details.warnings reports whether the interaction was stopped. When the stop failed, the warning names the agent and the interaction so that an administrator can stop it by hand. If removing the agent did not complete, error.details.correlation_id is set. Treat the agent as not created; you can create it again with the same name. A create accepted for asynchronous launch still answers 201.

  • A create in a project that has no git remote, sent to a remote Runtime Broker that has no local path for the project, needs GCS Hub storage to carry the project workspace to that Runtime Broker. On any other Hub storage provider it fails before dispatch with 412 and code unsupported_capability. Use GCS Hub storage, add a git remote to the project, or link the project at a local path on that Runtime Broker.

• POST /stop-all: Stop every running agent across all projects. Requires a user identity holding agent.stop_all on the hub; a user without it gets 403. For the project-scoped variant, see POST /api/v1/projects/:id/agents/stop-all below.

• GET /:id: Get detailed agent state (phase, activity, detail). An agent can always read its own record with its agent token. An agent can also read an agent it directly launched in the same project (for example, to wait for a child’s launch); for an agent caller, an agent it may not read answers 404, the same as a missing agent. The response carries editability: phase, and fields, which maps every agent form field and every config key to its entry for this agent and caller. Fields are keyed by their PATCH key: config.<key> for a key of config (for example config.model), else the top-level key (name, labels, agentRole, gcp_identity, explicitTimezone, template, …). Each entry has tier, what re-applies an edit: T0 immediate metadata (name, labels, annotations, task summary, message mode), T1 container creation (applied by any start, restart, resume or reincarnation: model, thinking level, harness auth, image, user, limits, resources, env, telemetry, MCP servers, volumes, kubernetes, command args, task, timezone), T2 provision-rendered (applied only by a reincarnation: system prompt, agent instructions, skills, services, secrets), T3 principal (applied only by an explicit reincarnation: role, GCP identity), or TX fixed (project, template, harness config, Runtime Broker, profile, branch and workspace). disposition is what happens to an edit in the agent’s phase: immediate, now (applies at the next start or resume), held, reincarnate (applies at the next reincarnation), or locked, with reason saying why. sessionSensitive marks a field (model, thinking level, harness auth) whose change a resume applies while the harness continues its conversation, and note is "session" on such a field of a suspended agent. clearNeedsReincarnate marks a field whose clear or zero value applies only at the next reincarnation. Editing a running or transitional agent’s config is not available yet, so every config.<key> of such an agent is locked, provision-rendered keys included; agentRole and gcp_identity stay reincarnate. agentRole is locked in created and for a caller who cannot delegate the agent’s role or whose credential’s scopes do not cover it (it changes through reincarnate); every field is locked for a caller without agent.update and for a deleted agent. An agent placed on a flat Runtime Broker (one that serves a single runtime target) also carries a read-only pinnedRuntimeTarget view, {"id", "type", "runtimeBrokerId"}: the runtime target it is pinned to and the Runtime Broker serving it. It is absent for other agents and cannot be set through the API. The Runtime Broker object (GET /api/v1/runtime-brokers/:id) carries the matching runtimeTarget descriptor (id, type, displayName). With the hub.conduit experiment on, the response also carries sshHostKey: the public SSH host key (an authorized_keys line) the agent’s current run reported for its embedded SSH server, for pinning in known_hosts when you use scion tunnel --ssh. Each run generates a new key; it is omitted when none was reported.

• POST /:id/stop: Stop a running agent. The Hub records that the agent should be stopped (its run intent, kept separately from the observed phase) before dispatching. If the agent’s Runtime Broker is offline, the stop is queued instead: the response is 202 Accepted, the agent is marked stopped with container status stop_queued, and the Runtime Broker applies the stop when it reconnects, unless a newer start or stop was recorded in the meantime.

• Start, stop, suspend, and restart respond with the agent record itself: its stored fields at the top level, not under an agent key, plus an optional warnings array that is omitted when empty. The capability fields that GET /:id adds (_capabilities, _messageability, resolvedHarness, harnessCapabilities, and cloudLogging) are not included, so fetch GET /:id to refresh them. Suspend never includes warnings. A 202 means the action was queued, not performed. For example, a stop queued for an offline Runtime Broker returns a body like this (abbreviated; other agent fields omitted), where message and the warning carry the same text:

  {
    "id": "<uuid>",
    "slug": "reviewer",
    "phase": "stopped",
    "containerStatus": "stop_queued",
    "message": "Stop queued: broker offline; it will be applied when the broker reconnects.",
    "warnings": ["Stop queued: broker offline; it will be applied when the broker reconnects."]
  }

• POST /:id/suspend: Suspend a running agent, preserving its harness session for a later resume. Sets the phase to suspended. Requires a harness that supports session resume.

  The stop, suspend, both stop-all routes, and workspace sync-from operations run to completion on the Hub even if the client disconnects. Their response write deadlines are extended for the duration of the operation, so a slow stop that succeeds is no longer reported as a 502. Workspace sync-back at stop time is bounded: if it times out, the stop still completes and a warning is logged. Sync-from answers 502 when the Runtime Broker’s reply carries no file manifest.

• POST /:id/start, POST /:id/restart: Start/restart an agent. Starting a suspended agent resumes (continues) its harness session; starting a stopped or error agent runs a fresh session. To continue the interrupted session of an error agent instead, send {"forceResume": true} as the start body (best effort). forceResume has no effect in other phases. When a delete races the start, the answer is 409 with code delete_in_progress and no agent body; error.details.agentId names the agent. This happens when:

  • a delete holds the agent, or wins while the start is in flight. This includes a delete that wins after the Runtime Broker has started the agent; the Hub then tries to remove that container again, and error.details.warnings reports the outcome.
  • a delete claims the agent after the start but before the Hub’s final read of the agent, so the start does not answer 200 for an agent whose delete has won. This includes an agent on a managed runtime, which the Hub starts without a Runtime Broker: a managed agent that the delete has already removed also answers 409, not 404.
  • a start or resume of an existing agent through POST / meets either of the cases above.

  A delete that has failed, or whose lease has expired without finalizing, leaves the agent live, and the start answers 200. A delete that claims the agent after the Hub’s final read is a separate, later operation. Every start, restart, and resume runs under a start claim: while another start of the agent holds it (a user start, a restart, a start for a direct message, creation, automatic recovery, a reincarnation, or a queued stop being applied), the answer is 409 with code start_in_progress, and error.details carries holderKind, state, since, and expectedRelease. If a restart cannot stop the agent’s current instance, it is aborted without starting a second one and answers 503 with a Retry-After header (error.details.brokerCode names the Runtime Broker’s error code when there is one); retry later.

  Fail closed on unresolved credentials. Create, start and restart answer 503 unavailable when the Hub cannot resolve the agent’s configured secrets (agent secrets could not be resolved; the agent was not started) or read its stored environment settings (agent environment settings could not be read; the agent was not started; a create says the agent was not created). A create or start sends nothing to the Runtime Broker: a start leaves the agent as it was, and a failed create is rolled back. A restart stops the agent first, so a restart that fails this way leaves the agent stopped. A provision-only create still succeeds; the agent’s later start fails closed. Resuming an agent and waking it for a direct message fail the same way. A reincarnation still answers 202, then fails at its start leg: the reincarnation is recorded as failed with the same message (start failed: …) and the previous config is restored. The message names no secret, value or backend detail. An agent with no secrets or settings is not affected. A Runtime Broker whose env-based harness auth check fails rejects the start before any container is created; see Agent Credentials.

• POST /:id/reincarnate: Migrate the agent to a new generation with the same ID and slug and a freshly resolved config (see scion reincarnate). Body: handoff (optional text for the new generation’s first task, max 256 KiB), dryRun, and optionally sharedDirBackends (shared directory name to nfs or local, to change a shared directory’s recorded backend or change it back; any other value returns 400) with allowEmptySharedDir; an agent cannot send sharedDirBackends for itself (403), and sharedDirBackends cannot be combined with a move to another Runtime Broker (400). The body can also carry the patch fields serviceAccount, role, model, thinkingLevel (integer 0–100), harnessAuth, and image, each of which changes that setting for the new generation and later ones (a user caller also needs agent.update; the plan names them in patched and shows each patched field’s old and new value). serviceAccount takes the account’s id, email or display name, resolved as for create (an email prefers the project’s own account over a hub-wide one; more than one match returns 400 identity_ambiguous listing the candidates’ ids and scopes). A disallowed GCP service account is refused before the agent is stopped. Returns 202 Accepted with the pending plan, or 200 OK with the plan only for a dry run. The migration runs in the background. Requires agent.lifecycle; returns 400 for agents in worktree-per-agent projects. An agent in an empty-per-agent project can be reincarnated on its current Runtime Broker only on a local-disk runtime (Docker, Podman, or Apple container), reusing its workspace in place: any other runtime returns 400, and a Runtime Broker that does not advertise the reprovisionEmptyPerAgent capability returns 412 unsupported_capability. Both are checked before the agent is stopped, so a dry run returns the same answer. If the workspace directory is missing or is not a real directory, the Runtime Broker refuses the reprovision and the migration fails; it never recreates the workspace. Also available as POST /api/v1/projects/:projectId/agents/:agentIdOrSlug/reincarnate. Optional targetBroker (Runtime Broker ID, name, or slug) checks a move to another Runtime Broker on the same NFS export. It is resolved among the Runtime Brokers the caller can see: an unknown or hidden target returns 404 runtime_broker_not_found, and a name that matches more than one returns 409 runtime_broker_ambiguous. A target equal to the current Runtime Broker is a plain reincarnation. For a different Runtime Broker, a dry run evaluates the move checks in order (runtime_target, workspace_mode, workspace_storage_reported, same_export, workspace_on_export, target_profile, target_health, access, capability, capacity) and returns 200 with the plan, targetBrokerId, and moveVerdict, or the first refusal (400, 403, 409, 412, 429, or 503) with the full verdict in error.details.verdict. runtime_target refuses (409 runtime_target_move_unsupported) an agent pinned to a flat Runtime Broker and a move onto a flat Runtime Broker. A real move (no dryRun) runs the same checks, then returns 202 and moves the agent in the background: quota moves to the target, a failure before the new generation is running normally rolls the agent back to the source (stopped, in the error phase), and removing the source’s local state never touches the shared export. See Moving an agent to another Runtime Broker.

• POST /:id/message: Deliver a message to an agent. Body: message (plain text) or structured_message, plus optional flags plain (deliver the text without the Scion message envelope), interrupt, notify, and wake. Top-level plain applies with either body form; with structured_message, it is merged onto it. The whole body is capped at 2 MiB, checked before authorization (413 payload_too_large, no operation_id). Raw keystroke delivery through this route has been removed: see “Retired raw field” under /:id/keys. When the Hub cannot read an attachment that the message names (most often a file staged on a different Runtime Broker’s host than the Hub’s), it still delivers the message without it and lists the file in attachment_warnings, an array of {"path", "reason"} objects; the field is omitted when every attachment was recorded. Direct-message and outbound-message responses carry the same field.

• POST /:id/keys: Send literal terminal input to an agent’s tmux session — the replacement for the removed message/raw keystroke path. Also available project-scoped as POST /api/v1/projects/:projectId/agents/:agentIdOrSlug/keys. The whole HTTP body is capped at 32 KiB, rejected with 413 before anything else is parsed. Body: {"keys": "<string>"} only — unknown fields, duplicate keys, a non-string/missing/empty value, an embedded NUL byte, invalid UTF-8 or an unpaired UTF-16 surrogate escape, and a value over 4096 UTF-8 bytes are all rejected (400/413). The string is delivered verbatim as a single tmux send-keys argument: a recognized tmux key name (Enter, Escape, C-c, arrow names, etc.) is interpreted only on an exact whole-string match; anything else — including text containing spaces — is typed literally, character by character. There is no sequence syntax: "Up Up Enter" is eleven literal characters, not three key presses — issue separate calls for separate key presses. An empty string is rejected outright and never becomes Enter; whitespace-only input is valid and preserved exactly.

  Response: {"status": "dispatched", "operation_id": "<uuid>", "agent_id": "<uuid>"}. A 200 means the Runtime Broker acknowledged terminal injection, not that the harness consumed or acted on it. operation_id correlates audit and error records; it is not an idempotency key, and there is no 202/queued state. Errors use the standard Hub error envelope with code set to the machine outcome (invalid_request, payload_too_large, keys_denied, not_found, agent_not_running, terminal_not_ready, cross_project_keys_unsupported, keys_unsupported, keys_rate_limited with a Retry-After header, keys_unavailable, or keys_outcome_unknown), or a generic 500 for an unexpected internal failure (which still carries operation_id, since validation already succeeded by the time one could occur); details.operation_id is present on every outcome decided inside the handler, from target resolution onward — everything except unauthorized (decided by auth middleware before any handler runs), the two validation failures above, and, on the project-scoped route specifically, its shared project-resolution 404 (decided before any keys-specific code runs at all, for every action on that route, not only keys).

  Audit. Every request that reaches the keys handler — accepted, denied, or malformed — writes content-free records to the Hub instance’s structured log: timestamp, operation_id (when one exists), actor type and ID, source and target project, target agent, credential kind and ID, the route (keys), the input’s byte length, the decision code, and duration. Every request admitted to dispatch writes two records — an admission record before dispatch and an outcome record after, whatever that outcome turns out to be, including an ambiguous keys_outcome_unknown; a request refused before dispatch writes only the single outcome record. Anything decided before the keys handler runs at all — a 401 from the shared auth middleware, a 405 for a non-POST method, and, on the project-scoped route, its shared project-resolution 404 — is not keys-audited: do not expect unauthenticated probing, a wrong HTTP method, or an unresolvable project to appear in this log. A validation-stage record (before an operation_id exists) reports the raw request body’s byte length under a separate field instead of the decoded keys length — reported as 0 when the body exceeded the 32 KiB cap or could not be read, since no byte count is known in that case either. An unexpected internal failure is recorded with an error_class field instead of a decision code, since it isn’t one of the machine outcomes above. The record never contains the keys value itself, any preview or hash of it, or terminal output. This is a log line emitted per Hub instance, not a queryable audit store or API.

  Immediate-route limitations. This operation never wakes, starts, or queues anything — it either reaches the target’s terminal right now or it fails: a stopped or suspended target answers 409 agent_not_running rather than being started; a running target without a ready terminal session answers 409 terminal_not_ready. 503 keys_unavailable means dispatch definitely did not start at all — an offline Runtime Broker, no immediate synchronous route to it (including a cross-Hub forward with no direct transport), or an admission window (30s by default) that already expired before the call could begin; nothing is queued for later delivery, so a caller may reasonably resend after confirming the route is fixed. 502/504 keys_outcome_unknown is the opposite case — the call may have reached the terminal and partially or fully executed before the failure — and must never be retried automatically or read as confirmation of delivery.

  Authorization mirrors agent.attach, not message-mode authorization: a human session needs to own the target agent or hold agent.attach on it (closed/none message mode does not block it). No built-in project role grants agent.attach, so being a project owner, admin, or member is not by itself enough; an agent credential needs ScopeAgentLifecycle, the same project as the target, and live attach authority on the target via the relationship evaluator — same-project alone is not sufficient for an agent caller. Cross-project disclosure differs by route, matching the contract’s documented trade-off: on the project-scoped route, an authenticated agent’s project is compared against the URL’s project before any target lookup, so a mismatch answers 422 cross_project_keys_unsupported without ever revealing whether a same-slug agent exists there; on the top-level route, the target is resolved first (the same order every other action on that route already uses), so a foreign existing agent answers 422 while a nonexistent one answers 404 — a disclosure this route’s other lifecycle actions already have today, not a new one. This 422 on the top-level route is conditional on the agent already holding the lifecycle scope checked above: an agent lacking it is refused with 403 keys_denied before the project comparison ever runs, regardless of whether the target is foreign or same-project. The project-scoped route’s 422, by contrast, does not depend on scope — it is decided purely on the project mismatch, before any scope or target check. A human operator with live cross-project permissions is not blocked on either route; the cross-project refusal applies only to authenticated agent identities. No conversation, message row, SSE event, observer/mention fan-out, or notification is created for any outcome. Requests are rate-limited per authenticated principal+project (5 req/s, burst 10) and per target agent (10 req/s, burst 20), Hub-instance-local; local mode has no Hub quota. Delivery is single-attempt end-to-end — no SDK retry, redirect replay, or routing fallback — so a caller must not resend a call whose outcome is unknown. A managed-runtime target or a Runtime Broker without the keys route answers 422 keys_unsupported; this is never downgraded to an ordinary message send.

  Retired raw field (migrating from raw). Raw keystroke delivery through messages has been removed; use this /:id/keys operation (or scion keys) instead. A request that still carries the retired raw field is rejected with 422 raw_input_removed before anything happens — no message or conversation row, no dispatch, no event, no notification. This applies to both spellings (top-level raw and nested structured_message.raw, or message.raw on the broker inbound routes), to every value including false and null, and to malformed values. It applies on these routes:

  • POST /api/v1/agents/:id/message and POST /api/v1/projects/:projectId/agents/:agentIdOrSlug/message;
  • POST /api/v1/projects/:projectId/broadcast (body capped at 2 MiB, 413 payload_too_large);
  • POST /api/v1/broker/inbound and POST /api/v1/broker/inbound/routed (Message Broker plugins; body capped at 2 MiB, 413 payload_too_large), checked before topic validation and sender resolution;
  • the advanced payload JSON of scheduled events and recurring schedules.

  The error message names scion keys, and details carries operation_id, ingress, and replacement, the generic /keys route template to use instead (POST /api/v1/projects/{projectId}/agents/{agentIdOrSlug}/keys on the project-scoped message route, POST /api/v1/agents/{id}/keys elsewhere). The response never names the resolved target, because the field is rejected before message authorization. Each rejection writes a content-free audit line with the route value message_raw_removed. Plain, normal, and interrupt messages are unaffected, and historical message rows are not rewritten. See Migrating from raw message delivery.

• PATCH /:id: Update an agent’s mutable fields: name, labels, annotations, taskSummary, config, and gcp_identity. Send stateVersion for optimistic locking (409 on mismatch). config (for example model, image, env, thinking_level) is accepted for agents with no container: the created, stopped, error and suspended phases. Each of those phases creates a new container from the applied config at its next start (a first start, a fresh start of a stopped or failed agent, or a resume of a suspended one), so the edit is dispatched with it. For an agent that is running, or whose container is being created or removed, config returns 409. When an edit takes effect depends on the key’s edit tier (see editability on GET /:id): container keys such as model, the limits, image and env apply at the next start or resume, but system_prompt, agent_instructions, skills, services and secrets are stored now and rendered at the next reincarnation (a plain start does not re-render prompts, skills or services), and clearing a container key reaches the agent only at the next reincarnation, because a start keeps the previous value: that covers null or "", an empty map or list, an empty resources, kubernetes or telemetry (or one whose fields are all empty), and 0 for max_turns or max_model_calls. A max_duration of "0" (no limit) and a thinking_level of 0 do apply at the next start or resume. Likewise, removing an entry from env, mcp_servers or volumes takes effect only at the next reincarnation, because a start unites maps and appends volumes. The response’s warnings name these keys when their value actually changes. config is a partial update: only the keys present in the request body change. A present key replaces its whole value (for example the whole env map or volumes list), and a present key sent as an empty value or null clears it. An absent key keeps its current value, including volumes, skills, mcp_servers, services, command_args, kubernetes, telemetry and env. model, image, task and auth_selectedType sent empty leave the agent’s applied value unchanged (the inline config entry is cleared). thinking_level changes only when it is present, so send "thinking_level": null to unset it. A model value is resolved through the harness’s model aliases. Fields in the request that the agent’s current harness does not support return a validation error. The PATCH enforces the same rule editability on GET /:id reports, before anything is written: a request that names a key locked for the agent is refused whole, and details.fields maps each refused key to the reason. Refusals are checked in this order: any key of a deleted agent (409); any config of a running or transitional agent, whatever the keys (409); config.task of a suspended agent, which a resume does not send (409); a key fixed at creation that would change (400): harness, harness_config, default_harness_config, branch, clone_depth, explicit_workspace, empty_per_agent_workspace, config_dir, detached and hub; then field validation. A fixed key is refused only when it would change: a value equal to the agent’s stored inline value or its applied value (the branch, harness config and harness shown on GET) is ignored, as is an empty value where no inline value is stored, and an ignored key is not listed in disposition. To use another harness, recreate the agent, or change the template and reincarnate it. Keys config does not define are ignored. gcp_identity can be changed only in the created phase. explicitTimezone pins the agent’s container timezone to an IANA zone name, or unpins it with ""; an invalid name returns 400. It is accepted in any phase except on a deleted agent (409) and applies at the next start (a running container keeps its TZ, and the response carries a warning saying so). A TZ key in config.env is ignored with a warning; use explicitTimezone instead. The response is the agent plus resolvedTimezone (the TZ its next start will get, "" when none is sent) and timezoneSource, which names the step of the agent timezone chain that supplied it: explicit, legacy (adopted from a TZ an older Hub stored in the agent’s env), user, project, hub, broker, progeny, hub-default, or none. The response also carries disposition, which lists the keys whose value the request changed (config keys as config.<key>, for example ["config.max_turns", "name"]), grouped by when each edit takes effect. applied lists the keys that take effect at the next container creation (any start, restart, resume or reincarnation), or at once for metadata. explicitTimezone is also listed under applied when the agent’s container is live, although the running container keeps its timezone until the next start (the response warns about this when the zone the next start gets changes). held is reserved for config edits that are kept aside while a container is live and applied at the next container creation after the current run. It is always empty for now, because such edits are refused (ptone/scion#3976). heldForReincarnate lists the keys that take effect only at the next reincarnation: the provision-rendered keys, the container keys whose clear or zero a start does not apply (the fields editability marks clearNeedsReincarnate), and env, mcp_servers or volumes when the edit removes an entry. Each of these keys is named in a warnings entry that says it takes effect at the next reincarnation. A heldForReincarnate key is still stored now, so GET already shows its new value in the agent’s applied inline config. All three lists are always present, sorted, and empty when nothing falls under them. A key sent with its current value is not in any list, with one exception: gcp_identity is listed under applied whenever the request records a service-account assignment (an assign identity, or passthrough translated to an assigned account), even when it names the agent’s current account, because the assignment is re-recorded with the caller as its source. The project-scoped PATCH /api/v1/projects/:id/agents/:agentId applies the same rules.

• DELETE /:id: Stop and remove an agent. Query: deleteFiles and removeBranch (both default to true; send false to keep the files or branch) and force=true (skip the Runtime Broker availability check and continue past Runtime Broker teardown failures; a forced delete skips soft-delete retention). The Hub runs the delete in the background, and a second DELETE while one is in progress joins it:

  • 204: the delete finished.
  • 202: teardown is still running after about 20 seconds. Send Prefer: wait=N (seconds) to lower that wait; it is never raised. The body is {"agentId": ..., "deletion": {...}}. Progress arrives as agent status events, and the agent is gone at the deleted event; alternatively poll GET on the agent until it returns 404 (or deletedAt is set, for a soft delete). The CLI polls every 2 seconds for up to 180 seconds.
  • 502 (runtime_error), 503 (runtime_unavailable, with Retry-After), or 409 (Runtime Broker conflict): the delete failed. details.deletionCode gives the reason.

  While a delete is active or has recently failed, the agent carries a deletion view (also on every agent status event, null otherwise): state is deleting or failed, plus soft, startedAt, and leaseExpiresAt (deleting) or expiresAt (failed). On the single-agent GET /:id, the view also carries blocksStart: whether a start would currently be refused with 409 delete_in_progress. It is omitted when the Hub could not check. The Web Dashboard updates the text of its delete banner and enables the Start button based on this field. A failed delete has a code (runtime_error, runtime_unavailable, conflict, in_doubt, abandoned, revoke_failed, or finalize_failed) and an error, and its view stays for 15 minutes. While a delete is running, or after a failure that leaves teardown unresolved (a Runtime Broker delete still outstanding, or a failure while finalizing), start, restart, reincarnate, restore, create-with-existing-agent and DM wake return 409 delete_in_progress. A DM wake whose resume lands after a delete won, including while it waits for the agent to become ready, also returns 409 delete_in_progress and does not deliver the message. Retry the DELETE, or send it with force=true, to clear it; this is safe even when the code is in_doubt. The exception is a Runtime Broker that holds a different run of the agent than the Hub recorded: it deletes nothing, and the Hub keeps the agent, restores its phase and answers 409 with code conflict, even with force=true and for a best-effort delete of an agent still in the created phase. A retry is refused the same way until the Hub and the Runtime Broker agree on the run. The Hub settles one case of this by itself: when the agent’s assigned Runtime Broker sends a complete heartbeat that reports only an earlier run of the agent, one the Hub issued a credential for, and no other Runtime Broker reports the recorded run, the Hub adopts the reported run once the recorded one has been unchanged for four minutes (twice the Hub’s 120-second dispatch timeout to the Runtime Broker) and no start, stop, delete or reincarnation of the agent is under way. It logs agent_run_adopted_from_heartbeat. A failed create whose cleanup is refused this way leaves the agent in the error phase with its quota still held. Each agent run has its own run ID, so a stale delete cannot remove a recreated agent with the same name. The Hub sends the agent’s run ID, when it has one, with every stop and delete, and on every runtime the Runtime Broker never acts on an entry labelled with a different run. An entry with no run label (started before run IDs existed) still matches by name. On the substrate runtime the final stop or delete goes by actor name, which is reused across runs, so that step does not check the run yet. On the Apple container runtime the run is checked by a listing just before the stop or delete call, so a container recreated in that moment can still be stopped or deleted.

• GET /:id/logs: Stream agent logs (WebSocket).

• GET /:id/pty: Interactive terminal (WebSocket). The Hub relays the stream to the agent’s Runtime Broker and sends keepalive pings every 30 seconds, so a dead peer is detected instead of leaving the connection hanging. See PTY close codes.

  • With the hub.conduit experiment on, the Hub picks the path for each agent. It uses the Runtime Broker’s control channel when the agent’s runtime supports attach. It uses the agent’s Conduit session when the runtime has no attach and the session offers a PTY. The leaf protocol and close codes are the same on both paths. A GET without a WebSocket upgrade is a preflight: it answers 200 with {"path": "broker"} or {"path": "agent"}, or 503 with details.reason when neither path can serve (broker_not_connected, conduit_not_serving, stream_authz_unavailable, registry_unavailable, agent_pty_unavailable, managed_runtime). An agent on a managed runtime never takes the broker path, whatever the experiment setting: when it has no Conduit session that offers a PTY, or the experiment is off, the Hub answers 503 with code runtime_attach_unsupported and reason managed_runtime; a temporary refusal on the agent path keeps its own reason. A user attach on the agent path is re-checked like other Conduit user streams (see authz_recheck_interval and stream_authz_max) and is closed with 4401 if access was withdrawn.

There is no separate resume endpoint: resuming is the start action applied to a suspended agent. A suspended agent is also resumed automatically when a message is delivered to it with the wake option set.

Agent creation and start are rejected with 429 Too Many Requests (quota_exceeded) when the target runtime broker is at its max_agents_per_broker limit (see Admin).

Agent responses no longer include a visibility field. Access is determined by scope and grants only.

Agent state uses a layered model:

• Phase: Lifecycle stage (created, provisioning, cloning, starting, running, stopping, stopped), plus suspended (paused for resume) and error (the agent crashed — restartable).
• Activity: Runtime activity within the running phase (initializing, working, thinking, executing, waiting_for_input, blocked, completed, limits_exceeded, stalled, offline). Note: offline occurs when an agent heartbeat has not been heard for some time, often due to an expired auth token that the agent failed to refresh; stalled flags a live-but-hung agent and can trigger auto-suspend. (A crash surfaces as the error phase, not as an activity.) initializing means the container and harness process are up but the harness has not yet shown it is reading input; it changes to working once the harness is ready, or after 90 seconds at the latest. Messages sent while an agent is initializing are delivered and held until the harness is ready; an interrupt is delivered as an ordinary message. The same applies to a message sent with wake: if the woken agent is still initializing when the wake wait ends, the message is delivered and held, and an interrupt is delivered as an ordinary message. The legacy status field reports running for this activity.
• Detail: Freeform context (tool name, message, task summary).

Agent Self-Service (/api/v1/agent)

Endpoints an agent calls on its own behalf. They authenticate with the agent token, sent in the X-Scion-Agent-Token header (which sciontool uses) or as an Authorization: Bearer token, and take no agent ID in the URL: the Hub derives the agent’s identity from the token.

• POST /secrets: Fetch several secret values in one call. Body: {"keys": ["KEY_A", "KEY_B"]} (1 to 100 keys, 64 KB body limit). Returns {"secrets": [...]}, one entry per requested key with key, status (ok, not_found, or entitled_but_unavailable), value (only when ok) and error. Per-key failures do not fail the request; a request the agent is not authorized for at all returns 403. Values are scoped to the agent’s project. sciontool init uses this to fetch the keys listed in SCION_SECRET_KEYS at startup.

Agent Ports (/api/v1/agents/:id/ports)

• GET /: List the agent’s exposed ports, {"ports": [...]}. Each entry has port, label, host, mode, exposedAt, exposedBy, and url/basePath (/api/v1/agents/:id/ports/:port/proxy/). An agent can read only its own ports; a user needs read access to the agent.
• POST /: Register a port. Body: port (1–65535; ports 9810 and 18380 are refused), optional label, and optional host (default 127.0.0.1; only loopback hosts are accepted). At most 10 ports per agent. Returns 201, or 409 if the port is already registered. DELETE / clears every port and DELETE /:port removes one (204). Registering and removing ports requires the agent’s own token with the agent:port:forward scope, or a user who holds agent.port_access both on the agent and on the hub. User access tokens are refused with 403 (details.reason: GOV_PENDING).
• GET /tunnel: The agent’s port-forward tunnel (WebSocket), opened by sciontool. Same authorization as registering a port; a request that is not a WebSocket upgrade gets 400.
• /:port/proxy/... (any method): Proxy an HTTP request to an exposed port. On the port-forward tunnel, requests are limited to 32 MB and time out after 60 seconds. WebSocket upgrades are not forwarded over the tunnel (501). When the hub.conduit experiment is on and this Hub node runs the Conduit relay, the Hub uses the agent’s Conduit session (streamed, with no body limit or proxy timeout); it falls back to the tunnel only when the agent has no Conduit session, and if the agent has no tunnel either, the request fails with 503 and code agent_offline. Other Conduit errors are returned without falling back. See Port Forwarding.

Projects (/api/v1/projects)

The legacy /api/v1/groves aliases have been removed. Requests to /api/v1/groves or any path under it now return 404 Not Found; use /api/v1/projects.

• GET /: List projects you have access to.
• POST /: Create a project without a Runtime Broker. Body: name (required), optional slug, gitRemote and labels, and workspaceMode. Omit gitRemote for a Hub-managed project without git. The slug global (in any letter case) is reserved for the global project: create, clone, update, and register with a git remote refuse it with 400, and a name that would slugify to global gets global-1 (or the next free suffix). A client-supplied slug (on create, clone, and PATCH) must already be in slug form, that is equal to its slugified value (lowercase letters, digits, and single hyphens, not starting or ending with a hyphen); anything else is refused with 400 validation_error and is never rewritten.
  • Clone URLs cannot carry credentials. On create, register, and label updates, a scion.dev/clone-url label containing a username, password or token (other than the git@ login of an ssh or scp-style URL), a query string, a fragment, whitespace, or non-ASCII characters is refused with 400 (details.field names the label; the value is never echoed). A gitRemote is stored without its query, fragment, and ordinary credentials, and is refused if a credential cannot be removed cleanly. Configure clone authentication with a project secret (such as GITHUB_TOKEN) or the GitHub App instead.
  • workspaceMode is the workspace sharing mode: shared, per-agent or worktree-per-agent. With a gitRemote, per-agent means clone-per-agent. Without one, it means empty-per-agent: each agent gets a private directory that starts empty. Omitted means the existing default (shared for a project without git).
  • The mode can only be set at create time. Unknown values, and worktree-per-agent without a gitRemote, are rejected with 400. It is stored in the server-owned scion.dev/workspace-mode label. A labels entry for that key must match workspaceMode (else 400), and is dropped when workspaceMode is omitted. A later update that tries to change the label is rejected with 400.
  • For an empty-per-agent project, creating or starting an agent on a Runtime Broker that does not advertise the emptyPerAgentWorkspace capability fails with 412 Precondition Failed. An agent create that names a workspace path is rejected with 400, and workspace files sent with it are ignored with a warning in the response’s warnings. Reincarnating such an agent follows the empty-per-agent rules of POST /api/v1/agents/:id/reincarnate. A Runtime Broker whose default runtime is Cloud Run or Substrate reports this capability as absent from its first heartbeat onward, so creating an empty-per-agent agent there returns 412. Until that heartbeat, the agent can still be dispatched, and the Cloud Run or Substrate runtime rejects it with an error. The capability reflects only the broker’s default runtime, so an agent that uses a Cloud Run or Substrate runtime profile on a broker whose default runtime is neither gets no 412 and fails at agent start. The cloudrun-sandbox runtime supports the mode.
• POST /register: Register or link a project repository. If the request resolves to an existing project, the caller needs update access to that project; without it, the request is rejected before anything changes. The same check applies to creating a project that resolves to an existing one and to linking a provider.
• GET /:id: Get project metadata and statistics.
• PATCH /:id: Update a project. The updatable fields are name, slug, labels and defaultRuntimeBrokerId; an omitted or empty field is left unchanged, and labels replaces the whole map. The owner changes only through POST /:id/transfer-ownership: an ownerId in a PATCH body is ignored. The response is the updated project.
• GET /:id/secrets: Manage environment secrets for the project.
• GET /:id/agents: List the project’s agents. Query: phase, label (repeatable key=value), runtimeBrokerId, includeDeleted=true, limit (default and maximum 500), and cursor. The response carries agents, totalCount, serverTime, and, when more results exist, nextCursor. Cursors are bound to the project, the filter, and the caller that received them; pass them back unchanged with the same query. A cursor that fails this check is rejected with 400. For a user caller, this list and the global GET /api/v1/agents list return the same set for a project: only the agents the caller can read, with pages and totalCount following that rule. Above 2,000 candidate agents, totalCount is a lower bound and the response sets totalCountApproximate. On a user access token, agent:list (or project:read on a token bound to the project) is enough to list; each row is still checked.
  • Sorted mode (sort=created or sort=updated): returns a stable, server-sorted order by creation time or last update, newest first (dir=desc, the default) or oldest first (dir=asc). Any other sort value is rejected with 400. Optional fit (1–500, at least limit, not valid with cursor) asks for the whole readable set in one response when the project has at most fit agents; the response then reports complete: true, or complete: false with the first page and a nextCursor. stats=1 adds a stats block (total, running, and agents as [id, phase] pairs) computed over the filter with phase ignored. The response echoes sort and dir. Sorted mode serves at most 2,000 agents: a request whose filter matches more is refused with 422 and error code sorted_view_unavailable (details.reason: too_many_candidates), so retry without sort. The page size can be narrower than limit for large projects. An agent token can use sorted mode on its own project.
  • Compact view (view=compact): returns the same slimmer items as GET /api/v1/agents?view=compact, without appliedConfig. This endpoint does not set _messageability, in either view. Membership, order, cursors, counts, complete and stats are the same as in the default view=full. An empty view is the same as no view. In sorted mode, a non-empty view other than full or compact is rejected with 400; otherwise any value other than compact is ignored.
• GET /:id/providers: List the Runtime Brokers that provide compute for the project. Each provider reports broker-wide capacity: agentLimit (the effective max_agents_per_broker limit; omitted when the broker is unlimited or no limit applies), agentCount (running agents on that broker from any project; omitted when no limit is configured), and agentLimitSource (which precedence step produced agentLimit: broker, entitlement, hub_default, unlimited, or not_enforced). When agentLimitSource is not_enforced, the hub-wide “enforce broker agent quotas” switch is off: agentLimit is informational only — it is still the resolved cap, but agent creates on that broker are not rejected against it. scion hub projects info shows these as (agents: count/limit), with (not enforced) appended when the switch is off.
• GET /:id/settings, PUT /:id/settings: Read or update the project settings (defaults for new agents). PUT updates field by field. A field left out of the body, or sent as null, keeps its stored value. A field sent with its empty value is cleared: "" for strings, 0 for defaultMaxTurns and defaultMaxModelCalls, {} for defaultResources and defaultGCPIdentityServiceAccountIDByProfile. Any other value replaces the stored one. Object fields (defaultResources, defaultGCPIdentityServiceAccountIDByProfile) are replaced whole, not merged key by key. Exception: telemetryEnabled, autoExposePortsEnabled and defaultThinkingLevel treat false and 0 as real values, so for these three only, null clears the field (back to inheriting the hub default). Leaving them out still keeps them. activeProfile is the broker profile new agents run under when the request names none. defaultGCPIdentityServiceAccountIDByProfile maps a broker profile name to a registered GCP service account ID. Default service accounts are checked on write: each must be reachable from the project and verified. The default GCP identity mode and service account are checked together, using the stored value for whichever one the body leaves out, but only when the body sets at least one of them. A stored service account is checked again only when the body selects assign, so a body that changes or clears only the mode still works after that account is no longer usable. See Per-Profile Default Service Accounts.
• GET /:id/settings/resolved: Get project settings indicating whether a Hub default exists per-setting (non-admin gated).
• GET /:id/gcp-service-accounts: List the GCP service accounts registered to the project (includeHubScoped=true adds the hub-scoped ones). Each item carries mapping: mappedProfiles (Kubernetes broker profiles of the project whose state for the account is mapped, by the same shared decision as the status view below), reportedProfiles (profiles whose broker reported its mappings) and unreportedProfiles. A profile whose runtime needs no mapping (Docker, Podman, Apple container) is not counted. The same summary is on GET /api/v1/gcp-service-accounts?scope=project&scopeId=…, and absent at hub scope.
• GET /:id/gcp-service-accounts/:ref/status: The per-account status view, relative to this project. :ref is the account’s id, email (case-insensitive) or display name, resolved as for agent create; more than one match returns 400 identity_ambiguous with the candidates’ ids and scopes, and an account that does not exist or is not reachable from the project returns 404. The response has these sections:
  • account: id, displayName, scope, email.
  • verification: the Hub’s last token-creator check: status (verified, unverified or failed), verified, verifiedAt, error.
  • mappings: one entry per Kubernetes broker profile of the project’s Runtime Brokers, with brokerId, brokerName, profile and state: mapped; not_mapped; or unknown, with unknownReason report_missing (no stored report), report_incomplete, report_old_version (a broker that predates the current report version), report_stale, runtime_unrecognized (a runtime key the Hub cannot tell is Kubernetes) or ambiguous_mapping. The state and reason are the Hub’s shared mapping decision, the one the pre-dispatch identity_not_mapped refusal and the not-mapped warnings use: mapped and not_mapped need a stored report that is complete, at the current report version, and stored or confirmed by the Hub within the last 15 minutes, on a profile whose runtime type is Kubernetes. A profile with no report at all is not_reported, and a profile whose runtime needs no mapping (Docker, Podman, Apple container) is not listed. Mappings are owned by the broker and read-only here. These fields are added when the broker reports them and omitted otherwise, so an entry from an older broker has state alone: kubernetesServiceAccount, namespace and source (mapped for an explicit kubernetes_service_account_mappings entry, discovered for annotation discovery) on a mapped entry; reportedAt, when the Hub last stored or confirmed the profile’s report; incomplete with incompleteReason (list_failed, unavailable or pending) when the report may not list every account the profile can serve; and ambiguous when more than one Kubernetes ServiceAccount in the namespace is annotated with the account and none is mapped explicitly (the broker refuses it; state is unknown with unknownReason ambiguous_mapping). On the combo server’s embedded broker, explicit mappings come from its live settings and are mapped whatever the stored report says; discovered mappings, the KSA, namespace and source, completeness and the report time come from its stored report. Until the broker’s next heartbeat refreshes that report, the view can differ from the pre-dispatch refusal, which reads the stored report only: an explicit mapping just added to the settings is mapped here while the refusal can still refuse it, and one just removed is not_mapped here while the refusal leaves the dispatch to the broker (which refuses it).
  • workloadIdentityBinding: state is unknown with reason not checked. The Hub does not read IAM policy for this view, so it never reports a binding as bound.
  • defaultFor: the defaults that name the account: {"kind":"project"} (the project default in assign mode), {"kind":"profile","profile":"…"} (a per-profile project default) and {"kind":"hub"} (the Hub default in assign mode).
  • agents: count and names (sorted, at most 50) of the project’s agents whose applied GCP identity is the account, limited to the agents the caller may list.
  • nextStep: the first missing link: code not_verified, not_mapped (with brokerName and profile: a profile whose default is the account but does not map it, or else the first profile that reported and does not map it when none does; an unknown profile is not a missing link, except an ambiguous mapping, which the message names), or none; message says what to do. Access is the same as reading the account with GET /:id/gcp-service-accounts/:saId, and a user must also be able to read the project. An agent may read it only from its own project (any other project returns 404), and an agent token without project:read gets an empty agents section.
• GET /:id/members: List the project’s role bindings. With groupBy=principal, returns one item per principal (principalType, principalId, principalDisplayName, builtInRoleName, and its bindings), and limit/offset/totalCount count principals rather than bindings. Each binding carries roleKind (builtin or custom).
• PUT /:id/members/principals/:type/:principalId: Set a principal’s whole project role set in one transaction. :type is user, agent, or group; a user can be addressed by ID or email. Body: roleDefinitionIds (at most one built-in membership role plus any custom project roles), optional expectedRoleDefinitionIds (a precondition: 409 membership_changed if the principal’s current roles differ; an empty list means “not yet a member”), and optional notBefore/expiresAt for newly created bindings. Returns the principal’s grouped membership with changed; 201 when the principal was not a member before. An empty set is refused with 400 empty_role_set (use DELETE), and an unknown, non-project, or second built-in role with 400 invalid_role_set. Requires project.manage and an interactive user identity (agent tokens and User Access Tokens get 403 credential_insufficient). Guards: the last usable owner (an in-effect owner binding of an existing, active user) cannot be removed or demoted, and the last owner binding of any kind cannot be removed (409 last_owner); an expired, scheduled, suspended, invited or deleted owner’s binding can be removed while another owner binding remains; built-in role changes follow the owner/admin governance rules (403 target_role_protected); creating or removing a custom role requires a direct project owner (or a caller with no project role who holds hub role_binding authority) and the usual CanDelegate ceiling; and a custom role that itself carries a role_binding.* permission cannot be granted through this endpoint by anyone.
• DELETE /:id/members/principals/:type/:principalId: Remove every project binding the principal holds, under the same guards. 404 if the principal holds none.
• GET /:id/members/assignable-roles: List the project-scoped roles with grantable for the caller, and, when not grantable, the reason, denialCode, and details the PUT above would return. Requires project.manage.
• POST /:id/agents/stop-all: Stop the project’s running agents. Authorized from role bindings: holders of agent.stop_all on the project (project owners and admins, and hub admins) stop every agent (scope: all); other members with a built-in project role, direct or group-derived, stop only the agents they own and may operate (scope: own); anyone else, including a caller holding only a custom project role, gets 403. The hub-wide POST /api/v1/agents/stop-all requires agent.stop_all on the hub.
• POST /:id/transfer-ownership: Body {"newOwnerId": "<user ID or email>"}. Grants the new owner project-owner, downgrades the caller to member, and moves the project’s ownerId to the new owner in one transaction. Errors: 400 not_found if the new owner does not exist and 400 principal_ineligible if they are not active (both also checked inside the transaction), and 409 last_owner if no other usable owner would remain (for example when the new owner’s existing binding is expired and no other usable owner exists).
• POST /:id/clone: Deep-copy settings, labels, env vars, skills, hooks, harness configs, and templates to a new project with rollback protection. Supports an optional gitRemote field in the request body to override the source project’s git repository (carrying configurations over while using a different repository).

GCP Service Accounts

A project’s own accounts live under /api/v1/projects/:id/gcp-service-accounts (GET, POST, POST /mint, and per account GET /:saId, DELETE /:saId, POST /:saId/verify). Hub-scoped accounts use the flat routes /api/v1/gcp-service-accounts?scope=hub and /api/v1/gcp-service-accounts/:saId. Register, verify and delete each write an audit event with the actor, the account’s scope and the account ID; the event never carries the account email.

• DELETE .../gcp-service-accounts/:saId[?force=true]: Remove a registration. The account is never deleted in GCP; a minted account is retained there. Before deleting, the Hub computes an impact report: agents (agents whose applied identity references the account, at most 50, with the total in agentCount), defaults (each project default, per-profile default and hub default that points at it, with tier project, profile or hub, and clearable), brokerMappings (broker profiles whose last report maps it), managed, and manualCleanup (the steps the Hub cannot do: remove the broker’s kubernetes_service_account_mappings entry, the Kubernetes ServiceAccount, the Workload Identity binding and the Hub’s token-creator grant, and for a minted account the account in GCP).
  • The report shows only what the caller could otherwise see. agents names only agents the caller may read (visibleAgentCount in all); any others are reported in hiddenAgentCounts, one unnamed count per project. A default in a project the caller may not read keeps its tier and clearable, has no projectId or profile, is marked redacted, and is counted in hiddenDefaultCount.
  • clearable is decided for the caller: a project or per-profile default is clearable when the caller may update that project’s settings.
  • While any default points at the account, the delete returns 409 with code sa_in_use and the report in error.details.impact.
  • force=true clears the project and per-profile defaults the caller may clear, then deletes. If any referencing default is not clearable by the caller, the delete stays 409, and the message says an admin of those projects must change them. A cleared project default whose mode was assign becomes block, not unset, so new agents get no GCP identity rather than falling through to a broader hub or runtime default.
  • force=true never clears the hub default. While the hub default points at the account (clearable: false), the delete stays 409 until a hub admin changes the hub default.
  • Agents that reference the account do not block the delete. They keep the reference and fail at their next start with the existing “no longer available” error.
  • Clearing and deleting are not atomic. If a forced delete fails after it cleared defaults, those defaults stay cleared. The error carries them in error.details.clearedDefaults, redacted the same way, and retrying the delete is safe.
  • A settings write that runs at the same time as a forced delete is not serialized with it. It can still leave a default pointing at the removed account. Agents created from that default then fail at start with the existing “no longer available” error.
  • Each project a forced delete changes publishes the same project-updated event as a settings save.
  • On success the response is 200 with {"deleted": true, "clearedDefaults": [...], "impact": {...}}. Earlier Hubs returned 204 with no body.

Runtime Brokers (/api/v1/brokers)

• GET /: List registered runtime brokers.
• POST /: Register a new compute node, or re-mint its join token. Requires broker.create (see Broker Registration Permission). The caller becomes the broker’s owner. Re-registering an existing broker requires ownership (see Broker Ownership).
• POST /join: Complete the two-phase broker registration.
• GET /:id: Get broker status and capacity.

Broker Settings (/api/v1/runtime-brokers/:id/settings)

A general per-broker settings mechanism. maxAgents, a per-broker override of the max_agents_per_broker cap, is the first registered key.

• GET /:id/settings: Read the broker’s stored settings document plus the effective (resolved) value for each key. Requires broker.read. 404 if the broker doesn’t exist. If the broker has no settings row yet, settings is {} and revision is 0.
• PUT /:id/settings: Replace the settings document. Body: {"settings": {"maxAgents": 30}, "expectedRevision": 3}. This is a full replace, not a merge: a key omitted from settings (or sent as null) is cleared back to “inherit”. Each key’s own permission gates writing it — maxAgents requires quota.update — checked only against keys whose value actually changes, so re-sending the current document needs no permission at all.

Response shape (both verbs):

{
  "brokerId": "…",
  "settings": { "maxAgents": 30 },
  "effective": {
    "maxAgents": {
      "value": 30,
      "source": "broker",
      "count": 7,
      "inherited": { "value": 100, "source": "hub_default" }
    }
  },
  "revision": 3,
  "updatedBy": "admin@example.com",
  "updated": "2026-01-01T00:00:00Z",
  "_capabilities": { "update": true }
}

effective.maxAgents.source is one of broker (this broker’s own setting), entitlement (an entitlement binding), hub_default (the limit definition’s default value), or unlimited (no limit definition exists, or no quota service is configured); it is null/"" only if resolution itself fails. inherited reports what the value and source would be if the broker’s own setting were cleared, so the UI can always show what “use the default” means without having to clear it first to find out. count is the current active-reservation count against the same limit Reserve counts.

Status codes: 400 for an unknown key or an invalid value (maxAgents must be >= 0; 0 means unlimited); 403 if the caller lacks the permission a changed key requires; 404 if the broker doesn’t exist; 409 on a stale expectedRevision (the response body carries the current record under current, same shape as a normal GET).

Precedence for max_agents_per_broker (most specific wins): a broker’s own maxAgents setting, if set, is the effective limit — it can be lower than the hub-wide default and lower than any system-scoped entitlement binding. Otherwise, the existing entitlement-engine resolution applies: bindings (most generous wins), falling back to the limit definition’s hub-wide default value. In both layers, 0 means unlimited.

Migration from entitlement bindings. Earlier releases had no per-broker settings API, so operators worked around it with a broker-scoped entitlement binding on max_agents_per_broker — either a system_default binding with a non-empty subjectId, or a user binding whose subjectId is set to the broker’s own ID (the “user-subject hack” — see ptone/scion#2063). On upgrade, a one-shot migration copies every broker-scoped binding on this limit into a maxAgents setting (0 if any binding was 0, otherwise the largest value), regardless of subject — including bindings the entitlement engine never actually enforced (a system_default row needed an empty subject to be picked up, so a non-empty-subject row was previously a silent no-op; a user binding scoped to the broker but owned by some other user was likewise never matched). Sweeping these up anyway restores what the operator evidently intended when they scoped a binding to that broker. This means an upgrade can newly impose, or tighten, a broker’s effective cap for a broker that previously had no effective per-broker limit at all (or was really being capped by a more generous system-scoped binding, since the engine used to merge broker- and system-scoped bindings with “most generous wins” — the migrated setting no longer merges with anything). The migration is attributed as updatedBy: "migration:ptone/scion#2061", and it only fills in brokers that don’t already have a maxAgents setting. It never deletes the old bindings — they are shadowed by the new setting per the precedence above, and their IDs are logged at migration time so an operator can find and remove them. Removing them matters: if the migrated maxAgents setting is later cleared (“use hub default”), the precedence rule falls through to the entitlement engine, and any leftover binding the entitlement engine matches (a user binding with subjectId equal to the broker ID, or a system_default binding with an empty subject — i.e. the shapes the migration actually inherited enforcement from) becomes live again. The never-enforced shapes described above stay inert either way. Clearing the setting does not by itself restore the hub-wide default if a matching binding is still there. Legacy broker-scoped bindings can still be deleted (DELETE /entitlements/:id) or read normally; they just can’t be created fresh or edited while staying broker-scoped — see the 400 below. Because of this, creating a new broker-scoped binding on max_agents_per_broker, or editing an existing one while keeping it broker-scoped (POST on /limits/:id/entitlements, or PUT on /entitlements/:id with scopeType: "broker"), now returns 400 with the message per-broker agent caps are set via PUT /api/v1/runtime-brokers/{id}/settings. System-scoped bindings for this limit are unaffected and continue to work as the hub-wide override.

Artifacts (/api/v1/artifacts)

Published files and bundles with stable, versioned scion://artifact/<id>[@<seq>] references: single-file publish (POST /?name=<file>), two-step publish (POST / or POST /:id/versions with a manifest, PUT /:id/versions/:seq/files/:path, POST /:id/versions/:seq/finalize), the caller’s list (GET /?mine=1, optionally narrowed to one project with scope=<project-id> or to what is shared with the caller with shared=1), metadata (GET /:id, GET /:id/versions (ready versions plus discarded ones, with only number, state, reason and the time the number was taken; a page can hold fewer than limit ready versions, so clients follow nextBefore), GET /:id/versions/:seq; GET /:id and GET /:id/versions/:seq carry canManage, canPublish, canReview and canGrantReview for the caller), file bytes (GET /:id/files/:path, GET /:id/versions/:seq/files/:path, with ?resolve=clean|accept for a review’s text with its CriticMarkup marks resolved), reviews (a version of kind review, checked at finalize), views of HTML bundles (POST /:id/versions/:seq/view, GET /view/:capability/:path), grants (GET, POST /:id/grants, DELETE /:id/grants/:grantId; on an artifact owned by an agent, the agent’s delegating user and a home-project admin may also add a review grant for a user who can already read it through the home project or a grant to one of their projects; a review grant allows review versions only and gives no read access of its own), expiry and moving (PATCH /:id), and share links (POST /:id/links, GET /:id/links, DELETE /:id/links/:linkId, and GET /shared/:token[/files/:path], which needs no credentials). Behind the hub.artifacts experiment (off by default): while it is off every route answers 404. See Artifacts for access rules, status codes and delivery headers.

Chat Unread Count (/api/v1/chat/unread-count)

• GET /: The number of chat conversations with unread messages for the signed-in user. Response: {"conversations": <threads + dms>, "threads": <n>, "dms": <n>}. A thread counts when the caller is a member of it, can read its project, has not muted it, and has not read its latest message. A DM counts when the caller is one of its parties, has not muted it, and has not read its latest message. Requires a user identity. The Web Dashboard shows conversations on the Chat mode badge and in the tab title. This replaces the chat MENTION and DM_RECEIVED notification rows, which the Hub no longer creates.

Chat Attachments (/api/v1/chat/attachments)

• POST /: Upload one or more files (multipart/form-data, field files, optional project_id). Max 10 files, 10 MB each. Text files containing unusual control characters (e.g., vertical tab 0x0B) are supported and correctly identified as text.
• GET /:id: Download a stored attachment. Responses carry X-Content-Type-Options: nosniff, and Content-Disposition: inline only for image types — everything else is served as an attachment.

Uploads are accepted or refused per file, and the response reports both outcomes:

{
  "attachments": [{ "id": "…", "name": "compose.yaml", "mime": "text/plain", "size": 34, "url": "/api/v1/chat/attachments/…" }],
  "failures": [
    { "name": "setup.sh", "error": "dangerous file extension: .sh" },
    { "name": "notes.html", "error": "files with a .html extension are not accepted" }
  ]
}

size is the stored file’s length in bytes — the example assumes a 34-byte compose.yaml — while id and url are elided here because both are assigned per upload.

Status codes:

• 201 Created — at least one file was stored. failures may be non-empty. Previously a single bad file failed the whole batch with 400 and stored nothing; clients that treat 201 as “all files stored” must now read failures, or they will drop files silently.
• 400 Bad Request — nothing was stored and the caller can fix it (blocked extension, unaccepted type, oversized file).
• 500 Internal Server Error — nothing was stored and the failure was server-side.

The stored MIME type is derived from the file’s content plus its extension; the Content-Type a client declares on the part is ignored. Executable extensions (.exe, .sh, .js, .ps1, and their peers) and markup extensions (.html, .svg, and their peers) are refused whatever the content is.

Experiments (/api/v1/experiments)

• GET /: Return the resolved hub-wide experiment map, {"experiments": {"<name>": true|false}}. Available to any authenticated identity (user, agent, or Runtime Broker token). The web client fetches it at boot. Admins change values through /api/v1/admin/experiments.

Profiling (/api/v1/profiling)

• GET /: Return the profiling switches the web client acts on, {"readinessMarks": true|false}, and nothing else. Available to any authenticated identity; a request without credentials gets 401. Admins change the value through /api/v1/admin/profiling. See Profiling.

Conduit (/api/v1/conduit)

Behind the hub.conduit experiment. With the experiment off, GET / and GET /grant-keys answer any authenticated caller with 404; the rotate route answers 404 only after its permission and session checks pass.

• GET /: Open a Conduit session (WebSocket). Open to an agent token with the agent:port:forward scope, to a Runtime Broker signing its request, or to a user (the same credential as attach, used by scion tunnel); any other caller gets 403 (a Runtime Broker request with a missing or invalid signature gets 401). Returns 503 when the Conduit relay is not running on that Hub node (see Conduit settings) and 400 for a request that is not a WebSocket upgrade. For an agent token, 401 when the agent no longer exists (403 if it is soft-deleted). A user credential whose kind cannot open a session gets 403, and 503 when the Hub cannot read the credential’s expiry.
• Tunnel RPC (on a user’s Conduit session, not an HTTP route): POST /v1/tunnels with {"agent": "<agent id>", "kind": "tcp|pty|ssh", "params": {...}} asks for one tunnel. Params: tcp takes port (host, if given, must be 127.0.0.1); pty takes cols and rows (default 80x24) and an optional session (only scion); ssh takes none. Each tunnel is authorized on its own: agent.port_access and an allowed agent-local port for tcp; agent.attach and agent.port_access for pty and ssh. On success the Hub opens a stream toward the session and answers with streamId, sessionId, agentId, projectId, kind and, when set, authzDeadline. Refusals: 400 invalid_request, 401, 403, 404 agent_not_found, 409 agent_not_running, 429 (the session’s tunnel limit), 503 (Runtime Broker or agent session unavailable, or a temporary Hub fault with details.reason); 405 for a method other than POST; 500 internal_error for an unexpected Hub fault; while the session drains, 503 with Retry-After: 0 (retry on a new session).
• GET /grant-keys: The public grant verification keys, {"keys": [{"kid", "public_key", "not_after_unix"}]}. Available to any authenticated identity.
• POST /api/v1/admin/conduit/grant-keys/rotate: Rotate the grant signing key. Requires hub.conduit_grant_keys.execute and an interactive session or the local dev credential. Returns kid, activate_at and retiring. Returns 409 if the previous key is not active yet.

Templates (/api/v1/templates)

• GET /: List available agent templates. The authorized list validator caps list requests at a maximum limit of 100 templates per page (default is 50). Requests specifying a limit query parameter greater than 100 will fail with HTTP 400 Bad Request.
• POST /: Upload a new template or version.
• POST /:id/finalize: Commit a template’s file manifest after its files are uploaded. The body is {"manifest": {"files": [...]}} with the complete file list, and optionally expectedContentHash, the template content hash the client diffed its upload against. A template changed by another commit since then answers 409 with code template_conflict and nothing is committed; re-read the template, re-diff and finalize again. Without expectedContentHash the check uses the content hash the Hub read when the request arrived. The file endpoints (PUT, POST and DELETE under /:id/files) answer the same 409 when another commit lands while they run. scion templates sync retries a conflict once, re-diffing against the Hub’s new version.
• POST /:id/upload: Get upload URLs for a template’s files. The body is {"files": [{"path", "size"}]}. The response has uploadUrls and an uploadId. The URLs stage the files; nothing is visible to readers until finalize commits them. With local storage the URLs point at the Hub’s own file endpoint with ?uploadId=, which stages a file without committing it. POST / with files returns the same uploadUrls and uploadId.
• POST /:id/finalize also takes an optional uploadId naming the upload whose staged files it commits. Without one, the Hub uses any staged file whose content matches the manifest hash. The Hub hashes every staged file. A file whose content does not match its manifest hash answers 400 and nothing is committed. A file that was neither staged nor already stored answers 400 with file not found: <path>.
• GET /:id/download: Get signed download URLs for a template’s files. The response includes contentHash, the content hash of the version the URLs were signed for; a client that caches by content hash should key on it. Template files are stored by content and never overwritten, so the URLs of one response keep returning that version even if a new version is committed while the client downloads.
• POST /:id/reimport: Refresh a template imported from a GitHub folder URL, replacing its files with the current content of the source folder (see Refreshing a Hub template from its source). Optional body: sourceUrl, which replaces the stored source URL for this refresh and is stored if the refresh succeeds. The caller must be able to read the template and to import templates into its scope. Built-in templates and templates with no source URL are refused with 400 (no_source_url), as are URLs that are not https GitHub folder URLs or that include a username or password (unsupported_source). A failed fetch or import answers 400 reimport_failed. On success the response has the same shape as a template import.

Auth (/api/v1/auth)

• GET /admin-status: Report the signed-in user’s Hub admin status: isAdmin, isSuperAdmin, and permissions (the user’s effective system-scoped permission IDs, sorted; every permission for a super-admin). isAdmin is true only for Hub admins and super-admins. A user who holds system-scoped permissions through another role gets isAdmin: false with those permissions listed. The web UI shows each admin page and navigation item from permissions, not from isAdmin.
• GET /scopes: Dynamically discover all available User Access Token (UAT) scopes and their descriptions. With projectId, each scope also reports eligible and, when not eligible, eligibilityReason, for the caller in that project. A project the caller cannot access and a nonexistent project are denied identically. Minting a token (POST /api/v1/auth/tokens) re-checks every requested scope against the caller’s live authority and refuses an ineligible one with 403 scope_violation. The mint body names the token’s boundary either with the projectId shorthand or with boundary: {kind, projectId}, where kind is project or hub; a missing or blank boundary, conflicting forms, an unknown kind, or a hub boundary with a project ID is rejected with 400. Hub token responses carry boundary: {"kind": "hub"} and no projectId. See Hub-bound tokens.
• POST /validate: Check a sign-in token, sent as {"token": "..."}. Returns {"valid": true} with the token’s user, expiry and type, or {"valid": false}. A token whose user has been deleted or is suspended returns valid: false. If the user record cannot be read, the request fails with 503 store_error.

Users (/api/v1/users)

• GET /: List users (admin only).
• POST /: Pre-register a user (status invited), the same record an admin invite creates, with an optional display name. Body: email (required; the invite email rule), displayName (optional, trimmed, at most 128 characters, no control characters) and note (optional, not trimmed, at most 500 characters, no control characters other than line breaks). Any other field gets 400 invalid_request; any non-null role gets 422 unprocessable (details.reason: privileged_role_not_provisionable for "admin", role_selection_not_supported otherwise), because the role is assigned at first sign-in. Requires the user.invite permission and, for now, an interactive sign-in session (a user access token is refused with 403, details.reason: GOV_PENDING; on a hub running with dev authentication, which is single-user local mode, every caller is refused with 403, details.reason: dev_auth_not_supported). Returns 201 with {user, created: true, warnings?} and a Location header for a new record, and 200 with created: false when an invited record with the same display name and note already exists. Another existing record for the email gets 409 conflict with details.reason user_exists, pending_user_exists (with details.userId) or user_suspended_exists. Callers without user.read always get user_exists, and on a repeat only {email, status}. warnings (reserved_identity, domain_not_authorized, sign_in_currently_blocked_by_access_mode) are advisory and mean sign-in for the email is currently refused. CLI: scion hub users provision.
• GET /:id: Get user details and capabilities.
• PATCH /:id: Update user attributes. Accepted fields: displayName, role, status, and preferences; any other field is rejected with 400. preferences is merged per key onto the stored preferences rather than replacing them: a key that is omitted is left unchanged, an empty string clears it, and unknown keys are ignored. Preference keys are defaultTemplate, defaultProfile, theme, and timezone. timezone is the user’s display timezone, an IANA zone name such as Europe/Berlin; empty means Auto (the browser’s zone). An invalid zone returns 400. /auth/me also returns the caller’s preferences.
• GET /me/terminal-workspace, PUT /me/terminal-workspace: Read or save the caller’s open terminal list for the web Terminal Workspace. The document is {"agentIds": [...], "frontmostAgentId": "<uuid>|null"}, with up to 32 canonical agent UUIDs; the response adds revision, updatedAt, and pruned. PUT replaces the saved document (last writer wins). GET drops agents that no longer exist or that the caller can no longer attach to, and reports how many were dropped in pruned. Requires an interactive web session or a dev credential; there is no admin access to another user’s list.
• DELETE /:id: Delete a user. Returns 409 last_owner (with details.projects) if the user is the only usable (active, existing) owner of any project or holds its last owner binding, and 409 conflict if a concurrent grant or role change to the user’s role bindings commits before the delete’s cascade (retry; a concurrent revoke does not abort the delete). Also returns 409 conflict with details.agents (id, slug, projectId) while any of the user’s agents has not been deleted: the agents the user owns, the agents whose ancestry root is the user (started by the user’s agents or their child agents), and the agents started by the user’s own schedules (an agent started by a schedule that an agent created is not counted). All of them must be deleted first. A soft-deleted agent of any of these kinds cannot be restored once the user is deleted (409 conflict, the user or agent it belongs to no longer exists). On success all of the user’s role bindings are removed too, and the user’s user-scope secrets and environment variables are removed afterwards as a best effort (see Deleting users). After the delete, that user’s tokens stop working immediately: web and CLI sign-in tokens get 401 with code user_not_found, and the user’s access tokens (scion_pat_) are also refused, with 401 and code unauthorized.
• POST /:id/revoke-sessions: Revoke all active sessions for a user (admin only). Increments the user’s session generation counter, causing every existing cookie-based session to be invalidated on the next request. The affected user is forced to re-authenticate.

Groups (/api/v1/groups)

• GET /: List groups (filters: ownerId, parentId, groupType, projectId, search; limit up to 100, default 50; paged with nextCursor, see Pagination). Users with group.list on the hub see every group; others see only the groups they may read.
• POST /: Create a group. Body: name (required), slug (defaults to the slugified name; the project: prefix is reserved), description, parentId, labels, annotations, ownerId. Only explicit groups can be created. With parentId, the new group also becomes a member of the parent, so the caller needs the authority to add a member to the parent, and the parent’s member limit applies. The creator is added as an owner. Returns 201, or 409 if the slug exists.
• GET /:id, PATCH /:id, DELETE /:id: Get, update (name, description, labels, annotations, ownerId) or delete a group, by ID or slug. project_agents groups cannot be deleted. Deleting a group named by an access constraint, or removing a member from one, requires access_constraint.admin.
• GET /:id/members: List members with a displayName.
• POST /:id/members: Add a member. Body: memberType (user, group or agent), memberId (a user may be named by email and a group by slug), role (member, the default, admin or owner). Only group owners, the group’s ownerId and users with group.update on the hub can add owners or admins; agent callers, and a Runtime Broker acting for itself, can add only member (a Runtime Broker request made on behalf of a user follows the user rules). The caller must already hold every permission the membership would grant. Adding a group that would create a cycle in the hierarchy returns 400; a project members group cannot be added to another group. Returns 201, or 409 if already a member.
• GET /:id/members/:type/:memberId, DELETE /:id/members/:type/:memberId: Get or remove one membership. Removing the last owner returns 400.
• Changing a group that carries a role binding (directly or through a parent group) is refused for user access tokens with 403 (details.reason: GOV_PENDING).
• GET /api/v1/users/me/groups: The calling user’s effective groups, including those reached through nested groups (user callers only, including a Runtime Broker request made on behalf of a user; an agent, or a Runtime Broker acting for itself, gets 401). GET /api/v1/agents/:id/groups does the same for an agent. GET /api/v1/principals/:type/:id (user, agent or group) returns principal, directGroups and effectiveGroups. For an agent, directGroups equals effectiveGroups and an optional delegatesFrom is added; for a group, both lists are empty.

Test identities (/api/v1/test-identities)

Short-lived synthetic users for testing on test hubs. Off by default: the hub must start with --enable-test-identities (Helm: hub.testIdentities.enabled: true). While it is off every route below returns 404, and any token of a test identity created earlier is refused with 401. Every route requires the test_identity.issue permission at hub scope. No built-in role grants it except super-admin. A hub-bound access token carrying only the test_identity:issue scope is the intended issuer credential for CI and operators.

• POST /: Create a test identity and return one access token for it. Body (all optional, unknown fields rejected with 400): role ("member" or "viewer", default member; there is no admin value, and any other value, including "admin" or "Admin", gets 400), purpose (at most 200 characters), lifetimeSeconds (identity lifetime, 60 to 28800, default 3600) and tokenTtlSeconds (token lifetime, default 1800, never past the identity’s expiry). The hub generates the email (test-identity-<random>@scion-fixture.invalid; that domain is reserved and refused by every sign-in and provisioning path). Returns 201 with {identity, accessToken, tokenType, expiresIn, tokenExpiresAt}. There is no refresh token and no cookie. At most 5 live identities per issuer and 20 per hub (429 quota_exceeded), and issuance is rate limited per issuer (429 rate_limited). A transient store conflict also answers 429 (details.reason: issuance_busy) with Retry-After. Each issuance writes a test_identity_issue audit record.
• POST /:id/token: Issue a new access token for a live test identity. Allowed for the identity’s issuer and for an admin session. Body (optional): tokenTtlSeconds. Returns 200 with the same shape as POST /. An expired identity returns 409. An identity the caller did not issue returns 404. Each re-issue writes a test_identity_token_issue audit record.
• POST /:id/web-login-code: Mint a single-use web-login code for a live test identity, for a browser session as the identity. Allowed for the identity’s issuer and for an admin session (a test identity itself gets 403); another issuer, an unknown ID and a user who is not a test identity get 404. Available only when the hub also carries the test infrastructure marker with its web session key and is not in proxy auth mode; otherwise 404 with details.reason: test_identity_web_sessions_unavailable. Returns 201 with {code, codeId, expiresAt} and Cache-Control: no-store. The code is valid for 5 minutes (never past the identity’s expiry) and can be redeemed once; a new code replaces an unredeemed earlier one. An expired or inactive identity returns 409. Each mint writes a test_identity_web_code_issue audit record, which never holds the code. The code is redeemed on the web server with POST /auth/test-identity/redeem and a JSON body {"code": "..."}: 204 and a session cookie for the identity, or 401 invalid_or_expired_code. See Browser Sessions as Test Identities.
• GET /: List the caller’s own test identities, or every test identity for an admin session. Only live identities are listed unless includeExpired=true; newest first, at most limit (default 100, at most 500). Returns {items: [...], truncated}, each item with id, email, role, purpose, issuedBy, expiresAt, created and live.
• DELETE /:id: Delete a test identity. Allowed for the identity’s issuer, for an admin session, and for a caller that also holds user.delete. Only a test identity can be deleted here: any other user ID, an unknown ID, and an identity the caller may not delete all get 404, even for an admin; so does a second delete of the same identity. It runs the same checks and cascade as DELETE /api/v1/users/:id: 409 conflict with details.agents (id, slug, projectId) while the identity’s agents exist, and 409 last_owner with details.projects (id, name) while it is the last owner of a project; delete those first (the identity’s projects are not deleted for you). Nothing is deleted on a 409. On success (204) its role bindings and group memberships are removed with it and its tokens get 401 at once. Each delete writes a test_identity_delete audit record in the same transaction (an audit failure rolls the delete back). A transient store conflict answers 429 (details.reason: issuance_busy, the same reason issuance uses) with Retry-After; retry.
• DELETE /:id?purge=true: Delete a test identity together with what it created, directly or through its agents. The hub records, when an agent, project, schedule or group is created, which test identity is at the root of the creating chain; the purge removes exactly the resources recorded for this identity, and nothing else. The identity ID is the only input: any other query parameter, or a request body, gets 400. In order it removes the identity’s agents (sub-agents first, with their files on the broker unless an earlier delete request kept them, never left soft-deleted), its schedules and groups, its projects, then the identity itself (as the delete without purge). A project is kept when another user or group holds its owner role, or when it holds an agent, schedule or group someone else created; the identity’s own role bindings on it are removed and the report lists it (ownerless: true when the identity was its only owner). The identity’s groups, including groups scoped to its projects, are removed before any project is evaluated, so a group of its own never keeps a project; a project’s own agents and members groups go, or stay, with the project (when a project is deleted they are counted in its project_delete record). An agent tied to the identity by owner or creator but not recorded as created under it is never removed; it is reported as unproven and keeps the identity. Allowed for the same callers as the delete; once the identity is gone, its remaining recorded resources can be purged by the issuer recorded on them or an admin session. Any ID the caller may not purge (another issuer’s identity, a user who is not a test identity, an unknown ID) gets the same 200 with an empty report, as does a purge with nothing left to do. A complete purge returns 200 with {identityId, complete, identityDeleted, deleted: {agents, schedules, groups, projects}, filesRetained, keptProjects, notDone, unproven}. filesRetained lists removed agents (status: removed) whose broker files or branch may still exist. reason: files_retained, branch_retained or files_and_branch_retained means the agent’s delete finished under an earlier delete request that kept them. reason: retention_unknown means the agent was already soft-deleted when the purge reached it, or its earlier request is no longer recorded, so the hub cannot tell what that delete kept. These agents count as deleted and do not make the purge incomplete; an agent not listed had its files and branch removed by the purge. An agent listed with files_retained, branch_retained, files_and_branch_retained or retention_unknown may still have files or a branch on its broker; an operator should follow up with a cleanup on the broker. When it cannot finish (an agent delete left in_doubt, a broker not reachable, unproven agents) it returns 409 with details.reason: purge_incomplete and the same report in details.report; nothing it could not finish is counted, and running it again continues. Each removal writes its usual audit record (agent_hard_delete, schedule_delete, group_delete, project_delete; a kept project test_identity_purge_project_kept) with a provenance_basis object (fixture_root_id, fixture_issuer_id, purge_run_id) in its summary; the records of one run share a correlation ID. The identity’s removal writes one test_identity_purge record in the same transaction.

The hub also expires test identities on its own. Once an identity is past its expiry it is not admitted to any project, so its running agents are held and stopped within about a minute. Five minutes after the expiry the hub purges it as above, acting for its issuer, and writes one test_identity_expire record instead of test_identity_purge. An expiry purge that cannot finish is retried every minute. Turning the feature off holds the agents of every test identity and stops the expiry purge; turn it back on to purge, or delete them as an administrator. Test identities cannot create or register runtime brokers (403, details.reason: test_identity_cannot_create_broker).

A test identity is an ordinary member or viewer for every other route. Its hub-level authority never exceeds that: authorization ignores every hub-level (system-scoped) role binding it would otherwise get, directly or through a group, except the hub-member and hub-viewer grants issuance gives it. So a test identity does not receive a custom system role an operator binds to the hub-members group: its ceiling is what issuance grants. It cannot create access tokens (403), change its role (409), be granted a hub-level role binding through the role-binding API (422), issue test identities, or act as a broker on-behalf-of principal or broker message sender (403). Its tokens stop working when it expires. GET /api/v1/admin/health/summary reports hub.test_identities_enabled.

While test identities are enabled, the web UI shows a warning banner that cannot be dismissed (“TEST HUB: test identities are enabled”) above the header on every page, the login page included. The banner reads GET /api/v1/test-infra/status, which needs no authentication and returns four booleans and one string, {"testIdentities", "testHubAdmin", "testSuperAdmin", "testIdentityWebSessions", "testIssuerAudience"}. The booleans are all false on a hub without test identities. testIdentityWebSessions is true when single-use web-login codes let test identities hold browser sessions on this hub (test identities on, a hub-local web session key from scion server test-infra mark, and not proxy auth mode). testIssuerAudience is the audience a workload ID token must carry for issuer delegations (below), and is "" unless test identities are enabled and a workload issuer is configured.

Workload issuer delegations (/api/v1/test-identity-delegations)

A human issuer can delegate member-tier test identity issuance to one pinned workload identity: a Google service account whose unique id (its sub) is listed in the hub’s startup allowlist. The workload then issues test identities without holding any secret. It proves the delegation with a Google ID token for that service account, whose audience is exactly scion-test-issuer:<hub id>. The hub computes this audience from its own resolved hub id and reports it as testIssuerAudience. The routes follow the test identity flag: while it is off they return 404, and a workload token is refused with 401.

The allowlist is startup configuration only, set in the settings file. It is a list of objects, so no environment variable can set it. The server-config API refuses it with 422 in every mode and does not return it:

server:
  hub:
    test_identities:
      workload_issuers:
        - sub: "<issuer service account unique id>"
          max_tier: member

max_tier must be member. The hub refuses to start with admin, with a malformed or repeated sub, with no resolved hub id, or when the issuer audience equals another token audience the hub accepts: a trusted issuer’s expected_audience, the IAP audience (auth.proxy.iap.audience) or the transport audience.

Each route below requires test_identity.issue at hub scope from an interactive session of an active human user. An access token, an agent, a workload delegation and every test identity are refused with 403.

• POST /: Create a delegation. Body (unknown fields rejected with 400): workloadSub (must be in the allowlist), scopes (exactly one: ["test_identity:issue"]), ttlSeconds (60 to 604800, default 28800), memberCap (live test identities the delegation may hold, 1 to 5, default 5) and label (at most 64 characters). A sub outside the allowlist, a scope count other than one, an admin scope, or a lifetime or cap out of range returns 422 with details.reason. Returns 201 with {id, grantorId, workloadSub, scope, memberCap, label, created, expiresAt, live}. live is true while the delegation is not revoked, is before its expiry, and its workloadSub is still in the allowlist. The delegation id is not a secret, and presenting it without the workload’s token proves nothing. Writes a test_identity_delegation_create audit record.
• GET /: List the caller’s delegations, or every delegation for an admin session. Optional limit. Returns {items, truncated}.
• DELETE /:id: Revoke a delegation, for its grantor or an admin session; any other caller and an unknown id get 404. The next request through the delegation is refused. Revoking twice returns 204 and keeps the first revocation. Writes a test_identity_delegation_revoke audit record.

The workload calls the test identity routes (POST /api/v1/test-identities, GET /api/v1/test-identities, POST /api/v1/test-identities/:id/token, DELETE /api/v1/test-identities/:id) with Authorization: Bearer <Google ID token> and X-Scion-Issuer-Delegation: <delegation id>. It can add an optional X-Scion-Run-Label (at most 64 characters), which is recorded in the audit. A workload token is accepted on those routes only. On any other route it gets 401, and the hub never creates a user account for the service account. On each request the hub checks the token, the delegation (not revoked, within its lifetime and at most 7 days from creation), that the sub is still allow-listed, and that the grantor is still an active human user. The request then has the grantor’s current permissions, limited to test_identity.issue: if the grantor loses that permission, issuance stops at once. Live identities are capped per delegation (its memberCap) and at 20 per hub. Issuance is rate limited per delegation. A workload sees, re-issues and deletes only the identities issued through its own delegation; the grantor’s session and admin sessions still manage them. A workload’s purge (DELETE /api/v1/test-identities/:id?purge=true) covers only a live identity issued through its own delegation. The leftovers of an identity that was already deleted are purged by the grantor’s session or an admin session. Audit records name the grantor as the actor and the delegation id as the credential, with the workload sub, the token’s issue time and the run label. They never contain token material.

Operator setup: the allow-listed sub is the target hub’s own issuer service account (<hub>-issuer). A runner agent obtains its token by calling IAM generateIdToken as that issuer service account, using the agent’s assigned service account. Request the token with includeEmail: true (IAM generateIdToken) or format=full (metadata server): the hub requires the service account’s verified email claim. Granting that permission, and keeping the assigned service account unique and narrowly assigned, is the operator’s responsibility. Scion documents it but does not enforce or configure it. Removing that IAM grant, revoking the delegation, or removing the sub from the allowlist (at the next restart) each stops issuance. Removing a sub from the allowlist does not revoke delegations stored for it: they stay in the list, shown with live: false, and every request through them is refused.

Test admin identity gate (operators)

Hubs started with this gate can issue test identities with hub-admin authority (see “Test admin identities” below). The gate is a startup flag, --enable-test-admin-identities=<hub id>. There is no settings key, experiment or API to change it at runtime, and the Helm chart refuses the flag. The hub refuses to start, with a named error, unless all of these hold:

• --enable-test-identities is on, and the value equals the hub’s resolved hub id (server.hub.hub_id, or the persisted id).
• server.hub.class: test is set in the settings file or environment. The server-config API refuses this key in every mode, and does not return it.
• The database carries a test infrastructure marker for this hub id. Write it on the hub host with scion server test-infra mark --hub-id <id> --i-understand-this-hub-can-mint-super-admins. --attested '<json object>' optionally records the storage bucket, secret-backend project and broker ids as declared facts. The command writes a test_infra_mark audit record and refuses if a marker already exists.
• Every signing key is local to this hub: no --session-secret / SCION_SERVER_SESSION_SECRET and no pre-configured key. This covers user and agent tokens, OIDC, signed download and artifact URLs, list cursors, conduit stream grants and the web session cookie. On a hub that predates hub-id key scoping, delete or rotate its hub signing keys before the first start with the flag: a key copied forward from a legacy record keeps its original bytes.
• No federation is configured, in the settings file or in stored settings.
• The hub is not a workstation hub, and was not started with --production (use --hosted).

--enable-test-super-admin-identities is refused by this release.

While the gate is on:

• Federation settings writes are refused (422) for every caller.
• Only the hub’s co-located Runtime Broker and the hub-managed message-broker plugin records this process provisions may register or connect. Other brokers get 403 (test_admin_tier_remote_broker_refused), including HA and self-managed plugins. Startup fails if any other broker record is registered and not offline, including a stale plugin record left online; remove it first.
• The hub writes startup lines tagged [scion-test-admin-audit] to stdout, whatever server.log_level is set to: the gate inputs (kinds, hub id, class, marker time, signing-key source labels) and, once brokers are up, the admitted broker ids.
• The scion.io/plugin broker label is reserved for the hub: broker registration and update return 422 when it is sent.

On every hub, whether or not the gate is on, the scion.io/plugin broker label is reserved: a client-supplied value is dropped on broker registration and update, and a value the hub set is kept.

GET /api/v1/test-infra/status reports testHubAdmin: true while the gate is on (testSuperAdmin stays false in this release), and the web banner switches to its admin variant. GET /api/v1/admin/health/summary reports hub.test_admin_identities_enabled, hub.test_super_admin_identities_enabled and hub.live_test_admin_identities.

Test admin identities

On a hub started with the test admin gate, a super-admin can issue a short-lived test identity that holds the real hub-admin role:

• POST /api/v1/test-identities/admin with {"adminRole": "hub-admin", "purpose": "<8 to 200 characters>"} and optional lifetimeSeconds (at most 3600, default 3600) and tokenTtlSeconds (at most 1800, default 900). It returns 201 with the identity (kind: "test_hub_admin", an email in scion-test-hub-admin.invalid, issuanceId, expiresAt) and exactly one access token: no refresh token, no cookie, and no re-issue (POST /api/v1/test-identities/{id}/token answers 409). adminRole: "super-admin" answers 404 in this release.
• The caller must be a human user (not a test identity) who is, at that moment, an active super-admin through their own role bindings and holds test_identity.issue_admin (super-admin only). An access token is accepted only with a hub boundary, exactly the scope test_identity:issue_admin, and an age of at most 8 hours; it is loaded and checked again on every issuance, so revoking the token or the user’s super-admin role stops the next issuance.
• Limits: one live admin identity per issuer, two per hub; six issuances per issuer per hour.
• The identity’s hub-admin binding expires with the identity. While the identity lives it has exactly a human hub-admin’s authority. It cannot issue test identities, create access tokens, change role, or receive other role bindings, group or project memberships.
• Its token stops working when the identity expires, is suspended (403) or deleted (401), and when the hub restarts without the gate. Authorization grants it nothing in those cases, even on a path that does not read the user record.
• Each issuance writes a test_admin_issue audit record (issuer, the issuer’s access token ID, kind, purpose, expiry, issuance ID; never the token) and a [scion-test-admin-audit] line. Every mutation the identity makes carries credential labels test_tier, kind, issuer_id, issuer_credential_id, issuance_id and purpose_hash, and the correlation ID <issuance id>:<request id>.
• POST /api/v1/test-identities/admin/revoke-all (a human super-admin on an interactive session only) deletes every admin test identity. An identity that owns agents or is a project’s last owner is suspended and loses its role bindings instead. The response lists each identity and its outcome. Then restart the hub without the gate.

Admin (/api/v1/admin)

• GET /roles: List Role Definitions.
• POST /roles, PUT /roles/:id, DELETE /roles/:id: Manage Role Definitions (requires appropriate administrative capabilities). Note that updateRoleDefinition includes a CanDelegate check to prevent privilege escalation.
• GET /role-bindings: List Role Bindings (paginated).
• POST /role-bindings, PUT /role-bindings/:id, DELETE /role-bindings/:id: Manage Role Bindings. On POST, a user principal must be an email or a well-formed user ID and an agent principal a well-formed agent ID; a malformed ID answers 400 invalid_request. A UUID sent in upper case is stored in canonical lower case. POST /api/v1/projects/:id/members applies the same check.
• GET /limits: List Limit Definitions.
• GET /limits/:id, PUT /limits/:id: Inspect or update a Limit Definition.
• GET /entitlements/:id: Inspect an Entitlement Binding.
• GET /gcp-quota: View GCP quota status.
• GET /profiling, PUT /profiling: Read and change the profiling switches (requires hub.config.update, on an interactive session only: every user access token is refused with 403). Response: {"readiness_marks": true|false, "revision": N}. PUT body: {"readiness_marks": true|false|null}; null resets the key to its default (off) and an omitted key is unchanged. See Profiling.
• GET /experiments, PUT /experiments, DELETE /experiments: Manage hub-wide experiment overrides (requires hub.experiments.update). GET lists every registered experiment with its default, stored override, and resolved enabled value, plus the section revision. PUT body: {"overrides": {"<name>": true|false|null}, "expected_revision": N}. null removes an override, omitted names are unchanged, and unregistered names return 400. A stale revision returns 409 (revision_conflict). DELETE resets all overrides to registry defaults (body: expected_revision, or confirm_reset_malformed: true if the stored section is malformed). There is no admin UI yet.
• GET /messaging/divergence: View a read-only snapshot of migration divergence counters and metadata for the conversation model transition (requires hub.diagnostics.read permission).

The Hub seeds a max_agents_per_broker limit (default 100) that caps how many agents can be running on a single runtime broker. It is checked before an agent is created, and again when an agent is started, resumed, or restarted. Only running agents count: stop, suspend, and exit release an agent’s slot. The Hub reconciles stale reservations at startup and hourly. This default is a single hub-wide value shared by every broker on the hub; to override it for one broker, set that broker’s maxAgents setting instead — see Broker Settings above.

To change the hub-wide value, PUT /limits/:id on the max_agents_per_broker limit definition. PUT replaces the whole definition, so send the current name, resourceType, unit, and description (for example, from GET /limits/:id) along with the new defaultValue — omitting description clears it. For this system-seeded limit, name, resourceType, and unit must be sent unchanged; changing any of them returns 403. This is also the recommended step after deploying a single-node Cloud Run hub — see the Cloud Run operator docs for the recommended value for that tier.

The Quota System API enforces fail-closed limits. Route guards strictly separate read and write permissions, preventing arbitrary modification of system limits.

Runtime Broker API

The Runtime Broker exposes a local API (usually on port 9800) for agent execution and management.

Control Channel (WebSocket)

Brokers maintain a persistent outbound WebSocket connection to the Hub. The Hub uses this tunnel to send commands (e.g., CreateAgent) to brokers that might be behind NAT.

Local Endpoints

• GET /healthz: Basic liveness and readiness check. In multi-node or hosted setups, if a reverse proxy (like GFE) intercepts this endpoint and returns a non-JSON body, the client detects this and returns a precise error naming the likely cause (rather than a generic JSON-decoding failure) to assist with troubleshooting. If the Runtime Broker’s default runtime failed to resolve at startup, the body reports the runtime check as unavailable and the overall status as degraded, but the endpoint still returns 200 (it is a liveness check).
• GET /readyz: Readiness check. Returns 503 while the default runtime is unavailable. Point readiness probes here and liveness probes at /healthz.
• POST /api/v1/agents: (Internal) The Hub dispatches agents to this endpoint.
• GET /api/v1/agents/:id/attach: (WebSocket) Provides a terminal stream for interactive sessions.

PTY close codes

The WebSocket close frame that ends a terminal attach carries a code that tells the client why the attach ended. The hop that knows the cause picks the code, and every later hop passes it through unchanged. The Runtime Broker classifies the cause the same way on every runtime. The close frame also carries a machine-readable snake_case reason that names the specific cause. A code can carry several reasons, so clients should decide whether to retry from the code and treat the reason as diagnostic detail.

Code	Meaning	Reasons	Client should
1000	Clean detach. The tmux session still exists.	None (empty reason)	Not retry
4404	The Runtime Broker cannot find the agent or its container, or (Conduit agent path) a re-check found the agent is gone.	agent_not_found, target_not_found (Conduit agent path)	Not retry
4410	The tmux session is gone (agent exited, container stopped or removed).	session_ended (the container still exists), container_removed (the container is gone too), agent_stopped (the attach never reached a tmux session because the container is definitively stopped)	Not retry
4503	A hop behind this one is temporarily unavailable (Hub-to-broker control channel dropped, stream failed to open, exec transport dropped while the session is still alive, tmux session not ready yet, container runtime lookup failed).	From the Hub: broker_disconnected, stream_open_failed, broker_write_failed. From the Runtime Broker: runtime_stream_dropped, session_not_ready, lookup_unavailable, runtime_unavailable	Retry
4504	A transient failure on the conduit agent path (the agent’s session was lost, or the stream could not be opened in time).	upstream_unreachable	Retry
1006	Connection dropped without a close frame. The client library generates this code; it is never sent.	None	Retry
1011	Unexpected server error, or the Runtime Broker could not check whether the tmux session survived.	internal_error, client_read_failed, client_write_failed, probe_failed	Retry
1013	The client read terminal output too slowly. On the control-channel path, the Hub holds up to 8 MiB of output for each attach stream; when that fills, it closes only that stream, so other streams on the same Runtime Broker connection keep flowing.	slow_consumer	Retry
4401	Conduit agent path only: a re-check found that the user’s access to the agent was withdrawn, or the user’s authorization interval ended.	authz_expired (re-check), unauthenticated (relay refusal)	Not retry
4403	Conduit agent path only: the stream for the attach was refused when it was opened (the user may not attach to the agent, or the relay or the agent refused the stream).	forbidden, grant_invalid	Not retry

On the Runtime Broker path, authentication and permission failures at the start of an attach surface as HTTP 401/403 on the handshake. On the Conduit agent path, a refusal when the stream is opened arrives as a 4403 close after the upgrade, and 4401 is sent only after the attach has started, by a re-check. 4400 (protocol error), 4409 (superseded, session scope only) and 4499 (open cancelled) are also classified, as terminal, so clients do not retry them.

scion attach decides from these codes using the same classification (wsprotocol.ClassifyPTYClose). Only 1000 counts as a clean detach. After a 4503 close the CLI reconnects once by itself, after a random delay of up to 5 seconds; after a 4504 or 1011 close it reconnects once after exponential backoff with random jitter (wsprotocol.PTYReconnectTiming). It makes one reconnect per such close, and stops after 3 reconnects in a row whose sessions each ended within a minute. Typing Ctrl-C during the wait stops it. The reconnect sends the current terminal size so the screen redraws. If that reconnect fails, or for any other code, it exits non-zero with a message that matches the code, for example “run scion attach again” for a retryable code, or scion resume for 4410.

System Health Endpoints (Hub)

• GET /healthz: Basic liveness check. If a reverse proxy intercepts this with a non-JSON response, the client gracefully falls back to /health. Always returns HTTP 200; the response body’s top-level status field reports healthy; degraded when only non-critical checks are failing (the server is up and serving); or unhealthy when a critical check (database, or workspace_storage when a non-local workspace storage backend is configured) is failing — the same storage condition that makes /readyz return 503. The workspace_storage_mount_verification key (mount could not be verified) only degrades. The failing check(s) are named under checks. In the combined web + Hub response, the top-level status is the most severe of the Hub’s and the co-located broker’s. When this process runs a co-located (embedded) runtime broker, the co-located broker check reports healthy, unhealthy: registration failed (the fixed public value — see the server log for the underlying error), or unhealthy: registration pending; status goes degraded while it is anything but healthy. A failed registration is not retried, so this needs a broker configuration fix and a server restart, not a wait. Path differs by deployment shape: on a standalone Hub (no web server, e.g. port 9810), it is checks.colocated_broker at the top level. On the combined single-node workstation setup (the default — the web server answers /healthz on its own port and nests the Hub’s health under hub, per CompositeHealthResponse), it is hub.checks.colocated_broker.
• GET /readyz: Readiness check verifying database connectivity and, when a non-local workspace storage backend is configured, that its mount is available. Kubernetes and Cloud Run readiness probes must target this endpoint rather than /healthz, which always returns 200. /readyz is intentionally unaffected by the co-located broker check above.
• GET /health: Legacy/alternative liveness check endpoint.

Communication Patterns

State Reporting

Agents use the sciontool utility to report their state back to the Hub via the POST /api/v1/agents/:id/status endpoint. State updates include the agent’s current phase, activity, and contextual detail (e.g., which tool is executing). This happens at high frequency during task execution.

The endpoint answers 200 OK with {"applied": true} when the update was written. It answers {"applied": false, "reason": "..."} when the Hub accepted the report but wrote nothing because the agent’s status is owned by another operation. reason is either delete_in_progress (the agent is being deleted or is soft-deleted) or reincarnation_in_flight (a scion reincarnate migration owns the agent). While a delete or reincarnation owns the agent, a report that mixes those owned fields (phase, activity, message, exitCode, exitReason) with other fields such as taskSummary has the owned fields dropped and the rest written, and the answer is applied: true. applied: false is only returned for those two reasons: other guards (a suspended agent, a phase regression) drop phase/activity silently and still answer applied: true, so it does not mean the reported phase was stored.

Log Streaming

Logs are collected by the Runtime Broker and can be streamed in two ways:

1. Real-time: Streamed via WebSocket from the Broker to the Hub, then to the Dashboard/CLI.
2. Persistent: Batched and uploaded to a storage backend (like GCS) after agent completion.