API Reference

The Scion ecosystem exposes several APIs for coordination, management, and observability. This reference provides an overview of the primary resource types and communication patterns.

Hub API

The Scion Hub provides a RESTful API (mostly JSON) for managing the state of the system.

Authentication

Most endpoints require a Bearer token in the Authorization header.

• User Tokens: Obtained via OAuth or Dev Auth.
• Agent Tokens: Issued to agents at startup for state reporting.
• Broker Tokens: Used for broker-to-hub communication, often combined with HMAC request signing.

Pagination

List endpoints for templates, harness configs, groups, artifacts, starred messages, and skills (when called with a user access token) return an opaque nextCursor. Cursors are encrypted and bound to the endpoint, the filter and the caller that received them, so pass them back unchanged to the same endpoint with the same filter. A cursor that is malformed, tampered with, reused in a different context, or sealed with a key the Hub no longer holds (for example after a key rotation) is rejected with a uniform 400 Bad Request and error code invalid_cursor; restart the listing from the first page. With a server keyring (server.keyring_file), a cursor stays valid only while the signing generation that sealed it is before its verify_until. A cursor has no lifetime of its own, so retiring a generation ends every cursor it sealed, and the next request gets invalid_cursor. Every other paginated list endpoint answers a cursor or pageToken it cannot use (malformed, issued for another list, or naming an entry that no longer exists) the same way: 400 with error code invalid_cursor and the message invalid cursor: restart pagination from the first page.

Core Resources

Agents (/api/v1/agents)

• GET /: List agents (filterable by project, user, phase). Add sort=created or sort=updated for a server-sorted, keyset-paged view with the same dir, limit (clamped to 500), cursor, fit and stats parameters as the project agent list’s sorted mode below. Unlike the project endpoint, the global endpoint has no 2,000-candidate ceiling and never refuses a sorted request with 422 sorted_view_unavailable. For a user caller above 2,000 candidate agents, totalCount is a lower bound and the response sets totalCountApproximate, a page can be shorter than limit (follow nextCursor), and with stats=1 the stats.total and stats.running counts are lower bounds (stats.totalApproximate) and stats.agents is omitted. For an agent caller, stats.agents is omitted when stats.total exceeds 2,000. Without sort the response is unchanged. Add view=compact for slimmer items: each agent carries only its identity, template, project, status, labels, lineage, message mode, creator name, harnessVendor, modelSize (the canonical model size alias small, medium, large, or extra-large when the agent’s model came from a size alias, otherwise omitted, or "" on status events; harnessVendor is the harness’s model vendor, omitted for a custom or unresolved harness; both fields are also included in view=full, on GET /:id, and on agent created and status SSE events) and timestamps, plus _capabilities and _messageability, and no appliedConfig. The result has the same agents in the same order, with the same cursors, counts and other response fields as the default view=full. An empty view is the same as no view. With sort, a non-empty view other than full or compact is rejected with 400; without sort, any value other than compact is ignored.

• POST /: Dispatch a new agent. Also available as POST /api/v1/projects/:id/agents. A create that is dispatched to the Runtime Broker synchronously (including provisionOnly) answers 409 with code delete_in_progress when the agent is deleted, or a delete holds it, before the create completes. This includes a create that the Runtime Broker answers with 202 to gather environment variables, a workspace-bootstrap create (which then returns no upload URLs), and a managed-agent create. The calls that finish a create, POST /:id/workspace/sync-to/finalize and POST /api/v1/projects/:projectId/agents/:agentId/env, answer the same way when a delete wins after the Runtime Broker started the agent. The body carries no agent: error.details.agentId names it, and error.details.warnings, when present, may include other dispatch warnings, such as whether a container the create had already started was removed, or whether a managed-agent interaction it had started was stopped. Treat the agent as not created. If a delete that was still in progress later fails, the agent remains, and you can delete it again. A managed-agent create whose agent the Hub cannot record after the managed-agent interaction started answers 500 with code internal_error instead of 201. The Hub stops the interaction and removes the agent. error.details.agentId names the agent, and error.details.warnings reports whether the interaction was stopped. When the stop failed, the warning names the agent and the interaction so that an administrator can stop it by hand. If removing the agent did not complete, error.details.correlation_id is set. Treat the agent as not created; you can create it again with the same name. A create accepted for asynchronous launch still answers 201. When the synchronous dispatch to the Runtime Broker ends without an answer (the Hub’s 120-second dispatch timeout, or no response to a request it sent), the create, and the env submit POST /api/v1/projects/:projectId/agents/:agentId/env, answer 504 with code start_outcome_unknown. error.details.agentId and error.details.phase describe the agent. The Hub keeps the agent rather than removing it, because the Runtime Broker may still be starting it. The agent’s message says so. The agent moves to running if the start completes. If the Runtime Broker shows nothing running, the Hub revokes the agent’s credential and stops waiting for the start; an agent still in the created phase also returns to rest. You can delete the agent at any time.

  • A create in a shared-workspace git project sent to a remote Runtime Broker that has no local path for the project, and that does not report the Hub’s shared workspace storage (nfs), fails before dispatch with 409: the Hub’s workspace is on storage that Runtime Broker cannot reach. On a Hub with shared workspace storage the same applies to a project with no git remote. On a Hub with local workspace storage, a linked project with no git remote (one registered from an existing checkout) runs on such a Runtime Broker with the Runtime Broker’s own copy of the workspace, which the CLI workspace upload and scion sync to fill; the Hub’s Files tab answers 409 for such a project. A project created on the Hub with no git remote is refused with 409 like a shared-workspace project. Clone-per-agent and empty-per-agent projects are unaffected.
  • Creating an agent on a flat Runtime Broker (scion.io/broker-role=flat) with a profile fails before anything is written with 422 runtime_profile_unsupported, regardless of the profile name.

• POST /stop-all: Stop every running agent across all projects. Requires a user identity holding agent.stop_all on the hub; a user without it gets 403. For the project-scoped variant, see POST /api/v1/projects/:id/agents/stop-all below.

• GET /:id: Get detailed agent state (phase, activity, detail). An agent can always read its own record with its agent token. An agent can also read an agent it launched, directly or through its descendants, in the same project (for example, to wait for a child’s launch). This covers only this single-agent read: a descendant’s logs, message logs, event streams and workspace stay closed to it. For an agent caller, an agent it may not read answers 404, the same as a missing agent. For any caller without agent.attach on the agent, and for every agent delegated credential (on GET /:id, GET /, GET /api/v1/projects/:id/agents, and mutation responses), appliedConfig.env, appliedConfig.inlineConfig.env, appliedConfig.inlineConfig.telemetry.cloud.headers, every appliedConfig.inlineConfig.mcp_servers[*].env and .headers, every appliedConfig.inlineConfig.services[*].env, and the same maps under appliedConfig.createInputs.inlineConfig, pendingConfig.config and effectiveNextRun are left out of the response; the MCP server and service entries themselves stay listed. The response carries editability: phase, and fields, which maps every agent form field and every config key to its entry for this agent and caller. Fields are keyed by their PATCH key: config.<key> for a key of config (for example config.model), else the top-level key (name, labels, agentRole, gcp_identity, explicitTimezone, template, …). Each entry has tier, what re-applies an edit: T0 immediate metadata (name, labels, annotations, task summary, message mode), T1 container creation (applied by any start, restart, resume or reincarnation: model, thinking level, harness auth, image, user, limits, resources, env, telemetry, MCP servers, volumes, kubernetes, command args, task, timezone), T2 provision-rendered (applied by a reincarnation: system prompt, agent instructions, skills, services, secrets; when the agent’s Runtime Broker advertises changedFields, all but secrets are also applied at the next start, and their disposition is now in the phases with no container), T3 principal (applied only by an explicit reincarnation: role, GCP identity), or TX fixed (project, template, harness config, Runtime Broker, profile, branch and workspace). disposition is what happens to an edit in the agent’s phase: immediate, now (applies at the next start or resume), held, reincarnate (applies at the next reincarnation), or locked, with reason saying why. sessionSensitive marks a field (model, thinking level, harness auth) whose change a resume applies while the harness continues its conversation, and note is "session" on such a field of a suspended agent. clearNeedsReincarnate marks a field whose clear or zero value applies only at the next reincarnation; it is not set on max_duration, thinking level, resources and env when the agent’s Runtime Broker advertises clearedFields. It stays set on max_turns and max_model_calls, whose 0 (Unlimited) no start applies. With changedFields, it is set on agent_instructions, skills and services: a start does not apply a clear of those. It is set on system_prompt unless the Runtime Broker also advertises clearedFields. For a suspended agent, note is also "session" on the four re-rendered keys. Unless the hub.agent_edit_held experiment is on, editing a running or transitional agent’s config is refused, so every config.<key> of such an agent is locked, provision-rendered keys included; agentRole and gcp_identity stay reincarnate. With the experiment on, a container key of such an agent is held (saved now, applied at the next start, restart or resume) and a provision-rendered key is held when the agent’s Runtime Broker advertises changedFields (system_prompt, agent_instructions, skills and services, applied at the next start), and otherwise reincarnate (saved to pendingConfig, which a reincarnation does not apply yet; see PATCH /:id); secrets is always reincarnate. agentRole is locked in created and for a caller who cannot delegate the agent’s role or whose credential’s scopes do not cover it (it changes through reincarnate); every field is locked for a caller without agent.update and for a deleted agent. An agent placed on a flat Runtime Broker (one that serves a single runtime target) also carries a read-only pinnedRuntimeTarget view, {"id", "type", "runtimeBrokerId"}: the runtime target it is pinned to and the Runtime Broker serving it. It is absent for other agents and cannot be set through the API. The Runtime Broker object (GET /api/v1/runtime-brokers/:id) carries the matching runtimeTarget descriptor (id, type, displayName). With the hub.conduit experiment on, the response also carries sshHostKey: the public SSH host key (an authorized_keys line) the agent’s current run reported for its embedded SSH server, for pinning in known_hosts when you use scion tunnel --ssh. Each run generates a new key; it is omitted when none was reported. When the agent has pending config edits (see PATCH /:id), the response carries pendingConfig: config, the pending keys as the edits sent them (a null value clears the key when it is applied; env values, the telemetry cloud.headers, every MCP server’s env and headers and every service’s env are shown only to callers who may attach to the agent, as in appliedConfig), savedBy (user:<id> or agent:<id>), savedByName, savedAt, baseGeneration and baseStateVersion (the agent’s generation and stateVersion the last save was made against). ?view=edit adds effectiveNextRun: the applied config with every pending key applied, as the agent’s next run would get it (container keys at the next start, and on a Runtime Broker that advertises changedFields the re-rendered provision keys too; the other provision-rendered keys are not applied by any action yet), without createInputs. For a caller who can view the agent’s env (the same check that decides whether appliedConfig.env is shown), the response also carries resolvedTimezone and timezoneSource, with the same values and meaning as on the PATCH /:id response; for other callers both are omitted. The project-scoped GET /api/v1/projects/:id/agents/:agentId returns the same body.

• POST /:id/stop: Stop a running agent. The Hub records that the agent should be stopped (its run intent, kept separately from the observed phase) before dispatching. If the agent’s Runtime Broker is offline, the stop is queued instead: the response is 202 Accepted, the agent is marked stopped with container status stop_queued, and the Runtime Broker applies the stop when it reconnects, unless a newer start or stop was recorded in the meantime.

• Start, stop, suspend, and restart respond with the agent record itself: its stored fields at the top level, not under an agent key, plus an optional warnings array that is omitted when empty. The capability fields that GET /:id adds (_capabilities, _messageability, resolvedHarness, harnessCapabilities, and cloudLogging) are not included, so fetch GET /:id to refresh them. Suspend never includes warnings. A 202 means the action was queued, not performed. For example, a stop queued for an offline Runtime Broker returns a body like this (abbreviated; other agent fields omitted), where message and the warning carry the same text:

  {
    "id": "<uuid>",
    "slug": "reviewer",
    "phase": "stopped",
    "containerStatus": "stop_queued",
    "message": "Stop queued: broker offline; it will be applied when the broker reconnects.",
    "warnings": ["Stop queued: broker offline; it will be applied when the broker reconnects."]
  }

• POST /:id/suspend: Suspend a running agent, preserving its harness session for a later resume. Sets the phase to suspended. Requires a harness that supports session resume.

  The stop, suspend, both stop-all routes, and workspace sync-from operations run to completion on the Hub even if the client disconnects. Their response write deadlines are extended for the duration of the operation, so a slow stop that succeeds is no longer reported as a 502. Sync-from answers 502 when the Runtime Broker’s reply carries no file manifest.

• POST /:id/start, POST /:id/restart: Start/restart an agent. Starting a suspended agent resumes (continues) its harness session; starting a stopped or error agent runs a fresh session. To continue the interrupted session of an error agent instead, send {"forceResume": true} as the start body (best effort). forceResume has no effect in other phases. When the hub.async_agent_start and hub.async_agent_launch experiments are both on and every live Hub instance supports it, send {"acceptAsyncLaunch": true} in the start or restart body to have the Hub answer as soon as the Runtime Broker accepts the start: 200 with the agent in phase starting and an active launch (see Asynchronous agent start). When a delete races the start, the answer is 409 with code delete_in_progress and no agent body; error.details.agentId names the agent. This happens when:

  • a delete holds the agent, or wins while the start is in flight. This includes a delete that wins after the Runtime Broker has started the agent; the Hub then tries to remove that container again, and error.details.warnings reports the outcome.
  • a delete claims the agent after the start but before the Hub’s final read of the agent, so the start does not answer 200 for an agent whose delete has won. This includes an agent on a managed runtime, which the Hub starts without a Runtime Broker: a managed agent that the delete has already removed also answers 409, not 404.
  • a start or resume of an existing agent through POST / meets either of the cases above.

  A delete that has failed, or whose lease has expired without finalizing, leaves the agent live, and the start answers 200. A delete that claims the agent after the Hub’s final read is a separate, later operation. Every start, restart, and resume runs under a start claim: while another start of the agent holds it (a user start, a restart, a start for a direct message, creation, automatic recovery, a reincarnation, or a queued stop being applied), the answer is 409 with code start_in_progress, and error.details carries holderKind, state, since, and expectedRelease. If a restart cannot stop the agent’s current instance, it is aborted without starting a second one and answers 503 with a Retry-After header (error.details.brokerCode names the Runtime Broker’s error code when there is one); retry later.

  Fail closed on unresolved credentials. Create, start and restart answer 503 unavailable when the Hub cannot resolve the agent’s configured secrets (agent secrets could not be resolved; the agent was not started) or read its stored environment settings (agent environment settings could not be read; the agent was not started; a create says the agent was not created). A create or start sends nothing to the Runtime Broker: a start leaves the agent as it was, and a failed create is rolled back. A restart stops the agent first, so a restart that fails this way leaves the agent stopped. A provision-only create still succeeds; the agent’s later start fails closed. Resuming an agent and waking it for a direct message fail the same way. A reincarnation still answers 202, then fails at its start leg: the reincarnation is recorded as failed with the same message (start failed: …) and the previous config is restored. The message names no secret, value or backend detail. An agent with no secrets or settings is not affected. A Runtime Broker whose env-based harness auth check fails rejects the start before any container is created; see Agent Credentials.

• POST /:id/reincarnate: Migrate the agent to a new generation with the same ID and slug and a freshly resolved config (see scion reincarnate). Body: handoff (optional text for the new generation’s first task, max 256 KiB) and dryRun. The body can also carry the patch fields serviceAccount, role, model, thinkingLevel (integer 0–100), harnessAuth, and image, each of which changes that setting for the new generation and later ones (a user caller also needs agent.update; the plan names them in patched and shows each patched field’s old and new value). serviceAccount takes the account’s id, email or display name, resolved as for create (an email prefers the project’s own account over a hub-wide one; more than one match returns 400 identity_ambiguous listing the candidates’ ids and scopes). A disallowed GCP service account is refused before the agent is stopped. For a legacy agent created before createInputs was recorded, any remaining inline env keys are kept as explicit for the new generation, with a warning in plan.warnings. Returns 202 Accepted with the pending plan, or 200 OK with the plan only for a dry run. The migration runs in the background; its first step clears the previous run’s exitCode and exitReason. Requires agent.lifecycle; returns 400 for agents in worktree-per-agent projects. An agent in an empty-per-agent project can be reincarnated on its current Runtime Broker only on a local-disk runtime (Docker, Podman, or Apple container), reusing its workspace in place: any other runtime returns 400, and a Runtime Broker that does not advertise the reprovisionEmptyPerAgent capability returns 412 unsupported_capability. Both are checked before the agent is stopped, so a dry run returns the same answer. If the workspace directory is missing or is not a real directory, the Runtime Broker refuses the reprovision and the migration fails; it never recreates the workspace. Also available as POST /api/v1/projects/:projectId/agents/:agentIdOrSlug/reincarnate. Optional targetBroker (Runtime Broker ID, name, or slug) and forceMove (accepted only when targetBroker names a different Runtime Broker; 400 otherwise) check or run a move to another Runtime Broker. targetBroker is resolved among the Runtime Brokers the caller can see: an unknown or hidden target returns 404 runtime_broker_not_found, and a name that matches more than one returns 409 runtime_broker_ambiguous. A target equal to the current Runtime Broker is a plain reincarnation. For a different Runtime Broker, a dry run evaluates the move checks in order (runtime_target, workspace_mode, workspace_storage_reported, same_export, workspace_on_export, workspace_recreatable, target_profile, target_health, access, capability, capacity) and returns 200 with the plan, targetBrokerId, and moveVerdict (carrying transferMode — export or recreate — and consequences, what the move keeps, carries, recreates and loses), or the first refusal (400, 403, 409, 412, 429, or 503) with the full verdict in error.details.verdict (and, when a request without forceMove is refused only because the workspace is not on a shared export and a forced move onto that target could apply, forceWouldApply: true). runtime_target refuses (409 runtime_target_move_unsupported) an agent pinned to a flat Runtime Broker and a move onto a flat Runtime Broker. A real move (no dryRun) runs the same checks and returns 202 (with transferMode set to export or recreate) to move the agent in the background: quota moves to the target, a failure before the new generation is running normally rolls the agent back to the source (stopped, in the error phase; on a recreate move, the rollback removes only target directories marked with that move’s reincarnation ID and never touches the remote git branch or the source’s workspace), and removing the source’s local state after a completed move never touches the shared export. See Moving an agent to another Runtime Broker.

• POST /:id/message: Deliver a message to an agent. Body: message (plain text) or structured_message, plus optional flags plain (deliver the text without the Scion message envelope), interrupt, notify, and wake. Top-level plain applies with either body form; with structured_message, it is merged onto it. The whole body is capped at 2 MiB, checked before authorization (413 payload_too_large, no operation_id). Raw keystroke delivery through this route has been removed: see “Retired raw field” under /:id/keys. When the Hub cannot read an attachment that the message names (most often a file staged on a different Runtime Broker’s host than the Hub’s), it still delivers the message without it and lists the file in attachment_warnings, an array of {"path", "reason"} objects; the field is omitted when every attachment was recorded. Direct-message and outbound-message responses carry the same field. Behind the hub.agent_delegation experiment, an agent delegated credential is admitted on POST /:id/message and POST /api/v1/projects/:projectId/agents/:agentIdOrSlug/message when both the grant (agent.message on the target) and the actor agent’s own messaging evaluation (message modes, cross-project policy, and standing) allow; the issuer’s ownership or ancestry never pierces a message mode. A delegated send is limited to a direct text message (instruction, chat, or reply) to the route’s own agent (no group recipient, thread, conversation, mention, notification subscription, or interrupt), is charged to the actor agent’s send budget, and records the actor agent as sender (agent:<slug>) and senderId with authorizingUserId and delegationGrantId stamped on the stored message.

• POST /:id/keys: Send literal terminal input to an agent’s tmux session — the replacement for the removed message/raw keystroke path. Also available project-scoped as POST /api/v1/projects/:projectId/agents/:agentIdOrSlug/keys. The whole HTTP body is capped at 32 KiB, rejected with 413 before anything else is parsed. Body: {"keys": "<string>"} only — unknown fields, duplicate keys, a non-string/missing/empty value, an embedded NUL byte, invalid UTF-8 or an unpaired UTF-16 surrogate escape, and a value over 4096 UTF-8 bytes are all rejected (400/413). The string is delivered verbatim as a single tmux send-keys argument: a recognized tmux key name (Enter, Escape, C-c, arrow names, etc.) is interpreted only on an exact whole-string match; anything else — including text containing spaces — is typed literally, character by character. There is no sequence syntax: "Up Up Enter" is eleven literal characters, not three key presses — issue separate calls for separate key presses. An empty string is rejected outright and never becomes Enter; whitespace-only input is valid and preserved exactly.

  Response: {"status": "dispatched", "operation_id": "<uuid>", "agent_id": "<uuid>"}. A 200 means the Runtime Broker acknowledged terminal injection, not that the harness consumed or acted on it. operation_id correlates audit and error records; it is not an idempotency key, and there is no 202/queued state. Errors use the standard Hub error envelope with code set to the machine outcome (invalid_request, payload_too_large, keys_denied, not_found, agent_not_running, terminal_not_ready, cross_project_keys_unsupported, keys_unsupported, keys_rate_limited with a Retry-After header, keys_unavailable, or keys_outcome_unknown), or a generic 500 for an unexpected internal failure (which still carries operation_id, since validation already succeeded by the time one could occur); details.operation_id is present on every outcome decided inside the handler, from target resolution onward — everything except unauthorized (decided by auth middleware before any handler runs), the two validation failures above, and, on the project-scoped route specifically, its shared project-resolution 404 (decided before any keys-specific code runs at all, for every action on that route, not only keys).

  Audit. Every request that reaches the keys handler — accepted, denied, or malformed — writes content-free records to the Hub instance’s structured log: timestamp, operation_id (when one exists), actor type and ID, source and target project, target agent, credential kind and ID, the route (keys), the input’s byte length, the decision code, and duration. Every request admitted to dispatch writes two records — an admission record before dispatch and an outcome record after, whatever that outcome turns out to be, including an ambiguous keys_outcome_unknown; a request refused before dispatch writes only the single outcome record. Anything decided before the keys handler runs at all — a 401 from the shared auth middleware, a 405 for a non-POST method, and, on the project-scoped route, its shared project-resolution 404 — is not keys-audited: do not expect unauthenticated probing, a wrong HTTP method, or an unresolvable project to appear in this log. A validation-stage record (before an operation_id exists) reports the raw request body’s byte length under a separate field instead of the decoded keys length — reported as 0 when the body exceeded the 32 KiB cap or could not be read, since no byte count is known in that case either. An unexpected internal failure is recorded with an error_class field instead of a decision code, since it isn’t one of the machine outcomes above. The record never contains the keys value itself, any preview or hash of it, or terminal output. This is a log line emitted per Hub instance, not a queryable audit store or API.

  Immediate-route limitations. This operation never wakes, starts, or queues anything — it either reaches the target’s terminal right now or it fails: a stopped or suspended target answers 409 agent_not_running rather than being started; a running target without a ready terminal session answers 409 terminal_not_ready. 503 keys_unavailable means dispatch definitely did not start at all — an offline Runtime Broker, no immediate synchronous route to it (including a cross-Hub forward with no direct transport), or an admission window (30s by default) that already expired before the call could begin; nothing is queued for later delivery, so a caller may reasonably resend after confirming the route is fixed. 502/504 keys_outcome_unknown is the opposite case — the call may have reached the terminal and partially or fully executed before the failure — and must never be retried automatically or read as confirmation of delivery.

  Authorization mirrors agent.attach, not message-mode authorization: a human session needs to own the target agent or hold agent.attach on it (closed/none message mode does not block it). No built-in project role grants agent.attach, so being a project owner, admin, or member is not by itself enough; an agent credential needs ScopeAgentLifecycle, the same project as the target, and live attach authority on the target via the relationship evaluator — same-project alone is not sufficient for an agent caller. Cross-project disclosure differs by route, matching the contract’s documented trade-off: on the project-scoped route, an authenticated agent’s project is compared against the URL’s project before any target lookup, so a mismatch answers 422 cross_project_keys_unsupported without ever revealing whether a same-slug agent exists there; on the top-level route, the target is resolved first (the same order every other action on that route already uses), so a foreign existing agent answers 422 while a nonexistent one answers 404 — a disclosure this route’s other lifecycle actions already have today, not a new one. This 422 on the top-level route is conditional on the agent already holding the lifecycle scope checked above: an agent lacking it is refused with 403 keys_denied before the project comparison ever runs, regardless of whether the target is foreign or same-project. The project-scoped route’s 422, by contrast, does not depend on scope — it is decided purely on the project mismatch, before any scope or target check. A human operator with live cross-project permissions is not blocked on either route; the cross-project refusal applies only to authenticated agent identities. No conversation, message row, SSE event, observer/mention fan-out, or notification is created for any outcome. Requests are rate-limited per authenticated principal+project (5 req/s, burst 10) and per target agent (10 req/s, burst 20), Hub-instance-local. Delivery is single-attempt end-to-end — no SDK retry, redirect replay, or routing fallback — so a caller must not resend a call whose outcome is unknown. A managed-runtime target or a Runtime Broker without the keys route answers 422 keys_unsupported; this is never downgraded to an ordinary message send.

  Retired raw field (migrating from raw). Raw keystroke delivery through messages has been removed; use this /:id/keys operation (or scion keys) instead. A request that still carries the retired raw field is rejected with 422 raw_input_removed before anything happens — no message or conversation row, no dispatch, no event, no notification. This applies to both spellings (top-level raw and nested structured_message.raw, or message.raw on the broker inbound routes), to every value including false and null, and to malformed values. It applies on these routes:

  • POST /api/v1/agents/:id/message and POST /api/v1/projects/:projectId/agents/:agentIdOrSlug/message;
  • POST /api/v1/projects/:projectId/broadcast (body capped at 2 MiB, 413 payload_too_large);
  • POST /api/v1/broker/inbound and POST /api/v1/broker/inbound/routed (Message Broker plugins; body capped at 2 MiB, 413 payload_too_large), checked before topic validation and sender resolution;
  • the advanced payload JSON of scheduled events and recurring schedules.

  The error message names scion keys, and details carries operation_id, ingress, and replacement, the generic /keys route template to use instead (POST /api/v1/projects/{projectId}/agents/{agentIdOrSlug}/keys on the project-scoped message route, POST /api/v1/agents/{id}/keys elsewhere). The response never names the resolved target, because the field is rejected before message authorization. Each rejection writes a content-free audit line with the route value message_raw_removed. Plain, normal, and interrupt messages are unaffected, and historical message rows are not rewritten. See Migrating from raw message delivery.

• PATCH /:id: Update an agent’s mutable fields: name, labels, annotations, taskSummary, config, and gcp_identity. Send stateVersion for optimistic locking (409 on mismatch). config (for example model, image, env, thinking_level) is accepted for agents with no container: the created, stopped, error and suspended phases. Each of those phases creates a new container from the applied config at its next start (a first start, a fresh start of a stopped or failed agent, or a resume of a suspended one), so the edit is dispatched with it. For an agent that is running, or whose container is being created or removed, config returns 409 unless the hub.agent_edit_held experiment is on (see below). When an edit takes effect depends on the key’s edit tier (see editability on GET /:id): container keys such as model, the limits, image and env apply at the next start or resume. system_prompt, agent_instructions, skills and services are rendered into files when the agent is provisioned. The Hub records each one a PATCH changes in changed_fields and sends that list with every start. A Runtime Broker that advertises the changedFields capability renders those files again at the start, so for its agents these four keys also apply at the next start or resume, including an agent’s first start from created. The Runtime Broker renders a key again only when its result differs from the last time it rendered it. A change made inside the agent to the installed skills or .scion/scion-services.yaml, or to the prompt files of a harness that reads them from the agent home, is therefore overwritten only when the Hub’s value changes again. Container-script harnesses (such as Claude, Gemini CLI or Codex) still rebuild their managed instruction block from Scion’s inputs at every start, as before. A start does not apply a clear the merge into the stored config keeps: an empty agent_instructions or services value, an emptied skill list, or a removed skill (the merge appends skill references, and a start uninstalls nothing). Those edits are reported in heldForReincarnate with a warning, like the clear of system_prompt on a Runtime Broker without clearedFields. With an older Runtime Broker, these keys are stored now and rendered at the next reincarnation, and a start that carries changed keys to it returns a warnings entry naming them. secrets is always rendered at the next reincarnation. Clearing a container key reaches the agent only at the next reincarnation, except as described for cleared_fields below, because a start keeps the previous value: that covers null or "", an empty map or list, an empty resources, kubernetes or telemetry (or one whose fields are all empty), and 0 for max_turns or max_model_calls. A max_duration of "0" (no limit) and a thinking_level of 0 do apply at the next start or resume. Likewise, removing an entry from env, mcp_servers or volumes takes effect only at the next reincarnation, because a start unites maps and appends volumes. The response’s warnings name these keys when their value actually changes. The Hub also records some of these clears as cleared_fields and sends them with every start: a clear of max_turns, max_model_calls, max_duration, thinking_level, resources or system_prompt, and each removed env key. A Runtime Broker that advertises the clearedFields capability applies them at the start, without a reincarnation, so the inherited value applies again. The inherited value is the one provisioning derived from the agent’s template chain (the agent’s template, with the project or global default template layered under it), or the defaults when the chain sets none; templates never supply resources, so a cleared resources comes from the defaults. This holds both for a template the Runtime Broker resolves by name and for one loaded from its template cache (a Hub template). The Runtime Broker writes that value into the agent’s stored config, so the agent keeps it even if the template is later renamed or removed. If the chain cannot be resolved, for example because the cached template is no longer available or the template name no longer resolves, the agent keeps its current value, so a clear never removes a limit, and the clear applies at the next reincarnation. A removed env key is removed in every case. A cleared system_prompt is rendered again from the template at the start by a Runtime Broker with changedFields; with an older one it waits for the next reincarnation. For an agent whose Runtime Broker advertises clearedFields, disposition lists these clears under applied and no warning is returned, but only when the clear was recorded: a removed env key counts only if the caller can view the agent’s env (the auto-expose keys and TZ, which the Hub derives for each start, are not counted). A count limit sent as 0 (Unlimited) is still reported held for reincarnation with a warning; only a null clear of it is reported applied. An older Runtime Broker ignores the list and keeps the previous value, so for its agents the warnings and disposition describe the reincarnation case. A later edit that sets a key again takes it off the list, and a new agent or a reincarnated generation starts with an empty list. Other clears, such as model, image, or volumes, mcp_servers or telemetry as a whole, are not recorded. A resources value that leaves out a sub-field the agent stores (requests or limits cpu/memory, or disk) keeps that sub-field until the next reincarnation, and is reported in heldForReincarnate with a warning on every Runtime Broker. Neither cleared_fields nor changed_fields is returned by GET. changed_fields is kept until a reincarnation, which renders every key afresh; a new agent starts with an empty list. A supporting Runtime Broker also stops repeating volumes on each start: it keeps one volume per target, a later one replacing an earlier one, except that the first /workspace volume is always kept. config is a partial update: only the keys present in the request body change. A present key replaces its whole value (for example the whole env map or volumes list), and a present key sent as an empty value or null clears it. An absent key keeps its current value, including volumes, skills, mcp_servers, services, command_args, kubernetes, telemetry and env. model, image, task and auth_selectedType sent empty leave the agent’s applied value unchanged (the inline config entry is cleared). thinking_level changes only when it is present, so send "thinking_level": null to unset it. A model value is resolved through the harness’s model aliases. Fields in the request that the agent’s current harness does not support return a validation error. The PATCH enforces the same rule editability on GET /:id reports, before anything is written: a request that names a key locked for the agent is refused whole, and details.fields maps each refused key to the reason. Refusals are checked in this order: any key of a deleted agent (409); without hub.agent_edit_held, any config of a running or transitional agent, whatever the keys (409); config.task of a suspended agent, which a resume does not send (409); a key fixed at creation that would change (400): harness, harness_config, default_harness_config, branch, clone_depth, explicit_workspace, empty_per_agent_workspace, config_dir, detached, hub, cleared_fields and changed_fields (which only the Hub sets); then field validation. A fixed key is refused only when it would change: a value equal to the agent’s stored inline value or its applied value (the branch, harness config and harness shown on GET) is ignored, as is an empty value where no inline value is stored, and an ignored key is not listed in disposition. To use another harness, recreate the agent, or change the template and reincarnate it. Keys config does not define are ignored. gcp_identity can be changed only in the created phase. explicitTimezone pins the agent’s container timezone to an IANA zone name, or unpins it with ""; an invalid name returns 400. It is accepted in any phase except on a deleted agent (409) and applies at the next start (a running container keeps its TZ, and the response carries a warning saying so). A TZ key in config.env is ignored with a warning; use explicitTimezone instead. The response is the agent plus resolvedTimezone (the TZ its next start will get, "" when none is sent) and timezoneSource, which names the step of the agent timezone chain that supplied it: explicit, legacy (adopted from a TZ an older Hub stored in the agent’s env), user, project, hub, broker, progeny, hub-default, or none. The response also carries disposition, which lists the keys whose value the request changed (config keys as config.<key>, for example ["config.max_turns", "name"]), grouped by when each edit takes effect. applied lists the keys that take effect at the next container creation (any start, restart, resume or reincarnation), or at once for metadata. explicitTimezone is also listed under applied when the agent’s container is live, although the running container keeps its timezone until the next start (the response warns about this when the zone the next start gets changes). held lists the config keys kept aside while a container is live or being created or removed, and applied at the next container creation after the current run (with the hub.agent_edit_held experiment on; without it such edits are refused, so it is empty). heldForReincarnate lists the keys that take effect only at the next reincarnation: the provision-rendered keys the agent’s Runtime Broker does not render again at start (all of them without changedFields, and secrets always), the container keys whose clear or zero a start does not apply (the fields editability marks clearNeedsReincarnate), and env, mcp_servers or volumes when the edit removes an entry (for env, without clearedFields, or when the removals were not recorded). Each of these keys is named in a warnings entry that says it takes effect at the next reincarnation. Without the hub.agent_edit_held experiment, a heldForReincarnate key is still stored now, so GET already shows its new value in the agent’s applied inline config; with it, the key is kept in pendingConfig instead (see below). All three lists are always present, sorted, and empty when nothing falls under them. A key sent with its current value is not in any list, with one exception: gcp_identity is listed under applied whenever the request records a service-account assignment (an assign identity, or passthrough translated to an assigned account), even when it names the agent’s current account, because the assignment is re-recorded with the caller as its source. With the hub.agent_edit_held experiment on (off by default), config is also accepted for an agent that is running, or whose container is being created or removed (provisioning, cloning, starting, running, stopping): its keys are kept in the agent’s pendingConfig instead of the applied config, listed under held (or heldForReincarnate for a clear or removal a start does not apply), and applied at the agent’s next start, restart, resume or wake that creates a container (a start of an agent that is already running keeps its container, so its held edits stay held). A stored pending key that can no longer be applied (it no longer decodes, or is no longer an editable key) is removed at the next start or config edit, audited as agent_pending_config_discarded with the trigger unappliable. Provision-rendered keys (system_prompt, agent_instructions, skills, services, secrets) are kept in pendingConfig in every phase, because a start does not re-render them, unless the agent’s Runtime Broker advertises changedFields: then system_prompt, agent_instructions, skills and services are written now in a phase with no container (as with the experiment off), and a held edit of them is applied by the next start, which lists it in changed_fields (secrets always stays pending); a start that leaves them pending answers with the warning Held for reincarnation: <keys>. A reincarnation does not apply pending edits yet; they stay pending. and sets the agent’s status message to it. A reincarnation does not apply or clear pending edits yet: they stay pending through it, and the pending keys a start applies (container keys, and the re-rendered provision keys on a Runtime Broker that advertises changedFields) are applied at the next start after it, over the values the reincarnation set. Keep the experiment off until reincarnation applies pending edits. A save is validated as an immediate edit is, overlays only the keys it names on the pending config, leaves out a key whose value equals what the next run would get, and bumps stateVersion; a save that arrives while a start is in progress waits for the next start. A config PATCH of an agent with no container first applies its pending container keys (and, on a Runtime Broker that advertises changedFields, its pending re-rendered provision keys), then the request, so only the provision-rendered keys that broker does not re-render stay pending (secrets always does). A reconcile-driven restart, which recreates the container from the Runtime Broker’s stored config, leaves pending edits as they are. With the experiment on, a request that names agentRole is refused with 409 (change the role by reincarnating the agent); without it agentRole is ignored. With the experiment off, stored pending edits are not applied; a config edit written then removes the pending keys it names (audited as agent_pending_config_discarded with the trigger superseded), so a stale pending value cannot later override it, and other pending keys are kept. The Hub audits pending edits as agent_pending_config_saved, agent_pending_config_applied (with the trigger: update, start, resume, restart or wake) and agent_pending_config_discarded (with the trigger superseded for keys a write-through replaced, or unappliable for stored keys that can no longer be applied), each naming the keys. The project-scoped PATCH /api/v1/projects/:id/agents/:agentId applies the same rules.

• DELETE /:id/pending-config: Discard the agent’s pending config edits (agent.update; 404 while the hub.agent_edit_held experiment is off). An optional ?stateVersion= must match the agent’s (409 otherwise). The response is the agent plus discarded, the keys removed (empty when there were none, in which case nothing is written); otherwise stateVersion is bumped.

• DELETE /:id: Stop and remove an agent. Query: deleteFiles and removeBranch (both default to true; send false to keep the files or branch) and force=true (skip the Runtime Broker availability check and continue past Runtime Broker teardown failures; a forced delete skips soft-delete retention). The Hub runs the delete in the background, and a second DELETE while one is in progress joins it:

  • 204: the delete finished.
  • 202: teardown is still running after about 20 seconds. Send Prefer: wait=N (seconds) to lower that wait; it is never raised. The body is {"agentId": ..., "deletion": {...}}. Progress arrives as agent status events, and the agent is gone at the deleted event; alternatively poll GET on the agent until it returns 404 (or deletedAt is set, for a soft delete). The CLI polls every 2 seconds for up to 180 seconds.
  • 502 (runtime_error), 503 (runtime_unavailable, with Retry-After), or 409 (Runtime Broker conflict): the delete failed. details.deletionCode gives the reason.

  While a delete is active or has recently failed, the agent carries a deletion view (also on every agent status event, null otherwise): state is deleting or failed, plus soft, startedAt, and leaseExpiresAt (deleting) or expiresAt (failed). On the single-agent GET /:id, the view also carries blocksStart: whether a start would currently be refused with 409 delete_in_progress. It is omitted when the Hub could not check. The Web Dashboard updates the text of its delete banner and enables the Start button based on this field. A failed delete has a code (runtime_error, runtime_unavailable, conflict, in_doubt, abandoned, revoke_failed, or finalize_failed) and an error, and its view stays for 15 minutes. While a delete is running, or after a failure that leaves teardown unresolved (a Runtime Broker delete still outstanding, or a failure while finalizing), start, restart, reincarnate, restore, create-with-existing-agent and DM wake return 409 delete_in_progress. A DM wake whose resume lands after a delete won, including while it waits for the agent to become ready, also returns 409 delete_in_progress and does not deliver the message. Retry the DELETE, or send it with force=true, to clear it; this is safe even when the code is in_doubt. The exception is a Runtime Broker that holds a different run of the agent than the Hub recorded: it deletes nothing. The Hub then sends one more delete, for exactly the run the Runtime Broker reported, when the request has force=true or that run is one of the agent’s earlier runs (one the Hub started, as a failed start or restart can leave behind). The run the Hub recorded is sent after it, so files the Runtime Broker keeps for that run are handled as the request asks, too. If that delete succeeds, the agent is removed; a soft delete records the deleted run on the retained agent. If the reported run is not a valid run ID, if the Runtime Broker refuses that delete too because yet another run holds the agent, or if that delete fails any other way, the Hub keeps the agent, restores its phase and answers as for any failed delete (409 with code conflict for a refusal), even with force=true; it never removes the agent while a run it was told about may still be running. Without force=true and with any other reported run, including a best-effort delete of an agent still in the created phase, the Hub keeps the agent and answers 409 with code conflict, and a retry is refused the same way until the Hub and the Runtime Broker agree on the run or the delete is forced. The Hub settles one case of this by itself: when the agent’s assigned Runtime Broker sends a complete heartbeat that reports only an earlier run of the agent, one the Hub issued a credential for, and no other Runtime Broker reports the recorded run, the Hub adopts the reported run once the recorded one has been unchanged for 19 minutes (the Runtime Broker’s 15-minute bound on a synchronous start, plus twice the Hub’s 120-second dispatch timeout to the Runtime Broker) and no start, stop, delete or reincarnation of the agent is under way. It logs agent_run_adopted_from_heartbeat. A failed create whose cleanup is refused this way leaves the agent in the error phase with its quota still held. Each agent run has its own run ID, so a stale delete cannot remove a recreated agent with the same name. The Hub sends the agent’s run ID, when it has one, with every stop and delete, and on every runtime the Runtime Broker never acts on an entry labelled with a different run. An entry with no run label (started before run IDs existed) still matches by name. On the substrate runtime the final stop or delete goes by actor name, which is reused across runs, so that step does not check the run yet. On the Apple container runtime the run is checked by a listing just before the stop or delete call, so a container recreated in that moment can still be stopped or deleted.

• GET /:id/logs: Stream agent logs (WebSocket).

• POST /:id/exec: Run a command in the agent’s container. Body: command (array of strings, required) and timeout (seconds, optional). Returns {"output", "exitCode"}. A failure the Runtime Broker reports returns 502 with code runtime_error. With the hub.conduit experiment on, a command whose request reached the Runtime Broker but whose result was not received returns 502 with code outcome_unknown: the command may have run. The Hub does not retry it; error.details carries operation, broker_id and request_id, and the body never carries command output. Do not retry a command that is not safe to run twice.

• GET /:id/pty: Interactive terminal (WebSocket). The Hub relays the stream to the agent’s Runtime Broker and sends keepalive pings every 30 seconds, so a dead peer is detected instead of leaving the connection hanging. Behind the hub.agent_delegation experiment, an agent delegated credential is admitted on the WebSocket handshake and preflight when its grant and agent_delegation_policy allow agent.attach (other agent.attach routes such as exec, env, reset-auth, and keys are not admitted to delegated credentials). A delegated attach is served only on the Runtime Broker’s terminal path; an attach that would use the agent’s conduit path is refused with 503. See PTY close codes.

  • With the hub.conduit experiment on, the Hub picks the path for each agent. It uses the Runtime Broker’s control channel when the agent’s runtime supports attach. It uses the agent’s Conduit session when the runtime has no attach and the session offers a PTY. The leaf protocol and close codes are the same on both paths. A GET without a WebSocket upgrade is a preflight: it answers 200 with {"path": "broker"} or {"path": "agent"}, or 503 with details.reason when neither path can serve (broker_not_connected, conduit_not_serving, stream_authz_unavailable, registry_unavailable, agent_pty_unavailable, managed_runtime). An agent on a managed runtime never takes the broker path, whatever the experiment setting: when it has no Conduit session that offers a PTY, or the experiment is off, the Hub answers 503 with code runtime_attach_unsupported and reason managed_runtime; a temporary refusal on the agent path keeps its own reason. A user attach on the agent path is re-checked like other Conduit user streams (see authz_recheck_interval and stream_authz_max) and is closed with 4401 if access was withdrawn.

There is no separate resume endpoint: resuming is the start action applied to a suspended agent. A suspended agent is also resumed automatically when a message is delivered to it with the wake option set.

Agent creation and start are rejected with 429 Too Many Requests (quota_exceeded) when the target Runtime Broker is at its max_agents_per_broker limit (see Admin); error.details.runtimeBrokerId names that Runtime Broker. A request that would reserve capacity on one Runtime Broker for an agent that already holds an active reservation on another (a create, start, restart or direct-message wake) is refused with 409 broker_reservation_scope_conflict; the existing reservation is kept and nothing is moved.

Agent responses no longer include a visibility field. Access is determined by scope and grants only.

Agent state uses a layered model:

• Phase: Lifecycle stage (created, provisioning, cloning, starting, running, stopping, stopped), plus suspended (paused for resume) and error (the agent crashed — restartable).
• Activity: Runtime activity within the running phase (initializing, working, thinking, executing, waiting_for_input, blocked, completed, limits_exceeded, stalled, offline). Note: offline occurs when an agent heartbeat has not been heard for some time, often due to an expired auth token that the agent failed to refresh; stalled flags a live-but-hung agent and can trigger auto-suspend. (A crash surfaces as the error phase, not as an activity.) initializing means the container and harness process are up but the harness has not yet shown it is reading input; it changes to working once the harness is ready, or after 90 seconds at the latest. Messages sent while an agent is initializing are delivered and held until the harness is ready; an interrupt is delivered as an ordinary message. The same applies to a message sent with wake: if the woken agent is still initializing when the wake wait ends, the message is delivered and held, and an interrupt is delivered as an ordinary message. The legacy status field reports running for this activity.
• Detail: Freeform context (tool name, message, task summary).

Agent Self-Service (/api/v1/agent)

Endpoints an agent calls on its own behalf. They authenticate with the agent token, sent in the X-Scion-Agent-Token header (which sciontool uses) or as an Authorization: Bearer token, and take no agent ID in the URL: the Hub derives the agent’s identity from the token.

• POST /secrets: Fetch several secret values in one call. Body: {"keys": ["KEY_A", "KEY_B"]} (1 to 100 keys, 64 KB body limit). Returns {"secrets": [...]}, one entry per requested key with key, status (ok, not_found, or entitled_but_unavailable), value (only when ok) and error. Per-key failures do not fail the request; a request the agent is not authorized for at all returns 403. Values are scoped to the agent’s project. sciontool init uses this to fetch the keys listed in SCION_SECRET_KEYS at startup.

Agent Ports (/api/v1/agents/:id/ports)

• GET /: List the agent’s exposed ports, {"ports": [...]}. Each entry has port, label, host, mode, exposedAt, exposedBy, and url/basePath (/api/v1/agents/:id/ports/:port/proxy/). An agent can read only its own ports; a user needs read access to the agent.
• POST /: Register a port. Body: port (1–65535; ports 9810 and 18380 are refused), optional label, and optional host (default 127.0.0.1; only loopback hosts are accepted). At most 10 ports per agent. Returns 201, or 409 if the port is already registered. DELETE / clears every port and DELETE /:port removes one (204). Registering and removing ports requires the agent’s own token with the agent:port:forward scope, or a user who holds agent.port_access both on the agent and on the hub. User access tokens are refused with 403 (details.reason: GOV_PENDING).
• GET /tunnel: The agent’s port-forward tunnel (WebSocket), opened by sciontool. Same authorization as registering a port; a request that is not a WebSocket upgrade gets 400.
• /:port/proxy/... (any method): Proxy an HTTP request to an exposed port. On the port-forward tunnel, requests are limited to 32 MB and time out after 60 seconds. WebSocket upgrades are not forwarded over the tunnel (501). When the hub.conduit experiment is on and this Hub node runs the Conduit relay, the Hub uses the agent’s Conduit session (streamed, with no body limit or proxy timeout); it falls back to the tunnel only when the agent has no Conduit session, and if the agent has no tunnel either, the request fails with 503 and code agent_offline. Other Conduit errors are returned without falling back. See Port Forwarding.

Agent Delegation Grants (/api/v1/agents/:id/delegations)

Issuing and exchanging agent-bound delegation grants requires the hub.agent_delegation experiment (404 while off). The inspection and revocation routes (GET /, GET /:grantId, DELETE /:grantId, DELETE /:grantId/credentials/:credentialId, POST /revoke-all, GET /api/v1/users/me/delegations, and POST /api/v1/users/me/delegations/revoke-all) remain available even when the experiment is off so operators, issuers, agent owners, and the bound agent itself can inspect and revoke or relinquish grants. See Delegation and Revocation.

• POST /: Issue a delegation grant bound to the agent’s current project and generation (201). Requires agent.delegation.create and an interactive session of a local user for an agent the user controls (owner or in ancestry); dev sessions, user access tokens, federated users, and agent tokens are refused (403). Body: boundary ({"kind": "project", "projectId": "<uuid>"} or {"kind": "hub"}), permissions (a non-empty list from the delegable catalogue for that boundary, narrowed by the agent_delegation_policy operational settings section and checked against the issuer’s live permissions), optional expiresAt (default 7 days, maximum 30 days), optional maxCredentialTtlSeconds (1 to 3600, default 900), name (required), and optional purpose and labels. Capped at 10 active grants per agent and 50 per issuer (409 grant_limit_reached).
• POST /:grantId/exchange: Called by the bound agent with its own agent token (sent by sciontool init’s in-container Unix socket proxy at ~/.scion/delegation.sock when a process in the container sends X-Scion-Delegation-Grant: <grant-id>) to mint a short-lived scion_adt_... bearer credential. Body: audience (this Hub), optional permissions (a subset of the grant’s), and optional ttlSeconds (capped by the grant). Returns 200 {"token", "expiresAt", "grantId", "credentialId", "audience", "permissions"} with Cache-Control: no-store. At most 2 active credentials per grant (minting a third supersedes the oldest).
• GET /, GET /:grantId, DELETE /:grantId, DELETE /:grantId/credentials/:credentialId, and GET /api/v1/users/me/delegations: List (every grant the caller may see, each with status active, revoked or expired), inspect, and revoke grants or individual exchanged credentials. Accessible to the issuer, the agent owner, a user holding the matching agent.delegation.list, agent.delegation.read, agent.delegation.revoke or agent.delegation.credential.revoke permission on the Hub (any grant) or on a project (only grants bounded to that project), or the bound agent itself (with its own agent token on /api/v1/agents/:id/delegations..., where DELETE /:grantId relinquishes the grant with reason relinquished (a user’s revocation records revoked)). Soft-deleting, hard-deleting, restoring, or claiming an agent for reincarnation revokes all unrevoked grants bound to it in the same transaction (agent_deleted, agent_restored, agent_reincarnated), and deleting a project revokes grants of all project agents in the cascade transaction (project_deleted). Suspending the agent (agent_suspended) or suspending or deleting the issuer (issuer_suspended, issuer_deleted) revokes grants after commit while exchange and use checks deny immediately.
• POST /revoke-all and POST /api/v1/users/me/delegations/revoke-all: Revoke unrevoked grants (including expired ones) and their active credentials in one transaction (200 {"revoked": [...], "skipped": [{"grantId", "agentId", "reason"}]}). Session only (agent tokens, delegated credentials, and user access tokens are refused with 403; the bound agent relinquishes its grants one by one via DELETE /:grantId). POST /api/v1/users/me/delegations/revoke-all revokes every unrevoked grant the caller issued (skipped is empty). POST /api/v1/agents/:id/delegations/revoke-all revokes every unrevoked grant of the agent that the caller may revoke-all: the grant’s issuer, the agent’s current owner, a user holding agent.delegation.revoke_all on the Hub (any grant; granted by the built-in super-admin and hub-admin roles), or a user holding agent.delegation.revoke_all on a project (only grants bounded to that project; built-in project roles do not hold agent.delegation.revoke_all, so project owners and admins revoke project-bounded grants one at a time via DELETE /:grantId unless granted a custom project role with agent.delegation.revoke_all). Unrevoked grants the caller can read (agent.delegation.read) but not revoke-all are left unrevoked and listed in skipped with reason: "forbidden"; grants the caller cannot read are omitted.

Projects (/api/v1/projects)

The legacy /api/v1/groves aliases and the project WebDAV endpoint (/api/v1/projects/:id/dav/) have been removed. Requests to /api/v1/groves or any path under it now return 404 Not Found (use /api/v1/projects), and requests under /api/v1/projects/:id/dav/ return 404 Not Found (use git or the project workspace file API at /api/v1/projects/:id/workspace/files and /api/v1/projects/:id/workspace/archive).

• GET /: List projects you have access to.
• POST /: Create a project without a Runtime Broker. Body: name (required), optional slug, gitRemote and labels, and workspaceMode. Omit gitRemote for a Hub-managed project without git. The slug global (in any letter case) is reserved for the global project: create, clone, update, and register with a git remote refuse it with 400, and a name that would slugify to global gets global-1 (or the next free suffix). A client-supplied slug (on create, clone, and PATCH) must already be in slug form, that is equal to its slugified value (lowercase letters, digits, and single hyphens, not starting or ending with a hyphen); anything else is refused with 400 validation_error and is never rewritten.
  • Clone URLs cannot carry credentials. On create, register, and label updates, a scion.dev/clone-url label containing a username, password or token (other than the git@ login of an ssh or scp-style URL), a query string, a fragment, whitespace, or non-ASCII characters is refused with 400 (details.field names the label; the value is never echoed). A gitRemote is stored without its query, fragment, and ordinary credentials, and is refused if a credential cannot be removed cleanly. Configure clone authentication with a project secret (such as GITHUB_TOKEN) or the GitHub App instead.
  • workspaceMode is the workspace sharing mode: shared, per-agent or worktree-per-agent. With a gitRemote, per-agent means clone-per-agent. Without one, it means empty-per-agent: each agent gets a private directory that starts empty. Omitted means the existing default (shared for a project without git).
  • The mode can only be set at create time. Unknown values, and worktree-per-agent without a gitRemote, are rejected with 400. It is stored in the server-owned scion.dev/workspace-mode label. A labels entry for that key must match workspaceMode (else 400), and is dropped when workspaceMode is omitted. A later update that tries to change the label is rejected with 400.
  • For an empty-per-agent project, creating or starting an agent on a Runtime Broker that does not advertise the emptyPerAgentWorkspace capability fails with 412 Precondition Failed. An agent create that names a workspace path is rejected with 400, and workspace files sent with it are ignored with a warning in the response’s warnings. Reincarnating such an agent follows the empty-per-agent rules of POST /api/v1/agents/:id/reincarnate. A Runtime Broker whose default runtime is Cloud Run or Substrate reports this capability as absent from its first heartbeat onward, so creating an empty-per-agent agent there returns 412. Until that heartbeat, the agent can still be dispatched, and the Cloud Run or Substrate runtime rejects it with an error. The capability reflects only the broker’s default runtime, so an agent that uses a Cloud Run or Substrate runtime profile on a broker whose default runtime is neither gets no 412 and fails at agent start. The cloudrun-sandbox runtime supports the mode.
• POST /register: Register or link a project repository. If the request resolves to an existing project, the caller needs update access to that project; without it, the request is rejected before anything changes. The same check applies to creating a project that resolves to an existing one and to linking a provider.
• GET /:id: Get project metadata and statistics. Behind the hub.agent_delegation experiment, an agent delegated credential is admitted when its grant and the agent_delegation_policy operational settings section allow project.read for the grant’s boundary (project or hub).
• PATCH /:id: Update a project. The updatable fields are name, slug, labels and defaultRuntimeBrokerId; an omitted or empty field is left unchanged, and labels replaces the whole map. The owner changes only through POST /:id/transfer-ownership: an ownerId in a PATCH body is ignored. The response is the updated project.
• DELETE /:id: Delete a project, with its configuration and its agent records. Requires a full session (scoped access tokens and agent tokens get 403), and only a direct project owner or a super-admin may delete. Callers without that authority get 403 (or 404) before anything else is checked. Returns 204 on success. While the project has an agent that is not soft-deleted (an agent never deleted, or one whose delete is in progress or failed), the delete is refused with 409 conflict and nothing is deleted: details.agents lists up to 50 of those agents (id, slug, and deletionState when a delete is in progress or failed: deleting, finalizing or failed; the key is absent for an agent never deleted), and details.truncated is true when more agents remain. Delete those agents first (DELETE /api/v1/agents/:id). Soft-deleted agents, whose runtime delete has already completed, do not block and are removed with the project. Query: force=true deletes the project anyway: its agent records are removed with it, and the Hub then asks the Runtime Brokers to delete those agents best-effort, logging any failure (an agent left running on a Runtime Broker is not retried). scion hub projects delete --force sends it.
• GET /:id/secrets: Manage environment secrets for the project.
• GET /:id/agents: List the project’s agents. Query: phase, label (repeatable key=value), runtimeBrokerId, includeDeleted=true, limit (default and maximum 500), and cursor. The response carries agents, totalCount, serverTime, and, when more results exist, nextCursor. Cursors are bound to the project, the filter, and the caller that received them; pass them back unchanged with the same query. A cursor that fails this check is rejected with 400. For a user caller, this list and the global GET /api/v1/agents list return the same set for a project: only the agents the caller can read, with pages and totalCount following that rule. Above 2,000 candidate agents, totalCount is a lower bound and the response sets totalCountApproximate. On a user access token, agent:list (or project:read on a token bound to the project) is enough to list; each row is still checked.
  • Sorted mode (sort=created or sort=updated): returns a stable, server-sorted order by creation time or last update, newest first (dir=desc, the default) or oldest first (dir=asc). Any other sort value is rejected with 400. Optional fit (1–500, at least limit, not valid with cursor) asks for the whole readable set in one response when the project has at most fit agents; the response then reports complete: true, or complete: false with the first page and a nextCursor. stats=1 adds a stats block (total, running, and agents as [id, phase] pairs) computed over the filter with phase ignored. The response echoes sort and dir. Sorted mode serves at most 2,000 agents: a request whose filter matches more is refused with 422 and error code sorted_view_unavailable (details.reason: too_many_candidates), so retry without sort. The page size can be narrower than limit for large projects. An agent token can use sorted mode on its own project.
  • Compact view (view=compact): returns the same slimmer items as GET /api/v1/agents?view=compact, without appliedConfig. This endpoint does not set _messageability, in either view. Membership, order, cursors, counts, complete and stats are the same as in the default view=full. An empty view is the same as no view. In sorted mode, a non-empty view other than full or compact is rejected with 400; otherwise any value other than compact is ignored.
• GET /:id/providers: List the Runtime Brokers that provide compute for the project. Each provider reports broker-wide capacity: agentLimit (the effective max_agents_per_broker limit; omitted when the broker is unlimited or no limit applies), agentCount (running agents on that broker from any project; omitted when no limit is configured), and agentLimitSource (which precedence step produced agentLimit: broker, entitlement, hub_default, unlimited, or not_enforced). When agentLimitSource is not_enforced, the hub-wide “enforce broker agent quotas” switch is off: agentLimit is informational only — it is still the resolved cap, but agent creates on that broker are not rejected against it. scion hub projects info shows these as (agents: count/limit), with (not enforced) appended when the switch is off.
• GET /:id/settings, PUT /:id/settings: Read or update the project settings (defaults for new agents). PUT updates field by field. A field left out of the body, or sent as null, keeps its stored value. A field sent with its empty value is cleared: "" for strings, 0 for defaultMaxTurns and defaultMaxModelCalls, {} for defaultResources and defaultGCPIdentityServiceAccountIDByProfile. Any other value replaces the stored one. bucket, runtimes, harnesses and profiles are not project settings (they come from server and Runtime Broker settings): a PUT that sets any of them to a value other than null is refused with 400 and nothing is saved. Object fields (defaultResources, defaultGCPIdentityServiceAccountIDByProfile) are replaced whole, not merged key by key. Exception: for telemetryEnabled, autoExposePortsEnabled and defaultThinkingLevel only, null clears the field (back to inheriting the hub default), because the two switches treat false as a real value and the thinking level has no empty value. Leaving them out still keeps them. defaultThinkingLevel must be between 1 and 100, as for the Hub default default_thinking_level: any other value, 0 included, is refused with 422 validation_error (details.field is defaultThinkingLevel) and nothing is saved. A level outside 1-100 stored before this rule (an earlier PUT accepted 0) reads as unset: GET does not report it, new agents inherit instead of getting 0, and the next save of any project setting removes it. activeProfile is the broker profile new agents run under when the request names none. defaultGCPIdentityServiceAccountIDByProfile maps a broker profile name to a registered GCP service account ID. Default service accounts are checked on write: each must be reachable from the project and verified. For defaultGCPIdentityServiceAccountIDByProfile, only the entries the body adds or changes are checked; an entry sent with its stored value is not checked again, and removing an entry is always allowed, so an entry whose account is no longer usable does not block a change to another profile. The response is the saved settings plus an optional warnings array of strings, omitted when empty. Warnings never fail the save. For each per-profile entry the body adds or changes, there is a warning when no Runtime Broker providing the project reports that profile, or when the Hub’s shared mapping decision (see mappings below) is not_mapped for the account on every provider profile of that name that it could decide on. An unknown decision, a profile whose runtime needs no mapping, or a provider whose profiles the Hub has no record of never produces a warning. The PUT is not conditional: two concurrent writes of defaultGCPIdentityServiceAccountIDByProfile each replace the whole map, so the later one wins. The default GCP identity mode and service account are checked together, using the stored value for whichever one the body leaves out, but only when the body sets at least one of them. A stored service account is checked again only when the body selects assign, so a body that changes or clears only the mode still works after that account is no longer usable. See Per-Profile Default Service Accounts.
• GET /:id/settings/resolved: Get project settings indicating whether a Hub default exists per-setting (non-admin gated).
• GET /:id/gcp-service-accounts: List the GCP service accounts registered to the project (includeHubScoped=true adds the hub-scoped ones). Each item carries mapping: mappedProfiles (Kubernetes broker profiles of the project whose state for the account is mapped, by the same shared decision as the status view below), reportedProfiles (profiles whose broker reported its mappings) and unreportedProfiles. A profile whose runtime needs no mapping (Docker, Podman, Apple container) is not counted. The same summary is on GET /api/v1/gcp-service-accounts?scope=project&scopeId=…, and absent at hub scope.
• GET /:id/gcp-service-accounts/:ref/status: The per-account status view, relative to this project. :ref is the account’s id, email (case-insensitive) or display name, resolved as for agent create; more than one match returns 400 identity_ambiguous with the candidates’ ids and scopes, and an account that does not exist or is not reachable from the project returns 404. The response has these sections:
  • account: id, displayName, scope, email.
  • verification: the Hub’s last token-creator check: status (verified, unverified or failed), verified, verifiedAt, error.
  • mappings: one entry per Kubernetes broker profile of the project’s Runtime Brokers, with brokerId, brokerName, profile and state: mapped; not_mapped; or unknown, with unknownReason report_missing (no stored report), report_incomplete, report_old_version (a broker that predates the current report version), report_stale, runtime_unrecognized (a runtime key the Hub cannot tell is Kubernetes) or ambiguous_mapping. The state and reason are the Hub’s shared mapping decision, the one the pre-dispatch identity_not_mapped refusal and the not-mapped warnings use: mapped and not_mapped need a stored report that is complete, at the current report version, and stored or confirmed by the Hub within the last 15 minutes, on a profile whose runtime type is Kubernetes. A profile with no report at all is not_reported, and a profile whose runtime needs no mapping (Docker, Podman, Apple container) is not listed. Mappings are owned by the broker and read-only here. These fields are added when the broker reports them and omitted otherwise, so an entry from an older broker has state alone: kubernetesServiceAccount, namespace and source (mapped for an explicit kubernetes_service_account_mappings entry, discovered for annotation discovery) on a mapped entry; reportedAt, when the Hub last stored or confirmed the profile’s report; incomplete with incompleteReason (list_failed, unavailable or pending) when the report may not list every account the profile can serve; and ambiguous when more than one Kubernetes ServiceAccount in the namespace is annotated with the account and none is mapped explicitly (the broker refuses it; state is unknown with unknownReason ambiguous_mapping). On the combo server’s embedded broker, explicit mappings come from its live settings and are mapped whatever the stored report says; discovered mappings, the KSA, namespace and source, completeness and the report time come from its stored report. Until the broker’s next heartbeat refreshes that report, the view can differ from the pre-dispatch refusal, which reads the stored report only: an explicit mapping just added to the settings is mapped here while the refusal can still refuse it, and one just removed is not_mapped here while the refusal leaves the dispatch to the broker (which refuses it).
  • workloadIdentityBinding: state is unknown with reason not checked. The Hub does not read IAM policy for this view, so it never reports a binding as bound.
  • defaultFor: the defaults that name the account: {"kind":"project"} (the project default in assign mode), {"kind":"profile","profile":"…"} (a per-profile project default) and {"kind":"hub"} (the Hub default in assign mode).
  • agents: count and names (sorted, at most 50) of the project’s agents whose applied GCP identity is the account, limited to the agents the caller may list.
  • nextStep: the first missing link: code not_verified, not_mapped (with brokerName and profile: a profile whose default is the account but does not map it, or else the first profile that reported and does not map it when none does; an unknown profile is not a missing link, except an ambiguous mapping, which the message names), or none; message says what to do. Access is the same as reading the account with GET /:id/gcp-service-accounts/:saId, and a user must also be able to read the project. An agent may read it only from its own project (any other project returns 404), and an agent token without project:read gets an empty agents section.
• GET /:id/members: List the project’s role bindings. With groupBy=principal, returns one item per principal (principalType, principalId, principalDisplayName, builtInRoleName, and its bindings), and limit/offset/totalCount count principals rather than bindings. Each binding carries roleKind (builtin or custom).
• PUT /:id/members/principals/:type/:principalId: Set a principal’s whole project role set in one transaction. :type is user, agent, or group; a user can be addressed by ID or email. Body: roleDefinitionIds (at most one built-in membership role plus any custom project roles), optional expectedRoleDefinitionIds (a precondition: 409 membership_changed if the principal’s current roles differ; an empty list means “not yet a member”), and optional notBefore/expiresAt for newly created bindings. Returns the principal’s grouped membership with changed; 201 when the principal was not a member before. An empty set is refused with 400 empty_role_set (use DELETE), and an unknown, non-project, or second built-in role with 400 invalid_role_set. Requires project.manage and a user identity: an agent token gets 403 credential_insufficient, and a User Access Token follows the token rules below. Guards: the last usable owner (an in-effect owner binding of an existing, active user) cannot be removed or demoted, and the last owner binding of any kind cannot be removed (409 last_owner); an expired, scheduled, suspended, invited or deleted owner’s binding can be removed while another owner binding remains; built-in role changes follow the owner/admin governance rules (403 target_role_protected); creating or removing a custom role requires a direct project owner (or a signed-in caller with no project role who holds hub role_binding authority) and the usual CanDelegate ceiling; and a custom role that itself carries a role_binding.* permission cannot be granted through this endpoint by anyone.
• DELETE /:id/members/principals/:type/:principalId: Remove every project binding the principal holds, under the same guards. 404 if the principal holds none.
• GET /:id/members/assignable-roles: List the project-scoped roles with grantable for the caller, and, when not grantable, the reason, denialCode, and details the PUT above would return. Requires project.manage.
• GET /:id/visitors, POST /:id/visitors, PATCH /:id/visitors/:visitorId, DELETE /:id/visitors/:visitorId: List, grant, change the window of, and revoke project visitor grants for users of a trusted issuer (experimental: 404 while the federation.project_visitors experiment is off). A visitor has id (the visitor’s account ID, used by the item routes), bindingId, email, issuer, status (pending until the visitor’s first request, then active, or suspended), issuerAccepted (whether the Hub accepts visitors from that issuer now; while it is false the grant is kept but the visitor’s requests are refused), grantedBy, notBefore, expiresAt and createdAt; the list returns {"visitors": [...]}. POST takes issuer, email and optional notBefore/expiresAt, and creates a pending account when none exists for that issuer and email; the issuer must be trusted with allow_visitor_grants and the email must pass its allowed_emails and allowed_domains (400 with details.field otherwise; an invalid window names expiresAt); the platform service account and the proxy driver principals (server.auth.proxy.driver_principals) are never granted (400 with details.field: email); a repeated grant answers 409 (change its window with PATCH). PATCH takes notBefore and expiresAt and replaces both (an omitted bound is cleared). Every route requires project.manage_visitors (project owners and admins, and hub admins) and an interactive session (user access credentials get 403 with details.reason GOV_PENDING); granting, or a PATCH that extends the window (an earlier or cleared start, or a later or cleared expiry), also requires holding every permission of the project-visitor role on the project; shortening a window or revoking does not. Revoking takes effect on the visitor’s next request. A linked visitor’s requests are admitted only on the routes open to visitors: GET and HEAD of /api/v1/auth/me (GET reads the visitor’s own profile; HEAD answers 405, as for any caller); GET /api/v1/projects/:id/agents and GET /api/v1/projects/:id/agents/:agentId (an agent ID or slug) for a project the visitor holds a current grant on, answered with name, slug and status only ({"agents": [...]} for the list, every agent of the project that is not deleted, whatever its message mode; with ?messageable=true, only the agents the visitor may message now, the same check a send gets, and an agent whose check fails on an error is left out; other query parameters are ignored; a deleted agent is refused by ID and by slug); GET /api/v1/users/me/visitor-grants (the visitor’s own grants: projectId, projectSlug, projectName, status (active, scheduled before notBefore, or expired), notBefore and expiresAt, as {"grants": [...]}; a revoked grant is not listed); GET /api/v1/users/me/visitor-dms (the visitor’s own DMs with agents it may message now: conversationId, conversationKey, projectSlug, agentSlug and lastActivityAt, most recent first, as {"dms": [...]}); the two /users/me/visitor-* routes answer 404 for any caller that is not a visitor; POST /api/v1/projects/:id/agents/:agentId/message and POST /api/v1/agents/:agentId/message (a text message to an agent the visitor may message: a current grant on the agent’s project, and an agent message mode other than none; the visitor is the sender user:<visitor id>, its account ID, and a reply to that address reaches the visitor’s DM; the agent’s envelope metadata carries sender_visitor, sender_email (the issuer-verified email), sender_issuer (the issuer’s display name) and sender_display (for example alice@corp.example (visitor via idp.example)), which only the Hub sets; the response to a visitor also carries conversation_id and conversation_key (the visitor’s DM key with the agent, for the DM routes below), which no other sender’s response carries; a visitor’s message is text only: the request may carry message, or structured_message with msg, type (instruction, chat or reply) and recipient (not a group; version, timestamp, sender and sender_id are accepted, and the Hub sets the sender, timestamp and recipient), and any other field, such as plain, interrupt, wake, notify, mentions, urgent, status, attachments, metadata, channel, thread_id or conversation_id, answers 400 with details.field); and, with native chat on, GET /api/v1/chat/conversations/:dmKey/messages and GET/POST /api/v1/chat/conversations/:dmKey/read for the visitor’s own DM with such an agent (dm:agent:<agentId>:user:<visitor id>). Every other route, method and target, including an unknown project or agent, a deleted agent (its DM is also left out of visitor-dms), an agent the visitor may not message, and another principal’s DM, answers 403 with one uniform message. Authorization runs on every request. An agent can send to a visitor only in a DM the visitor started with that agent, by the visitor’s address user:<visitor id>; any other send to a visitor (before the visitor wrote, or from another agent) is answered as for an unknown user, and the visitor’s email never addresses the visitor.
• GET /:id/visitors/rules, POST /:id/visitors/rules, PATCH /:id/visitors/rules/:ruleId, DELETE /:id/visitors/rules/:ruleId: List, add, change the window of, and remove visitor rules: visitor access for every user of a trusted issuer whose verified email is in a domain (same experiment, admission and non-amplification rules as the grant routes above). A rule has id, issuer, domain, issuerAccepted, createdBy, notBefore, expiresAt and createdAt; the list returns {"rules": [...]}. POST takes issuer, domain and optional notBefore/expiresAt. The domain is matched exactly and case-insensitively against the part of the email after @: no wildcards and no subdomains (400 with details.field domain for a wildcard, an address or any other invalid domain). The issuer must be trusted with allow_visitor_grants, and, when set, its allowed_domains must list the domain and its allowed_emails must admit every address in it (a pattern such as *@corp.example). A repeated (issuer, domain) answers 409. A rule is not itself a grant: on a matching user’s request (with the email claim present and, where the issuer sends it, email_verified true) the Hub links or creates that user’s visitor account and adds a grant whose source is rule and sourceRuleId is the rule’s ID, carrying the rule’s window as it is when the grant is added. When that user’s grant by email on the project has expired or has not started yet, the rule’s grant replaces it; an active grant by email is kept, and the rule adds nothing on that project. A grant by email to a user who is already a visitor through a rule answers 409; to keep access independently of the rule, revoke the visitor, then grant the email. After the first request the visitor’s account follows the issuer subject, as for grants by email: a later email change at the issuer does not remove a grant a rule added while the rule stands. GET /:id/visitors shows each visitor’s source (email or rule). PATCH replaces the rule’s window and that of every grant it added; DELETE removes the rule and every grant it added in one transaction, and grants by email are never changed by a rule. Revoking one rule visitor with DELETE /:id/visitors/:visitorId stops the rule adding that person again; such a visitor’s window follows the rule, so PATCH /:id/visitors/:visitorId answers 409 for it. While the issuer’s allow_visitor_grants is off, no rule can be added for it and no rule adds anyone. Deleting a project removes its rules and every grant they added.
• POST /:id/agents/stop-all: Stop the project’s running agents. Authorized from role bindings: holders of agent.stop_all on the project (project owners and admins, and hub admins) stop every agent (scope: all); other members with a built-in project role, direct or group-derived, stop only the agents they own and may operate (scope: own); anyone else, including a caller holding only a custom project role, gets 403. The hub-wide POST /api/v1/agents/stop-all requires agent.stop_all on the hub.
• POST /:id/transfer-ownership: Body {"newOwnerId": "<user ID or email>"}. Grants the new owner project-owner, downgrades the caller to member, and moves the project’s ownerId to the new owner in one transaction. Errors: 400 not_found if the new owner does not exist and 400 principal_ineligible if they are not active (both also checked inside the transaction), and 409 last_owner if no other usable owner would remain (for example when the new owner’s existing binding is expired and no other usable owner exists).
• Membership changes by User Access Token (POST, PATCH and DELETE on /:id/members, PUT and DELETE on /:id/members/principals/:type/:principalId, POST /:id/transfer-ownership, and project-scoped bindings on /api/v1/admin/role-bindings): the token needs the project:manage scope and a boundary that covers the project (for a project-bound token, its holder must still be a member). The token then acts only through its holder’s own project role: the owner/admin governance rules, the direct-owner rule for admin and owner targets, last-owner protection and the one-binding rule apply as for a session, and a holder with no project role is refused (403 role_assignment_forbidden) even when they hold hub role_binding authority. A role is granted only when the token carries a scope for every permission of the role (403 target_role_protected otherwise); this includes the new owner’s role on a transfer. GET /:id/members/assignable-roles reports grantable under the same limits.
• POST /:id/clone: Deep-copy settings, labels, env vars, skills, hooks, harness configs, and templates to a new project with rollback protection. Supports an optional gitRemote field in the request body to override the source project’s git repository (carrying configurations over while using a different repository).
• GET /:id/workspace/files, PUT /:id/workspace/files/*, POST /:id/workspace/files, DELETE /:id/workspace/files/*, GET /:id/workspace/archive, POST /:id/workspace/pull: Browse, read, upload, delete, archive-download (ZIP), and git-pull a project’s workspace on the Hub. For Hub-managed projects and shared-workspace (shared) git projects, the Hub resolves projects/<uuid>/workspace on a shared workspace storage backend (nfs, cloudrun-volume, gke-shared-volume) or ~/.scion/projects/<slug> on the local backend (409 for empty-per-agent projects; 503 from the file and archive endpoints if a shared backend’s mount point is missing). For a linked project in a per-agent git mode, the workspace file endpoints resolve only a co-located Runtime Broker’s local path (409 when the project is served only by remote Runtime Brokers; the linked-project workspace cache and POST /:id/workspace/sync have been removed and answer 404). POST /:id/workspace/pull works only for shared-workspace git projects (409 otherwise, or when the workspace is not provisioned) and runs a hardened host-side git fetch and fast-forward, refusing if .git/config fails validation.

GCP Service Accounts

A project’s own accounts live under /api/v1/projects/:id/gcp-service-accounts (GET, POST, POST /mint, and per account GET /:saId, DELETE /:saId, POST /:saId/verify). Hub-scoped accounts use the flat routes /api/v1/gcp-service-accounts?scope=hub and /api/v1/gcp-service-accounts/:saId. Register, verify and delete each write an audit event with the actor, the account’s scope and the account ID; the event never carries the account email. When IAM check mode is enforce and the account belongs to a GCP project other than the Hub’s own, POST (201) and POST /:saId/verify (200) add an advisory message to the response’s warnings array if the Hub cannot read that GCP project. The message names the gcloud command that grants the Hub’s service account roles/iam.securityReviewer there.

• DELETE .../gcp-service-accounts/:saId[?force=true]: Remove a registration. The account is never deleted in GCP; a minted account is retained there. Before deleting, the Hub computes an impact report: agents (agents whose applied identity references the account, at most 50, with the total in agentCount), defaults (each project default, per-profile default and hub default that points at it, with tier project, profile or hub, and clearable), brokerMappings (broker profiles whose last report maps it), managed, and manualCleanup (the steps the Hub cannot do: remove the broker’s kubernetes_service_account_mappings entry, the Kubernetes ServiceAccount, the Workload Identity binding and the Hub’s token-creator grant, and for a minted account the account in GCP).
  • The report shows only what the caller could otherwise see. agents names only agents the caller may read (visibleAgentCount in all); any others are reported in hiddenAgentCounts, one unnamed count per project. A default in a project the caller may not read keeps its tier and clearable, has no projectId or profile, is marked redacted, and is counted in hiddenDefaultCount.
  • clearable is decided for the caller: a project or per-profile default is clearable when the caller may update that project’s settings.
  • While any default points at the account, the delete returns 409 with code sa_in_use and the report in error.details.impact.
  • force=true clears the project and per-profile defaults the caller may clear, then deletes. If any referencing default is not clearable by the caller, the delete stays 409, and the message says an admin of those projects must change them. A cleared project default whose mode was assign becomes block, not unset, so new agents get no GCP identity rather than falling through to a broader hub or runtime default.
  • force=true never clears the hub default. While the hub default points at the account (clearable: false), the delete stays 409 until a hub admin changes the hub default.
  • Agents that reference the account do not block the delete. They keep the reference and fail at their next start with the existing “no longer available” error.
  • Clearing and deleting are not atomic. If a forced delete fails after it cleared defaults, those defaults stay cleared. The error carries them in error.details.clearedDefaults, redacted the same way, and retrying the delete is safe.
  • A settings write that runs at the same time as a forced delete is not serialized with it. It can still leave a default pointing at the removed account. Agents created from that default then fail at start with the existing “no longer available” error.
  • Each project a forced delete changes publishes the same project-updated event as a settings save.
  • On success the response is 200 with {"deleted": true, "clearedDefaults": [...], "impact": {...}}. Earlier Hubs returned 204 with no body.

Runtime Brokers (/api/v1/brokers)

• GET /: List registered runtime brokers.
• POST /: Register a new compute node, or re-mint its join token. Requires broker.create (see Broker Registration Permission). The caller becomes the broker’s owner. Re-registering an existing broker requires ownership (see Broker Ownership). The Hub sets the reserved scion.io/plugin and scion.io/broker-role labels itself. On re-registration the Hub keeps the stored name, keeps the stored value of every existing label (it only adds absent keys), and fills the GCP host service account and project only while both are unset and were not set or cleared through PATCH /:id; each non-empty requested value that is not applied is listed in keptSettings (field, storedValue, requestedValue, reason). When the registration would write gcpHostServiceAccountEmail or gcpHostProjectId (a new Runtime Broker, or an existing one whose host service account and project are unset and not pinned), the caller must be a user principal, and, when gcp_iam_check_mode is enforce, must also hold iam.serviceAccounts.actAs on the requested service account (403 before any write on denial).
• POST /join: Complete the two-phase broker registration.
• GET /:id: Get broker status and capacity. When the Hub has observed the Runtime Broker’s host GCP identity (a co-located server reads it from the metadata server), the list, GET /:id and the health summary include hostIdentity.state (match, mismatch, unset_detected, pinned_absent, none or unknown); for callers that may update the Runtime Broker, and in the PATCH response, it also carries observedEmail and observedAt. It is omitted for a Runtime Broker that was never observed. For a flat Runtime Broker (scion.io/broker-role=flat), GET /:id also includes runtimeTargetPresentation (the instance’s reported defaults for new agents: key, name, target and agentDefaults, with environment variable names but no values, no kubeconfig context or service-account policy, and credentials in the image registry redacted) and runtimeTargetPresentationObservedAt from the instance’s own heartbeats.
• PATCH /:id: Update the Runtime Broker’s name, user labels, and GCP host identity (gcpHostServiceAccountEmail, gcpHostProjectId). Setting either pins the GCP host pair across re-registrations; gcpHostPinned: false hands it back to registration. Setting or changing gcpHostServiceAccountEmail or gcpHostProjectId requires a user principal and, when gcp_iam_check_mode is enforce, iam.serviceAccounts.actAs on the requested service account (403 before any write on denial; name- and labels-only updates, an unchanged pair, and clearing the email do not run the check). A PATCH that sets or changes the pair writes it with compare-and-set against the stored pair the actAs decision was evaluated against (409 Conflict if the stored pair changed concurrently). The reserved scion.io/plugin and scion.io/broker-role labels are ignored in the request and their stored values kept.
• GET /:id/projects: List the projects the Runtime Broker provides for, limited to the projects the caller can read. Each entry carries projectId, projectName, gitRemote, agentCount (the project’s agents on this broker), localPath, and isDefault, which is true when this broker is the project’s default Runtime Broker.

In the list (GET /api/v1/runtime-brokers, without projectId) and in GET /api/v1/runtime-brokers/:id, a Runtime Broker that runs in the process answering the request (the server’s co-located Runtime Broker or one of its co-located flat instances) carries the read-only hostedByThisServer: true. The field is omitted for every other Runtime Broker. With several Hub replicas behind a load balancer, it describes the replica that answered.

The list (with or without projectId) and GET /api/v1/runtime-brokers/:id carry a read-only versionStatus on each Runtime Broker: match, mismatch or unknown, comparing the Runtime Broker’s version with the Hub’s version (see Version mismatch). It is unknown when the Runtime Broker has not reported a version or when there is no single Hub version (no live Hub instance, or live instances running different versions), and it is omitted for plugin records. Both list responses also carry hubVersion, the Hub version compared with: the version every live Hub instance reports, mixed when they differ, or empty when it is unknown. A Runtime Broker reports its version (version, the release tag or the short commit) when it joins or registers, and in the optional version field of its heartbeat (POST /api/v1/runtime-brokers/:id/heartbeat). The heartbeat field is applied only on a heartbeat the Runtime Broker sends with its own credentials (not on a heartbeat sent by a user), and an omitted or empty value leaves the stored version unchanged. A co-located Runtime Broker records the server’s version each time the server starts.

Broker Settings (/api/v1/runtime-brokers/:id/settings)

A general per-broker settings mechanism. maxAgents, a per-broker override of the max_agents_per_broker cap, is the first registered key.

• GET /:id/settings: Read the broker’s stored settings document plus the effective (resolved) value for each key. Requires broker.read. 404 if the broker doesn’t exist. If the broker has no settings row yet, settings is {} and revision is 0.
• PUT /:id/settings: Replace the settings document. Body: {"settings": {"maxAgents": 30}, "expectedRevision": 3}. This is a full replace, not a merge: a key omitted from settings (or sent as null) is cleared back to “inherit”. Each key’s own permission gates writing it — maxAgents requires quota.update — checked only against keys whose value actually changes, so re-sending the current document needs no permission at all.

Response shape (both verbs):

{
  "brokerId": "…",
  "settings": { "maxAgents": 30 },
  "effective": {
    "maxAgents": {
      "value": 30,
      "source": "broker",
      "count": 7,
      "inherited": { "value": 100, "source": "hub_default" }
    }
  },
  "revision": 3,
  "updatedBy": "admin@example.com",
  "updated": "2026-01-01T00:00:00Z",
  "_capabilities": { "update": true }
}

effective.maxAgents.source is one of broker (this broker’s own setting), entitlement (an entitlement binding), hub_default (the limit definition’s default value), or unlimited (no limit definition exists, or no quota service is configured); it is null/"" only if resolution itself fails. inherited reports what the value and source would be if the broker’s own setting were cleared, so the UI can always show what “use the default” means without having to clear it first to find out. count is the current active-reservation count against the same limit Reserve counts.

Status codes: 400 for an unknown key or an invalid value (maxAgents must be >= 0; 0 means unlimited); 403 if the caller lacks the permission a changed key requires; 404 if the broker doesn’t exist; 409 on a stale expectedRevision (the response body carries the current record under current, same shape as a normal GET).

Precedence for max_agents_per_broker (most specific wins): a broker’s own maxAgents setting, if set, is the effective limit — it can be lower than the hub-wide default and lower than any system-scoped entitlement binding. Otherwise, the existing entitlement-engine resolution applies: bindings (most generous wins), falling back to the limit definition’s hub-wide default value. In both layers, 0 means unlimited.

Migration from entitlement bindings. Earlier releases had no per-broker settings API, so operators worked around it with a broker-scoped entitlement binding on max_agents_per_broker — either a system_default binding with a non-empty subjectId, or a user binding whose subjectId is set to the broker’s own ID (the “user-subject hack”). On upgrade, a one-shot migration copies every broker-scoped binding on this limit into a maxAgents setting (0 if any binding was 0, otherwise the largest value), regardless of subject — including bindings the entitlement engine never actually enforced (a system_default row needed an empty subject to be picked up, so a non-empty-subject row was previously a silent no-op; a user binding scoped to the broker but owned by some other user was likewise never matched). Sweeping these up anyway restores what the operator evidently intended when they scoped a binding to that broker. This means an upgrade can newly impose, or tighten, a broker’s effective cap for a broker that previously had no effective per-broker limit at all (or was really being capped by a more generous system-scoped binding, since the engine used to merge broker- and system-scoped bindings with “most generous wins” — the migrated setting no longer merges with anything). The migration is attributed as updatedBy: "migration:ptone/scion#2061", and it only fills in brokers that don’t already have a maxAgents setting. It never deletes the old bindings — they are shadowed by the new setting per the precedence above, and their IDs are logged at migration time so an operator can find and remove them. Removing them matters: if the migrated maxAgents setting is later cleared (“use hub default”), the precedence rule falls through to the entitlement engine, and any leftover binding the entitlement engine matches (a user binding with subjectId equal to the broker ID, or a system_default binding with an empty subject — i.e. the shapes the migration actually inherited enforcement from) becomes live again. The never-enforced shapes described above stay inert either way. Clearing the setting does not by itself restore the hub-wide default if a matching binding is still there. Legacy broker-scoped bindings can still be deleted (DELETE /entitlements/:id) or read normally; they just can’t be created fresh or edited while staying broker-scoped — see the 400 below. Because of this, creating a new broker-scoped binding on max_agents_per_broker, or editing an existing one while keeping it broker-scoped (POST on /limits/:id/entitlements, or PUT on /entitlements/:id with scopeType: "broker"), now returns 400 with the message per-broker agent caps are set via PUT /api/v1/runtime-brokers/{id}/settings. System-scoped bindings for this limit are unaffected and continue to work as the hub-wide override.

Artifacts (/api/v1/artifacts)

Published files and bundles with stable, versioned scion://artifact/<id>[@<seq>] references: single-file publish (POST /?name=<file>), two-step publish (POST / or POST /:id/versions with a manifest, PUT /:id/versions/:seq/files/:path, POST /:id/versions/:seq/finalize), the caller’s list (GET /?mine=1, optionally narrowed to one project with scope=<project-id>, which adds canCreate for the caller, or to what is shared with the caller with shared=1), metadata (GET /:id, GET /:id/versions (ready versions plus discarded ones, with only number, state, reason and the time the number was taken; a page can hold fewer than limit ready versions, so clients follow nextBefore), GET /:id/versions/:seq; GET /:id and GET /:id/versions/:seq carry canManage, canShare, canRetire, canPublish, canReview and canGrantReview for the caller, and scopeDeleted when the home project was deleted), file bytes (GET /:id/files/:path, GET /:id/versions/:seq/files/:path, with ?resolve=clean|accept for a review’s text with its CriticMarkup marks resolved), reviews (a version of kind review, checked at finalize), views of HTML bundles (POST /:id/versions/:seq/view, GET /view/:capability/:path), grants (GET, POST /:id/grants, DELETE /:id/grants/:grantId; on an artifact owned by an agent, the agent’s delegating user and a home-project admin may also add a review grant for a user who can already read it through the home project or a grant to one of their projects; a review grant allows review versions only and gives no read access of its own), expiry and moving (PATCH /:id), and share links (POST /:id/links, GET /:id/links (with the hub’s defaultTtlHours and maxTtlHours), DELETE /:id/links/:linkId, and GET /shared/:token[/files/:path], which needs no credentials). Behind the hub.artifacts experiment (off by default): while it is off every route answers 404. See Artifacts for access rules, status codes and delivery headers.

Chat Unread Count (/api/v1/chat/unread-count)

• GET /: The number of chat conversations with unread messages for the signed-in user. Response: {"conversations": <threads + dms>, "threads": <n>, "dms": <n>}. A thread counts when the caller is a member of it, can read its project, has not muted it, and has not read its latest message. A DM counts when the caller is one of its parties, has not muted it, and has not read its latest message. Requires a user identity. The Web Dashboard shows conversations on the Chat mode badge and in the tab title. This replaces the chat MENTION and DM_RECEIVED notification rows, which the Hub no longer creates.

Chat Starred Messages (/api/v1/chat/stars)

Behind the web.chat_starred_messages experiment (404 while off; stored stars are kept while the experiment is off). Stars are private to each user (user.<id>.chat.stars SSE events sync changes across the user’s open tabs) and cover project space threads (thread:project:...; DM conversations return 404). User access tokens require inbox:write for PUT and DELETE and inbox:read for GET, and for PUT and GET, live read access to the thread’s project within the token’s boundary (DELETE always answers 204 and removes only a star inside the token’s boundary). See Starred messages.

• PUT /:messageId: Star a message in a project thread. Body: {"conversationKey": "<key>"}. Idempotent: returns 200 {"starred": true, "starredAt": "..."} (keeping the original starredAt when already starred).
• DELETE /:messageId: Remove the caller’s star on the message (204).
• GET /?conversationKey=<key>[&limit=1..100][&cursor=...]: List the caller’s starred messages in that conversation, most recently starred first (default limit 30; paginated with a sealed nextCursor).
• GET /state?messageId=<id>...: Batch check for 1 to 200 distinct message UUIDs; returns {"starredMessageIds": [...]} for messages the caller has starred and can currently read.
• GET /:messageId: Return the current detail of one starred message (messageId, conversationKey, conversationKind, projectId, projectName, conversationName, senderName, messageCreatedAt, starredAt, excerpt, edited, permalink), checked against the caller’s live read access to the conversation’s project (404 otherwise).

Chat Attachments (/api/v1/chat/attachments)

• GET /: Report where the calling credential’s attachment uploads go: {"uploads": "artifact"|"legacy", "delivery_max_bytes"?: <n>}, where delivery_max_bytes is messaging.attachment_delivery_max_bytes (0 when agent attachment delivery is off). Open to user sessions, user access tokens and agent credentials; uploads is "artifact" only when the experiment is on, the artifact service is ready and the caller may write (an agent must be in good standing), and only then is delivery_max_bytes present.
• POST /: Upload one or more files (multipart/form-data, field files, optional project_id). Max 10 files; 10 MB each on legacy chat storage, or artifacts.max_file_bytes (32 MiB by default) when the hub.attachment_artifacts experiment is on and the artifact service is ready, which also opens uploads to agents in good standing. When hub.attachment_artifacts is off, files are stored in the Hub’s attachment store and copied into the project’s scratchpad shared directory (.attachments/_webchat/<id>/<name>) when a message carrying them is sent to agents. Text files containing unusual control characters (e.g., vertical tab 0x0B) are supported and correctly identified as text. When a message with attachments is delivered to an agent, the Runtime Broker redeems one single-use, HMAC-signed token per file at GET /api/v1/attachment-deliveries/:token, with its own Runtime Broker credential, to write the files into ~/.scion/inbox/<message-id>/.
• GET /:id: Download a stored attachment. An attachment stored as an artifact is served by the artifact service. Any other attachment is served to its uploader, to readers of its project and to readers of a message it is attached to (a user access token needs read access to the file’s project when the file has one); anyone else gets 404. Responses carry X-Content-Type-Options: nosniff, and Content-Disposition: inline only for image types — everything else is served as an attachment.

Uploads are accepted or refused per file, and the response reports both outcomes:

{
  "attachments": [{ "id": "…", "name": "compose.yaml", "mime": "text/plain", "size": 34, "url": "/api/v1/chat/attachments/…" }],
  "failures": [
    { "name": "setup.sh", "error": "files with a .sh extension are not accepted" },
    { "name": "notes.html", "error": "files with a .html extension are not accepted" }
  ]
}

size is the stored file’s length in bytes — the example assumes a 34-byte compose.yaml — while id and url are elided here because both are assigned per upload.

Status codes:

• 201 Created — at least one file was stored. failures may be non-empty. Previously a single bad file failed the whole batch with 400 and stored nothing; clients that treat 201 as “all files stored” must now read failures, or they will drop files silently.
• 400 Bad Request — nothing was stored and the caller can fix it (blocked extension, unaccepted type, oversized file).
• 500 Internal Server Error — nothing was stored and the failure was server-side.

The stored MIME type is derived from the file’s content plus its extension; the Content-Type a client declares on the part is ignored. Executable extensions (.exe, .sh, .js, .ps1, and their peers) and markup extensions (.html, .svg, and their peers) are refused whatever the content is.

Experiments (/api/v1/experiments)

• GET /: Return the resolved hub-wide experiment map, {"experiments": {"<name>": true|false}}. Available to any authenticated identity (user, agent, or Runtime Broker token). The web client fetches it at boot. Admins change values through /api/v1/admin/experiments.

Profiling (/api/v1/profiling)

• GET /: Return the profiling switches the web client acts on, {"readinessMarks": true|false}, and nothing else. Available to any authenticated identity; a request without credentials gets 401. Admins change the value through /api/v1/admin/profiling. See Profiling.

Conduit (/api/v1/conduit)

Behind the hub.conduit experiment. With the experiment off, GET / and GET /grant-keys answer any authenticated caller with 404; the rotate route answers 404 only after its permission and session checks pass.

• GET /: Open a Conduit session (WebSocket). Open to an agent token with the agent:port:forward scope, to a Runtime Broker signing its request, or to a user (the same credential as attach, used by scion tunnel); any other caller gets 403 (a Runtime Broker request with a missing or invalid signature gets 401). Returns 503 when the Conduit relay is not running on that Hub node (see Conduit settings); when the relay failed to start because of a setting, the 503 message names that setting. Returns 400 for a request that is not a WebSocket upgrade. For an agent token, 401 when the agent no longer exists (403 if it is soft-deleted). A user credential whose kind cannot open a session gets 403, and 503 when the Hub cannot read the credential’s expiry.
• Tunnel RPC (on a user’s Conduit session, not an HTTP route): POST /v1/tunnels with {"agent": "<agent id>", "kind": "tcp|pty|ssh", "params": {...}} asks for one tunnel. Params: tcp takes port (host, if given, must be 127.0.0.1); pty takes cols and rows (default 80x24) and an optional session (only scion); ssh takes none. Each tunnel is authorized on its own: agent.port_access and an allowed agent-local port for tcp; agent.attach and agent.port_access for pty and ssh. On success the Hub opens a stream toward the session and answers with streamId, sessionId, agentId, projectId, kind and, when set, authzDeadline. Refusals: 400 invalid_request, 401, 403, 404 agent_not_found, 409 agent_not_running, 429 (the session’s tunnel limit), 503 (Runtime Broker or agent session unavailable, or a temporary Hub fault with details.reason); 405 for a method other than POST; 500 internal_error for an unexpected Hub fault; while the session drains, 503 with Retry-After: 0 (retry on a new session).
• GET /grant-keys: The public grant verification keys, {"keys": [{"kid", "public_key", "not_after_unix"}]}. Available to any authenticated identity.
• POST /api/v1/admin/conduit/grant-keys/rotate: Rotate the grant signing key. Requires hub.conduit_grant_keys.execute and an interactive session or the local dev credential. Returns kid, activate_at and retiring. Returns 409 if the previous key is not active yet.
• POST /api/v1/admin/conduit/grant-keys/revoke: Revoke a grant key at once. The body is {"kid": "<key id>"}. Same permission and credentials as rotate. The key stops verifying anything, and streams whose latest grant it signed get no further renewal. Returns {"kid"}; 400 without a kid; 404 for a key the Hub does not hold (or when Conduit is off); 409 for the key that signs now (rotate first).

Templates (/api/v1/templates)

A template’s or harness config’s contentHash (sha256:<64 hex>) depends on the hub.content_hash_v2 experiment, off by default:

• Off: commits record a hash of the file contents only. A rename-only or chmod-only re-upload does not change it, so it is not visible to the expectedContentHash check: a finalize that diffed against the version before such a commit is accepted.
• On: commits record a hash that also covers each file’s path and executable bit. A rename-only or chmod-only commit changes it, and a finalize that diffed against the version before it answers 409.
• Each commit records the version active when it runs, and a row keeps it until its next commit. The 409 check compares the stored value exactly, so turning the experiment on or off is safe at any time; at most, a client holding a hash read before a toggle gets one 409 after an unchanged recommit, and succeeds after re-reading.
• When to turn it on: only when every CLI and agent image that talks to the Hub is at or after the release that added it. An older CLI compares its own contents-only hash with the Hub’s, so with the experiment on it reports templates as out of date, and scion start, resume and create with -t for a template it also has locally fail without a terminal (work around with --no-upload, or --yes, which re-uploads the template each time, or upgrade the CLI).

Do not compare a contentHash with a hash you compute from local files to decide whether they are in sync; compare the files manifest (paths, hash and mode) instead. Skill version content hashes (contentHash) cover file contents only, whatever the experiment; with the separate hub.skill_content_hash_v2 experiment on, skill version and skill resolve responses also carry contentHashV2, which covers paths and modes too (see Content hash forms).

• GET /: List available agent templates. The authorized list validator caps list requests at a maximum limit of 100 templates per page (default is 50). Requests specifying a limit query parameter greater than 100 will fail with HTTP 400 Bad Request.
• POST /: Upload a new template or version. A template’s agent config lives only in its files (scion-agent.yaml). A body that carries config answers 400 template_config_in_file (set it in scion-agent.yaml), and one that carries image answers 400 template_image_not_supported (the image comes from the harness-config or settings). The same rules apply to user templates.
• GET /:id: Get a template. The response includes agentConfig, a read-only snapshot of the template’s parsed scion-agent.yaml, derived by the Hub on every commit with the same parser and key order the Runtime Broker uses, together with the derived harness and defaultHarnessConfig. agentConfig is a faithful parse: it may include environment keys (image, resources, volumes, kubernetes, hub, telemetry) that agents ignore.
• PUT /:id: Update a template’s metadata only: name, slug, displayName and description. Every other field (the derived fields, files, content hash, storage location, scope and ownership) keeps its stored value whatever the body sends. A body that carries config or image answers the same 400s as POST /. To change a template’s config, edit its files and commit them (upload, then finalize). The same rules apply to user templates.
• POST /:id/finalize: Commit a template’s file manifest after its files are uploaded. The body is {"manifest": {"files": [...]}} with the complete file list, and optionally expectedContentHash, the template content hash the client diffed its upload against. A template changed by another commit since then answers 409 with code template_conflict and nothing is committed; re-read the template, re-diff and finalize again. Without expectedContentHash the check uses the content hash the Hub read when the request arrived. The file endpoints (PUT, POST and DELETE under /:id/files) answer the same 409 when another commit lands while they run. scion templates sync retries a conflict once, re-diffing against the Hub’s new version.
• POST /:id/upload: Get upload URLs for a template’s files. The body is {"files": [{"path", "size"}]}. The response has uploadUrls and an uploadId. The URLs stage the files; nothing is visible to readers until finalize commits them. With local storage the URLs point at the Hub’s own file endpoint with ?uploadId=, which stages a file without committing it. POST / with files returns the same uploadUrls and uploadId. When the CLI or transfer client uploads collected files (for templates, harness configs, skills, or workspace sync), each file is re-opened through an os.Root handle on the collected directory, verified to still be a regular file of the recorded size, and hashed as it is read. If a file’s size or SHA-256 hash no longer matches what was recorded at collection, the upload fails before it is finalized, so changed content is never committed.
• POST /:id/finalize also takes an optional uploadId naming the upload whose staged files it commits. Without one, the Hub uses any staged file whose content matches the manifest hash. The Hub hashes every staged file. A file whose content does not match its manifest hash answers 400 and nothing is committed. A file that was neither staged nor already stored answers 400 with file not found: <path>.
• GET /:id/download: Get signed download URLs for a template’s files. The response includes contentHash, the content hash of the version the URLs were signed for; a client that caches by content hash should key on it, or on a hash it computes from the files it downloaded. Template files are stored by content and never overwritten, so the URLs of one response keep returning that version even if a new version is committed while the client downloads.
• POST /:id/reimport: Refresh a template imported from a GitHub folder URL, replacing its files with the current content of the source folder (see Refreshing a Hub template from its source). Optional body: sourceUrl, which replaces the stored source URL for this refresh and is stored if the refresh succeeds. The caller must be able to read the template and to import templates into its scope. Built-in templates and templates with no source URL are refused with 400 (no_source_url), as are URLs that are not https GitHub folder URLs or that include a username or password (unsupported_source). A failed fetch or import answers 400 reimport_failed. On success the response has the same shape as a template import.

Harness configs (/api/v1/harness-configs)

A harness config’s files are its content. Its harness and config fields are derived from its config.yaml at every commit, and a key absent from config.yaml clears its field. Every change to the files goes through one commit. The commit checks the content hash, layout and storagePath the Hub read when the request arrived, and refuses the change if another commit landed in the meantime.

• GET /: List harness configs. Paginated with nextCursor (see Pagination).
• POST /: Create a harness config. The body has name, harness, scope (global, project or user) and optionally scopeId, slug, displayName, description and files ([{"path", "size"}]). A body that carries config answers 400 with code harness_config_config_in_file; set the config in config.yaml instead. With files, the response has uploadUrls and an uploadId, as for POST /:id/upload.
• PUT /:id and PATCH /:id: Change the metadata: name, slug, displayName and description. Every other field keeps its stored value. Unlike a template PUT, a body that carries other fields (for example the config a GET returned) is not refused; those fields are ignored. A metadata change never reverts a file commit.
• POST /:id/upload: Get upload URLs for a harness config’s files. The body is {"files": [{"path", "size"}]}. The response has uploadUrls and an uploadId. The URLs stage the files; nothing is visible to readers until finalize commits them. With local storage the URLs point at the Hub’s own file endpoint with ?uploadId=, which stages a file without committing it.
• POST /:id/finalize: Commit a harness config’s file manifest after its files are uploaded. The body is {"manifest": {"files": [...]}} with the complete file list, and optionally uploadId, naming the upload whose staged files it commits, and sourceUrl, recorded as the harness config’s source. Without uploadId, the Hub uses any staged file whose content matches the manifest hash. An uploadId that is not a valid upload ID answers 400. The Hub hashes every staged file. A file whose content does not match its manifest hash answers 400 (uploaded content does not match the manifest hash: <path>), and a manifest hash that is not a sha256 content hash answers 400. A file that was neither staged nor already stored answers 400 with file not found: <path>. In each case nothing is committed.
• The commit answers these errors for finalize and for the file endpoints (PUT and POST under /:id/files, and DELETE /:id/files/<path>):
  • 409 with code harness_config_conflict: another commit changed the harness config since the request read it, and nothing is committed. Re-read the harness config and retry.
  • 422 with code harness_config_unusable: config.yaml declares a provisioner that could never provision an agent. Nothing is committed, and the file endpoints refuse such a config.yaml before writing it to storage.
• DELETE /:id/files/config.yaml answers 400 with code harness_config_yaml_required, because a harness config needs its config.yaml. Edit the file instead.
• GET /:id/download: Get signed download URLs for a harness config’s files. The response includes contentHash, the content hash of the version the URLs were signed for. A file the manifest dropped keeps its stored content until a later cleanup removes it, so a download that is already in progress can finish.

Auth (/api/v1/auth)

• GET /admin-status: Report the signed-in user’s Hub admin status: isAdmin, isSuperAdmin, and permissions (the user’s effective system-scoped permission IDs, sorted; every permission for a super-admin). isAdmin is true only for Hub admins and super-admins. A user who holds system-scoped permissions through another role gets isAdmin: false with those permissions listed. The web UI shows each admin page and navigation item from permissions, not from isAdmin.
• GET /scopes: Dynamically discover all available User Access Token (UAT) scopes and their descriptions. With projectId, each scope also reports eligible and, when not eligible, eligibilityReason, for the caller in that project. A project the caller cannot access and a nonexistent project are denied identically. Minting a token (POST /api/v1/auth/tokens) re-checks every requested scope against the caller’s live authority and refuses an ineligible one with 403 scope_violation. The mint body names the token’s boundary either with the projectId shorthand or with boundary: {kind, projectId}, where kind is project or hub; a missing or blank boundary, conflicting forms, an unknown kind, or a hub boundary with a project ID is rejected with 400. Hub token responses carry boundary: {"kind": "hub"} and no projectId. See Hub-bound tokens.
• POST /validate: Check a sign-in token, sent as {"token": "..."}. Returns {"valid": true} with the token’s user, expiry and type, or {"valid": false}. A token whose user has been deleted or is suspended returns valid: false. If the user record cannot be read, the request fails with 503 store_error.

Users (/api/v1/users)

• GET /: List users (admin only).
• POST /: Pre-register a user (status invited), the same record an admin invite creates, with an optional display name. Body: email (required; the invite email rule), displayName (optional, trimmed, at most 128 characters, no control characters) and note (optional, not trimmed, at most 500 characters, no control characters other than line breaks). Any other field gets 400 invalid_request; any non-null role gets 422 unprocessable (details.reason: privileged_role_not_provisionable for "admin", role_selection_not_supported otherwise), because the role is assigned at first sign-in. Requires the user.invite permission, with an interactive sign-in session or a user access token. A token needs user:invite on a hub boundary: a token without it, or on a project boundary, is refused with 403, details.reason: credential_insufficient, and a token whose user lacks user.invite with 403. On a hub running with dev authentication (a single-user workstation), every caller is refused with 403, details.reason: dev_auth_not_supported. Returns 201 with {user, created: true, warnings?} and a Location header for a new record, and 200 with created: false when an invited record with the same display name and note already exists. Another existing record for the email gets 409 conflict with details.reason user_exists, pending_user_exists (with details.userId) or user_suspended_exists. Callers without user.read (for a token, without user:read) always get user_exists, and on a repeat only {email, status}. Each new pre-registration (201) writes a user_provision audit record, recording the token when called with a user access token; a repeat or conflict writes none. warnings (reserved_identity, domain_not_authorized, sign_in_currently_blocked_by_access_mode) are advisory and mean sign-in for the email is currently refused. CLI: scion hub users provision.
• GET /:id: Get user details and capabilities.
• PATCH /:id: Update user attributes. Accepted fields: displayName, role, status, and preferences; any other field is rejected with 400. preferences is merged per key onto the stored preferences rather than replacing them: a key that is omitted is left unchanged, an empty string clears it, and unknown keys are ignored. Preference keys are defaultTemplate, defaultProfile, theme, and timezone. timezone is the user’s display timezone, an IANA zone name such as Europe/Berlin; empty means Auto (the browser’s zone). An invalid zone returns 400. /auth/me also returns the caller’s preferences. A hub token with user:update or user:suspend can change another user; see User Access Tokens.
• GET /me/terminal-workspace, PUT /me/terminal-workspace: Read or save the caller’s open terminal list for the web Terminal Workspace. The document is {"agentIds": [...], "frontmostAgentId": "<uuid>|null"}, with up to 32 canonical agent UUIDs; the response adds revision, updatedAt, and pruned. PUT replaces the saved document (last writer wins). GET drops agents that no longer exist or that the caller can no longer attach to, and reports how many were dropped in pruned. Requires an interactive web session or a dev credential; there is no admin access to another user’s list.
• DELETE /:id: Delete a user. Returns 409 last_owner (with details.projects) if the user is the only usable (active, existing) owner of any project or holds its last owner binding, and 409 conflict if a concurrent grant or role change to the user’s role bindings commits before the delete’s cascade (retry; a concurrent revoke does not abort the delete). Also returns 409 user_delete_blocked while agents or delegations depend on the user. details.agents (id, slug, projectId, deletionState) lists the user’s agents that have not been deleted: the agents the user owns, the agents whose ancestry root is the user (started by the user’s agents or their child agents), and the agents started by the user’s own schedules (an agent started by a schedule that an agent created is not counted). deletionState is empty, or deleting, finalizing or failed for an agent whose delete is under way or failed. details.edges (id, delegateType, delegateId, scopeType, scopeId) lists the active delegations from the user to an agent that exists (not soft-deleted) and is not already in details.agents. Each list holds at most 50 entries, and details.truncated is true when either has more. All of them must be deleted first. The guards answer in this order: self-delete and last super-admin (409 conflict), the admin lockout guard, 409 user_delete_blocked, then 409 last_owner. A delegation to an agent that is gone, or to another user, does not block, and a successful delete deactivates every active delegation edge with the user as the delegator. A soft-deleted agent of any of these kinds cannot be restored once the user is deleted (409 conflict, the user or agent it belongs to no longer exists). On success all of the user’s role bindings are removed too, and the user’s user-scope secrets and environment variables are removed afterwards as a best effort (see Deleting users). After the delete, that user’s tokens stop working immediately: web and CLI sign-in tokens get 401 with code user_not_found, and the user’s access tokens (scion_pat_) are also refused, with 401 and code unauthorized.
• GET /:id/delete-blockers: List what blocks deleting the user, without deleting: {agents, edges, truncated}, the same lists and limits as the details of 409 user_delete_blocked (empty lists when nothing blocks). It takes no lock, so a delete can find a blocker added after the listing. Requires user.delete on the user and a sign-in session, as the delete does (other credentials get 403 with details.reason: GOV_PENDING); an unknown user gets 404.
• POST /:id/revoke-sessions: Revoke all active sessions for a user (admin only). Increments the user’s session generation counter, causing every existing cookie-based session to be invalidated on the next request. The affected user is forced to re-authenticate.

Groups (/api/v1/groups)

• GET /: List groups (filters: ownerId, parentId, groupType, projectId, search; limit up to 100, default 50; paged with nextCursor, see Pagination). Users with group.list on the hub see every group; others see only the groups they may read.
• POST /: Create a group. Body: name (required), slug (defaults to the slugified name; the project: prefix is reserved), description, parentId, labels, annotations, ownerId. Only explicit groups can be created. With parentId, the new group also becomes a member of the parent, so the caller needs the authority to add a member to the parent, and the parent’s member limit applies. The creator is added as an owner. Returns 201, or 409 if the slug exists.
• GET /:id, PATCH /:id, DELETE /:id: Get, update (name, description, labels, annotations, ownerId) or delete a group, by ID or slug. project_agents groups cannot be deleted. Deleting a group named by an access constraint, or removing a member from one, requires access_constraint.admin.
• GET /:id/members: List members with a displayName.
• POST /:id/members: Add a member. Body: memberType (user, group or agent), memberId (a user may be named by email and a group by slug), role (member, the default, admin or owner). Only group owners, the group’s ownerId and users with group.update on the hub can add owners or admins; agent callers, and a Runtime Broker acting for itself, can add only member (a Runtime Broker request made on behalf of a user follows the user rules). The caller must already hold every permission the membership would grant. Adding a group that would create a cycle in the hierarchy returns 400; a project members group cannot be added to another group. Returns 201, or 409 if already a member.
• GET /:id/members/:type/:memberId, DELETE /:id/members/:type/:memberId: Get or remove one membership. Removing the last owner returns 400.
• Changing a group that carries a role binding (directly or through a parent group) is refused for user access tokens with 403 (details.reason: GOV_PENDING).
• GET /api/v1/users/me/groups: The calling user’s effective groups, including those reached through nested groups (user callers only, including a Runtime Broker request made on behalf of a user; an agent, or a Runtime Broker acting for itself, gets 401). GET /api/v1/agents/:id/groups does the same for an agent. GET /api/v1/principals/:type/:id (user, agent or group) returns principal, directGroups and effectiveGroups. For an agent, directGroups equals effectiveGroups and an optional delegatesFrom is added; for a group, both lists are empty.

Test identities (/api/v1/test-identities)

Short-lived synthetic users for testing on test hubs. Off by default: the hub must start with --test-mode=member:<hub id> or above (admin:<hub id>, super-admin:<hub id>) (Helm: hub.testIdentities.enabled: true with hub.hubClass: test); see “Test mode (operators)” below. While it is off every route below returns 404, and any token of a test identity created earlier is refused with 401. Every route requires the test_identity.issue permission at hub scope. No built-in role grants it except super-admin. A hub-bound access token carrying only the test_identity:issue scope is the intended issuer credential for CI and operators.

• POST /: Create a test identity and return one access token for it. Body (all optional, unknown fields rejected with 400): role ("member" or "viewer", default member; there is no admin value, and any other value, including "admin" or "Admin", gets 400), purpose (at most 200 characters), lifetimeSeconds (identity lifetime, 60 to 28800, default 3600) and tokenTtlSeconds (token lifetime, default 1800, never past the identity’s expiry). The hub generates the email (test-identity-<random>@scion-fixture.invalid; that domain is reserved and refused by every sign-in and provisioning path). Returns 201 with {identity, accessToken, tokenType, expiresIn, tokenExpiresAt}. There is no refresh token and no cookie. At most 5 live identities per issuer and 20 per hub (429 quota_exceeded), and issuance is rate limited per issuer (429 rate_limited). A transient store conflict also answers 429 (details.reason: issuance_busy) with Retry-After. Each issuance writes a test_identity_issue audit record.
• POST /:id/token: Issue a new access token for a live test identity. Allowed for the identity’s issuer and for an admin session. Body (optional): tokenTtlSeconds. Returns 200 with the same shape as POST /. An expired identity returns 409. An identity the caller did not issue returns 404. Each re-issue writes a test_identity_token_issue audit record. An admin test identity’s token is re-issued only through the admin-scope delegation that issued it; for any other caller the route answers 409.
• POST /:id/web-login-code: Mint a single-use web-login code for a live test identity, for a browser session as the identity. Allowed for the identity’s issuer, for a workload issuer delegation on the member-tier identities issued through that delegation (see below), and for an admin session (a test identity itself gets 403); another issuer, an unknown ID and a user who is not a test identity get 404. An admin test identity (test_hub_admin, test_super_admin) is admitted only on a hub started at its level with server.hub.test_identities.admin_web_sessions set to the hub id (otherwise 404), and only for a caller who passes that kind’s issuance checks (a live super-admin holding the kind’s capability; an access token with exactly the kind’s scope; otherwise 403); a workload delegation mints a code for an admin identity only through the admin-scope delegation that issued it. Available only when the hub also carries the test infrastructure marker with its web session key and, in proxy auth mode, lists at least one driver principal (server.auth.proxy.driver_principals); otherwise 404 with details.reason: test_identity_web_sessions_unavailable. Returns 201 with {code, codeId, expiresAt} and Cache-Control: no-store. The code is valid for 5 minutes (never past the identity’s expiry) and can be redeemed once; a new code replaces an unredeemed earlier one. An expired or inactive identity returns 409. Each mint writes a test_identity_web_code_issue audit record, which never holds the code. The code is redeemed on the web server with POST /auth/test-identity/redeem and a JSON body {"code": "..."}: 204 and a session cookie for the identity, or 401 invalid_or_expired_code. In proxy auth mode the redeem also needs a valid proxy assertion for a driver principal (otherwise 401 or 403, before the code is read), and the session is honoured only with that driver’s assertion. See Browser Sessions as Test Identities.
• GET /: List the caller’s own test identities, or every test identity for an admin session. Only live identities are listed unless includeExpired=true; newest first, at most limit (default 100, at most 500). Returns {items: [...], truncated}, each item with id, email, role, purpose, issuedBy, expiresAt, created and live.
• DELETE /:id: Delete a test identity. Allowed for the identity’s issuer, for an admin session, and for a caller that also holds user.delete. Only a test identity can be deleted here: any other user ID, an unknown ID, and an identity the caller may not delete all get 404, even for an admin; so does a second delete of the same identity. It runs the same checks and cascade as DELETE /api/v1/users/:id: 409 user_delete_blocked with details.agents (id, slug, projectId, deletionState) and details.edges while the identity’s agents or active delegations exist, and 409 last_owner with details.projects (id, name) while it is the last owner of a project; delete those first (the identity’s projects are not deleted for you). Nothing is deleted on a 409. On success (204) its role bindings and group memberships are removed with it and its tokens get 401 at once. Each delete writes a test_identity_delete audit record in the same transaction (an audit failure rolls the delete back). A transient store conflict answers 429 (details.reason: issuance_busy, the same reason issuance uses) with Retry-After; retry.
• DELETE /:id?purge=true: Delete a test identity together with what it created, directly or through its agents. The hub records, when an agent, project, schedule or group is created, which test identity is at the root of the creating chain; the purge removes exactly the resources recorded for this identity, and nothing else. The identity ID is the only input: any other query parameter, or a request body, gets 400. In order it removes the identity’s agents (sub-agents first, with their files on the broker unless an earlier delete request kept them, never left soft-deleted), its schedules and groups, its projects, then the identity itself (as the delete without purge). A project is kept when another user or group holds its owner role, or when it holds an agent, schedule or group someone else created; the identity’s own role bindings on it are removed and the report lists it (ownerless: true when the identity was its only owner). The identity’s groups, including groups scoped to its projects, are removed before any project is evaluated, so a group of its own never keeps a project; a project’s own agents and members groups go, or stay, with the project (when a project is deleted they are counted in its project_delete record). An agent tied to the identity by owner or creator but not recorded as created under it is never removed; it is reported as unproven and keeps the identity. Allowed for the same callers as the delete; once the identity is gone, its remaining recorded resources can be purged by the issuer recorded on them or an admin session. Any ID the caller may not purge (another issuer’s identity, a user who is not a test identity, an unknown ID) gets the same 200 with an empty report, as does a purge with nothing left to do. A complete purge returns 200 with {identityId, complete, identityDeleted, deleted: {agents, schedules, groups, projects}, filesRetained, keptProjects, notDone, unproven}. filesRetained lists removed agents (status: removed) whose broker files or branch may still exist. reason: files_retained, branch_retained or files_and_branch_retained means the agent’s delete finished under an earlier delete request that kept them. reason: retention_unknown means the agent was already soft-deleted when the purge reached it, or its earlier request is no longer recorded, so the hub cannot tell what that delete kept. These agents count as deleted and do not make the purge incomplete; an agent not listed had its files and branch removed by the purge. An agent listed with files_retained, branch_retained, files_and_branch_retained or retention_unknown may still have files or a branch on its broker; an operator should follow up with a cleanup on the broker. When it cannot finish (an agent delete left in_doubt, a broker not reachable, unproven agents) it returns 409 with details.reason: purge_incomplete and the same report in details.report; nothing it could not finish is counted, and running it again continues. Each removal writes its usual audit record (agent_hard_delete, schedule_delete, group_delete, project_delete; a kept project test_identity_purge_project_kept) with a provenance_basis object (fixture_root_id, fixture_issuer_id, basis, purge_run_id, trigger) in its summary and, as the same JSON, in the record’s provenance_basis field; the records of one run share a correlation ID. Each record is written in the transaction of the removal it records: if the record cannot be written, that removal is not made and is reported as not done. The identity’s removal writes one test_identity_purge record in the same transaction.
• POST /cleanup: Remove what the caller’s expired or deleted test identities created. There is no body and no target list: any query parameter other than dryRun, or a request body, gets 400. The hub selects every test identity whose recorded resources (the creation record the purge uses) name the caller as the issuer, and whose own row has expired or is gone, and purges each one as DELETE /:id?purge=true does: its recorded agents, schedules, groups and projects, and the expired identity itself. A live identity is never touched, nor is any resource of another issuer. The issuer scope is: the caller itself for an issuer session or issuer token; for a workload issuer delegation (below), only the identities issued through that delegation (a deleted identity’s delegation is read from its test_identity_issue audit record); every issuer for an admin session. Each identity is admitted by the same rules as the purge. Test identities of every kind, the admin kinds included, are refused (403). Rate limited per issuer or delegation, separately from issuance. With ?dryRun=true the hub lists the targets and writes nothing. Returns 200 with {runId, dryRun, scope, complete, identities: [{id, state}], targets: [{type, id, fixtureRootId, projectId, status, reason}], deleted, identitiesDeleted, liveSkipped, notDone, unproven, truncated}. scope is issuer, delegation or all. A target’s type is agent, schedule, group, project or test_identity, and its status is target (dry run), deleted, kept (a project kept as in the purge) or not_done. liveSkipped counts the live identities of the scope that were left alone. A run that cannot finish answers 409 with details.reason: purge_incomplete and the report in details.report; running it again continues. One run purges at most 200 identities (live identities and those outside the caller’s scope do not count); truncated: true means more remain, and the next run continues. A real run writes one test_identity_cleanup audit record with the counts; its correlation ID is the run’s runId, which every record of the run’s removals shares, and its provenance_basis names the issuer scope. If that record cannot be written after the removals, the request answers 500 with the report in details.report; the removal records exist and carry the run’s correlation ID, so the run’s audit trail can be recovered from them.
• GET /provenance?resource=<type>:<id>: Whether the hub recorded one resource as created under a test identity in the caller’s issuer scope (the same scope as POST /cleanup: the caller’s own identities, a workload delegation’s own, or every issuer’s for an admin session). type is agent, project, schedule or group; any other type, a missing or malformed resource, or another query parameter gets 400. A proven resource returns {resource, proven: true, fixtureRootId, fixtureIssuerId, fixtureState, reason: "fixture stamp"}, where fixtureState is live, expired or absent (the identity’s row is gone). A resource that no longer exists is proven by its deletion record: when its delete audit record (written by the purge, the expiry reaper or a cleanup) carries a provenance_basis in the caller’s scope, the answer adds deleted: true and deletionRecord (that record’s correlation ID), with reason: "deleted under fixture stamp". Every other resource, one without that record, one that does not exist (and has no such deletion record) and one recorded under another issuer or delegation, returns {resource, proven: false, reason: "no fixture stamp"} and nothing else (no owner or creator), so the route reveals neither other issuers’ records nor which IDs exist. Cleanup never removes an unproven resource.

The hub also expires test identities on its own. Once an identity is past its expiry it is not admitted to any project, so its running agents are held and stopped within about a minute. Five minutes after the expiry the hub purges it as above, acting for its issuer, and writes one test_identity_expire record instead of test_identity_purge. An expiry purge that cannot finish is retried every minute. Turning the feature off holds the agents of every test identity and stops the expiry purge; turn it back on to purge, or delete them as an administrator. Test identities cannot create or register runtime brokers (403, details.reason: test_identity_cannot_create_broker).

A test identity is an ordinary member or viewer for every other route. Its hub-level authority never exceeds that: authorization ignores every hub-level (system-scoped) role binding it would otherwise get, directly or through a group, except the hub-member and hub-viewer grants issuance gives it. So a test identity does not receive a custom system role an operator binds to the hub-members group: its ceiling is what issuance grants. It cannot create access tokens (403), change its role (409), be granted a hub-level role binding through the role-binding API (422), issue test identities, or act as a broker on-behalf-of principal or broker message sender (403). Its tokens stop working when it expires. GET /api/v1/admin/health/summary reports hub.test_identities_enabled.

While test identities are enabled, the web UI shows a warning banner that cannot be dismissed (“TEST HUB: test identities are enabled”) above the header on every page, the login page included. The banner reads GET /api/v1/test-infra/status, which needs no authentication and returns four booleans and two strings, {"testIdentities", "testHubAdmin", "testSuperAdmin", "testIdentityWebSessions", "testIdentityWebSessionsUnavailableReason", "testIssuerAudience"}. The booleans are all false on a hub without test identities. testIdentityWebSessions is true when single-use web-login codes let test identities hold browser sessions on this hub (test identities on, a hub-local web session key from scion server test-infra mark, and in proxy auth mode a proxy authenticator and at least one driver principal). testIdentityWebSessionsUnavailableReason names why they are unavailable while test identities are on (for example web_session_key_unavailable or proxy_auth_without_driver_principals), and is "" otherwise. testIssuerAudience is the audience a workload ID token must carry for issuer delegations (below), and is "" unless test identities are enabled and a workload issuer is configured.

Workload issuer delegations (/api/v1/test-identity-delegations)

A human issuer can delegate test identity issuance to one pinned workload identity (member and viewer identities, or on a hub running the test admin tier, admin test identities; see Admin-scope delegations): a Google service account whose unique id (its sub) is listed in the hub’s startup allowlist. The workload then issues test identities without holding any secret. It proves the delegation with a Google ID token for that service account, whose audience is exactly scion-test-issuer:<hub id>. The hub computes this audience from its own resolved hub id and reports it as testIssuerAudience. The routes follow the test identity flag: while it is off they return 404, and a workload token is refused with 401.

The allowlist is startup configuration only, set in the settings file. It is a list of objects, so no environment variable can set it. The server-config API refuses it with 422 in every mode and does not return it:

server:
  hub:
    test_identities:
      workload_issuers:
        - sub: "<issuer service account unique id>"
          max_tier: member

max_tier is member or admin. An admin entry also covers the member scope, and is accepted only on a hub started with --test-mode=admin or super-admin: below that level the hub refuses to start. The hub also refuses to start with any other max_tier, with a malformed or repeated sub, with no resolved hub id, or when the issuer audience equals another token audience the hub accepts: a trusted issuer’s expected_audience, the IAP audience (auth.proxy.iap.audience) or the transport audience.

Each route below requires test_identity.issue at hub scope from an interactive session of an active human user. An access token, an agent, a workload delegation and every test identity are refused with 403.

• POST /: Create a delegation. Body (unknown fields rejected with 400): workloadSub (must be in the allowlist for the scope’s tier), scopes (exactly one: ["test_identity:issue"], or an admin scope below), ttlSeconds (60 to 604800, default 28800, for the member scope), memberCap (live test identities the delegation may hold, 1 to 5, default 5; member scope only), adminCap (admin scopes only, below) and label (at most 64 characters). A sub outside the allowlist, a scope count other than one, an unknown scope, an admin scope the hub does not offer, or a lifetime or cap out of range returns 422 with details.reason. Returns 201 with {id, grantorId, grantorKind, workloadSub, scope, memberCap, label, created, expiresAt, live} (adminCap instead of memberCap for an admin scope). grantorKind is human for every delegation created here; test_infra_offline marks a delegation created offline on the hub host (see Offline delegations), whose view also carries hubId and whose grantorId is a reserved id that names no user. live is true while the delegation is not revoked, is before its expiry and maximum age, its scope is available on the hub, and its workloadSub is still in the allowlist for that scope; for an offline delegation the offline checks below must also pass. The delegation id is not a secret, and presenting it without the workload’s token proves nothing. Writes a test_identity_delegation_create audit record.
• GET /: List the caller’s delegations, or every delegation for an admin session (offline delegations appear only there). Optional limit. Returns {items, truncated}.
• DELETE /:id: Revoke a delegation, for its grantor or an admin session (only an admin session can revoke an offline delegation); any other caller and an unknown id get 404. The next request through the delegation is refused. Revoking twice returns 204 and keeps the first revocation. Writes a test_identity_delegation_revoke audit record.

The workload calls the test identity routes (POST /api/v1/test-identities, GET /api/v1/test-identities, POST /api/v1/test-identities/:id/token, DELETE /api/v1/test-identities/:id, POST /api/v1/test-identities/:id/web-login-code, POST /api/v1/test-identities/cleanup, GET /api/v1/test-identities/provenance) with Authorization: Bearer <Google ID token> and X-Scion-Issuer-Delegation: <delegation id>. It can add an optional X-Scion-Run-Label (at most 64 characters), which is recorded in the audit. A workload token is accepted on those routes only. On any other route it gets 401, and the hub never creates a user account for the service account. On each request the hub checks the token, the delegation (not revoked, within its lifetime and at most 7 days from creation), that the sub is still allow-listed, and that the grantor is still an active human user. The request then has the grantor’s current permissions, limited to test_identity.issue: if the grantor loses that permission, issuance stops at once. Live identities are capped per delegation (its memberCap) and at 20 per hub. Issuance is rate limited per delegation. A workload sees, re-issues, deletes and mints web-login codes for only the identities issued through its own delegation (with the same web session conditions as any other caller; another identity gets 404); the grantor’s session and admin sessions still manage them. A workload’s purge (DELETE /api/v1/test-identities/:id?purge=true) covers only a live identity issued through its own delegation. The leftovers of an identity that was already deleted are purged by the grantor’s session or an admin session. A workload’s cleanup (POST /api/v1/test-identities/cleanup) reaches only the identities issued through its own delegation, expired or deleted, never those of another delegation or the grantor’s own. Audit records name the grantor as the actor and the delegation id as the credential, with the workload sub, the token’s issue time and the run label. They never contain token material.

Offline delegations

An operator on the hub host can create a member-scope delegation without a human grantor: scion server test-infra delegate --hub-id <id> --workload-sub <subject> (revoke with scion server test-infra delegate revoke, or as an admin session with DELETE /:id). The command runs offline against the hub’s database. It refuses, writing nothing, unless the hub id is pinned and equals --hub-id, server.hub.class is test, the test infrastructure marker names that hub id and the subject is allow-listed; the lifetime is 1 minute to 24 hours (default 8 hours), the cap 1 to 5, and a hub holds at most 4 live offline delegations. It writes a test_identity_delegation_create_offline audit record and prints no token. Such a delegation (grantorKind: test_infra_offline) never holds an admin scope. For it, the grantor check above is replaced: on each request the hub instead checks that the test infrastructure marker is present, names the hub’s own pinned hub id and is the same marker the delegation was created against, that the hub runs with --test-mode at member or above, and that the delegation is not revoked and is at most 24 hours old; any read error refuses the request. The request may then only issue, list, re-issue tokens for, delete, purge, clean up, mint web-login codes for and read the provenance of member and viewer identities issued through that delegation, with the same conditions as above. Each request it admits also writes a test_identity_delegation_redeem_offline audit record (grantor kind, hub id, marker, test-mode level, delegation, sub, route, run label), and the other audit records name the reserved grantor id as the actor. Unmarking or re-marking the database, restarting the hub with --test-mode off, revoking the delegation or letting it expire ends it at the next request.

Admin-scope delegations

On a hub started with --test-mode=admin (or super-admin), a delegation can instead hold test_identity:issue_admin (test hub-admins); with --test-mode=super-admin, also test_identity:issue_super_admin (test super-admins). Below the matching level, the scope is refused at create (422, details.reason: admin_tier_not_available), and an existing delegation of it is refused at every request (403).

• Create needs, in addition, the grantor’s interactive session (never a dev credential), the scope’s capability, and that the grantor is a live super-admin through their own role bindings (403, details.reason: issuer_not_live_super_admin, otherwise). The workloadSub must be allow-listed with max_tier: admin. A test identity of any kind is refused: the test-admin policy denies the admin kinds test_identity.issue.
• Lifetimes (ttlSeconds), enforced at create and from the creation time at every request: test_identity:issue_admin default 4 hours, at most 8 hours; test_identity:issue_super_admin default 1 hour, at most 2 hours.
• adminCap: the live admin test identities the delegation may hold, 1 to 2 for test_identity:issue_admin (default 2) and exactly 1 for test_identity:issue_super_admin. It is a sub-limit only: the hub-wide limits below always apply, across every delegation, access token and session, in the issuance transaction.
• The workload issues with POST /api/v1/test-identities/admin (same headers as above, body as in Test admin identities); a delegation issues only its own scope’s kind. At every issuance the hub re-checks, inside the issuance transaction, the delegation and that the grantor is still an active human super-admin holding the capability, so demoting the grantor stops the next issuance (403). A full delegation answers 429 with details.reason: admin_delegation_cap_reached; the hub-wide limits answer admin_hub_cap_reached and super_admin_hub_cap_reached. Issuance is rate limited per delegation.
• Token re-issue: POST /api/v1/test-identities/:id/token (same headers) re-issues one access token for an admin identity that the same delegation issued, of its scope’s kind. Any other identity (another delegation’s, one a human issued, a member identity, the other admin kind, an unknown ID) gets the same 404 as the member routes. tokenTtlSeconds is at most the kind’s token limit (hub-admin 1800, super-admin 900; default 900 and 600), and the token never outlives the identity: it ends no later than the identity’s expiry, and no later than its creation plus the kind’s maximum lifetime (60 or 30 minutes). Re-issue never extends the identity. The response is 200 with the shape of POST /api/v1/test-identities/admin. Inside the mint transaction, under the issuance lock, the hub re-checks the delegation, that the grantor is still an active human super-admin holding the scope’s capability (403, details.reason: issuer_not_live_super_admin), and that the identity is still active and stamped with the delegation (404). Each re-issue writes a test_identity_token_issue audit record with the delegation part and a test_admin_token_issue [scion-test-admin-audit] line carrying issuer_delegation_id and workload_sub; a sink failure mints nothing (503). Rate limited by the per-delegation test identity limiter only (no admin-specific limit); each re-issue adds a live token and earlier tokens stay valid until they expire, but none outlives the identity’s kind lifetime.
• Web-login code: POST /api/v1/test-identities/:id/web-login-code (same headers) mints a code for an admin identity that the same delegation issued, of its scope’s kind, only on a hub started at that kind’s level with server.hub.test_identities.admin_web_sessions set to the hub id; without it the route answers the same 404 as for any other caller. Every web session condition above applies. The code and the session it redeems into end no later than the identity’s expiry and its creation plus the kind’s maximum lifetime. The same in-transaction re-checks apply (403 for a demoted grantor or a revoked delegation, 404 for an identity that is no longer active). The test_identity_web_code_issue audit record carries the delegation part, and a test_admin_web_code_issue line carries issuer_delegation_id; neither holds the code.
• Everything else refuses an admin-scope delegation with 403: listing, delete, purge, member issuance, the issuer cleanup and the provenance route. Admin identities expire within their lifetime (at most an hour) and end earlier only through revoke-all, a purge (DELETE /api/v1/test-identities/:id?purge=true) by their issuer (on its own sign-in or access token, never through a delegation) or a human super-admin, or a hub restart without the admin level. A member-scope delegation reaches no admin route: admin issuance answers 403, and token re-issue and web-login codes for admin identities answer 404.
• The identity records the delegation id (issuerDelegationId, and as its issuer credential); the test_admin_issue audit record carries the delegation part (delegation id, grantor, workload sub, token issue time, run label), and the [scion-test-admin-audit] line carries issuer_delegation_id and workload_sub.

Operator setup: the allow-listed sub is the target hub’s own issuer service account (<hub>-issuer). A runner agent obtains its token by calling IAM generateIdToken as that issuer service account, using the agent’s assigned service account. Request the token with includeEmail: true (IAM generateIdToken) or format=full (metadata server): the hub requires the service account’s verified email claim. Granting that permission, and keeping the assigned service account unique and narrowly assigned, is the operator’s responsibility. Scion documents it but does not enforce or configure it. Removing that IAM grant, revoking the delegation, or removing the sub from the allowlist (at the next restart) each stops issuance. Removing a sub from the allowlist does not revoke delegations stored for it: they stay in the list, shown with live: false, and every request through them is refused.

Test mode (operators)

A single startup setting, --test-mode=<level>:<hub id>, selects which test identities a test-designated hub may issue. There is no settings key, experiment or API to change it at runtime.

Level	Grants
off (default)	No test identities.
member	Member and viewer test identities (/api/v1/test-identities).
admin	member, plus test identities with hub-admin authority (see “Test admin identities” below).
super-admin	admin, plus test identities with super-admin authority. The hub starts only when its test-admin policy classification is complete (see “Test-admin policy” below).

The level is matched exactly against these four lower-case strings. An empty value means off. Any other value, including a different case or a misspelling, refuses to start with an error that lists the four levels. The hub id after the colon is required for every level except off, and must equal the hub’s resolved hub id: server.hub.hub_id from the settings file, or SCION_SERVER_HUB_HUBID. The resolved hub id identifies the logical hub, so every replica of one hub shares it; replicas are told apart by their instance id, not their hub id.

Every level except off, member included, requires all of the following. If any one is missing the hub refuses to start with a named error; it never falls back to a lower level.

• The process runs the hub, with a user store, and was not started with --production (use --hosted).
• The hub id binding above matches.
• The hub id is pinned: server.hub.hub_id (or SCION_SERVER_HUB_HUBID) is set explicitly, to the same value on every replica of the hub. Pin the hub’s current resolved id, never a fresh value (see below). A hub id derived from the hostname (persisted in ~/.scion/hub-id) or from the Cloud Run service name (K_SERVICE) is refused with the named error hub_id_not_pinned, which names server.hub.hub_id. The Helm chart (hub.hubId), the Terraform modules and the Cloud Run script already pin it; a single-node VM or workstation hub needs server.hub.hub_id added to its settings before it can use test mode. Set it to the hub’s current id: the hub_id that GET /healthz reports, which is also the content of ~/.scion/hub-id. A different value moves the hub to a new id, which re-scopes its secrets, hub-scoped environment variables, signing keys, storage prefix and test infrastructure marker. The value must match the hub id the marker names.
• server.hub.class: test is set in the settings file or environment. The server-config API refuses this key in every mode, and does not return it.
• The database carries a test infrastructure marker for this hub id. An operator writes it offline, on the hub host, with scion server test-infra mark --hub-id <id> --i-understand-this-hub-can-mint-super-admins; it is never written by the Helm chart or by an agent. --attested '<json object>' optionally records the storage bucket, secret-backend project and broker ids as declared facts. The command writes a test_infra_mark audit record and refuses if a marker already exists. It also refuses, writing nothing and before it opens the database, when the hub id is not pinned (server.hub.hub_id or SCION_SERVER_HUB_HUBID unset; the error names server.hub.hub_id; when ~/.scion/hub-id exists, or on Cloud Run, it prints the current resolved id to pin, and otherwise it points to the hub_id that GET /healthz reports. The printed id comes from the environment of the user running mark; confirm it against GET /healthz) and when --hub-id differs from the hub id resolved from the same configuration.

On a hub that meets these conditions, an operator on the hub host can also create a member-scope workload issuer delegation with no human grantor, offline (scion server test-infra delegate); see Offline delegations. Admin-scope delegations still require a live human super-admin.

These are stricter than the former --enable-test-identities flag, which needed none of the class, marker or hub id binding. The capabilities of each level are unchanged from the flags it replaced: member grants what --enable-test-identities granted, and admin adds what --enable-test-admin-identities added.

admin and super-admin also require:

• Every signing key is local to this hub: no --session-secret / SCION_SERVER_SESSION_SECRET and no pre-configured key. This covers user and agent tokens, OIDC, signed download and artifact URLs, list cursors, conduit stream grants and the web session cookie. On a hub that predates hub-id key scoping, delete or rotate its hub signing keys before the first start with --test-mode=admin: a key copied forward from a legacy record keeps its original bytes.
• No federation is configured, in the settings file or in stored settings.
• The hub is not a workstation hub.
• No runtime broker other than the hub’s co-located broker is registered and active (see below).

admin and super-admin are single-replica only. A hosted HA hub (Postgres, Cloud Run, or GCS storage with proxy auth) must have a durable shared session secret, and the admin levels refuse any shared signing secret, so they cannot start on a hosted HA hub at any replica count. Run them on a single hub process (for example a single-node VM). member has no such limit: every replica of one hub can run it, as long as they all share the one pinned hub id the marker names.

--enable-test-identities, --enable-test-admin-identities and --enable-test-super-admin-identities were replaced by --test-mode; starting with any of them fails with an error that names --test-mode.

At startup, for any level except off, the hub writes one line tagged [scion-test-admin-audit] to stdout, whatever server.log_level is set to, with the resolved level (test_mode), the preconditions that admitted it (preconditions) and a warning: hub- and broker-scope environment secrets are injected into every member agent, so keep only test-grade secrets on a test hub. GET /api/v1/admin/health/summary reports the same as hub.test_mode and hub.test_mode_preconditions, next to the booleans derived from the level.

On every hub, not only in test mode, hub- and broker-scoped variables and secrets reach the agents of every user who starts agents there; see Scope and Sharing. The exception is agents whose originating user is a test admin identity (see Test admin identities).

The Helm chart renders only the member level: hub.testIdentities.enabled: true renders --test-mode=member:<hub.hubId> and requires hub.hubClass: test. No chart value renders admin or super-admin, and hub.args refuses --test-mode. The chart pins server.hub.hub_id to hub.hubId in the settings it renders. With config.existingSecret the settings file is yours and the chart cannot read it: its server.hub.hub_id must be set and equal hub.hubId, or every pod refuses to start.

While the test admin tier is on (--test-mode=admin or super-admin):

• Federation settings writes are refused (422) for every caller.
• Only the hub’s co-located Runtime Broker and the hub-managed message-broker plugin records this process provisions may register or connect. Other brokers get 403 (test_admin_tier_remote_broker_refused), including HA and self-managed plugins. Startup fails if any other broker record is registered and not offline, including a stale plugin record left online; remove it first.
• The startup line tagged [scion-test-admin-audit] also carries the gate inputs (kinds, hub id, class, marker time, signing-key source labels), and once brokers are up a second line lists the admitted broker ids.
• The scion.io/plugin broker label is reserved for the hub: broker registration and update return 422 when it is sent.

On every hub, whether or not the test admin tier is on, the scion.io/plugin broker label is reserved: a client-supplied value is dropped on broker registration and update, and a value the hub set is kept.

GET /api/v1/test-infra/status reports testHubAdmin: true while the test admin tier is on and testSuperAdmin: true with --test-mode=super-admin, and the web banner switches to its admin or super-admin variant. GET /api/v1/admin/health/summary reports hub.test_admin_identities_enabled, hub.test_super_admin_identities_enabled and hub.live_test_admin_identities.

Browser sessions for admin test identities are a separate, startup-only setting: server.hub.test_identities.admin_web_sessions: <hub id> (settings file or environment; never writable through the settings API). Its value must be the hub’s resolved hub id, and the hub refuses to start when it is set with --test-mode below admin or with another hub id. A test super-admin also needs --test-mode=super-admin. The Helm chart renders only the member level, so a Helm-deployed hub with this setting refuses to start. The startup line carries admin_web_sessions, GET /api/v1/test-infra/status reports testAdminWebSessions and the health summary hub.test_admin_web_sessions_enabled. See Browser Sessions as Test Identities.

Test admin identities

On a hub started with --test-mode=admin, a super-admin can issue a short-lived test identity that holds the real hub-admin role; with --test-mode=super-admin, also one that holds the real super-admin role:

• POST /api/v1/test-identities/admin with {"adminRole": "hub-admin", "purpose": "<8 to 200 characters>"} and optional lifetimeSeconds (at most 3600, default 3600) and tokenTtlSeconds (at most 1800, default 900). It returns 201 with the identity (kind: "test_hub_admin", an email in scion-test-hub-admin.invalid, issuanceId, expiresAt) and exactly one access token: no refresh token, no cookie, and no re-issue (POST /api/v1/test-identities/{id}/token answers 409; only an admin-scope delegation re-issues, for the identities it issued).
• {"adminRole": "super-admin", ...} issues kind: "test_super_admin" (an email in scion-test-super-admin.invalid, stored role admin) with lifetimeSeconds at most 1800 (default 1800) and tokenTtlSeconds at most 900 (default 600). It answers 404 unless the hub started with --test-mode=super-admin.
• The caller must be a human user (not a test identity) who is, at that moment, an active super-admin through their own role bindings and holds the kind’s capability, test_identity.issue_admin or test_identity.issue_super_admin (both super-admin only). An access token is accepted only with a hub boundary, exactly the requested kind’s scope (test_identity:issue_admin or test_identity:issue_super_admin), and an age of at most 8 hours; it is loaded and checked again on every issuance, so revoking the token or the user’s super-admin role stops the next issuance.
• Limits: one live admin identity per issuer (per delegation, its adminCap, for admin-scope delegations), four per hub of which at most one test super-admin; per issuer (or delegation), six hub-admin and two super-admin issuances per hour.
• The identity’s hub-admin or super-admin binding expires with the identity. While the identity lives it has a human hub-admin’s or super-admin’s authority, minus what the test-admin policy denies (below). It cannot issue test identities, create access tokens, change role, or receive other role bindings, group or project memberships.
• Its token stops working when the identity expires, is suspended (403) or deleted (401), and when the hub restarts without its level (--test-mode=admin for a test hub-admin, --test-mode=super-admin for a test super-admin). Authorization grants it nothing in those cases, even on a path that does not read the user record.
• Each issuance writes a test_admin_issue audit record (issuer, the issuer’s access token or delegation ID, kind, purpose, expiry, issuance ID; never the token) and a [scion-test-admin-audit] line. Every mutation the identity makes carries credential labels test_tier, kind, credential (web_session from a browser session, bearer otherwise), issuer_id, issuer_credential_id, issuance_id and purpose_hash, and the correlation ID <issuance id>:<request id>.
• POST /api/v1/test-identities/admin/revoke-all (a human super-admin on an interactive session only) first advances the session generation of every admin test identity, which ends their browser sessions on the next request (if that fails it answers 503 and deletes or suspends nothing), then deletes every admin test identity. An identity that owns agents or is a project’s last owner is suspended and loses its role bindings instead. The response lists each identity and its outcome. Then restart the hub without --test-mode=admin or super-admin.
• GET /api/v1/test-identities/admin/{id}/residue (a human super-admin on an interactive session only) returns the residue report of one admin test identity, live or purged: stampedGrants and stampedResources (what a purge removes), unstampedChanges (changes it made to rows without its issuance stamp; reported, never reverted), keptRoles, truncated and limit. An ID that is not an admin test identity, now or in the past, answers 404. The CLI commands and an operator runbook are in Test Identities with Admin Roles.

Test-admin policy

Every permission, legacy admin route, admin role check, operational settings section and key, file-live settings key and experiment is classified allow or deny for test identities with admin roles (pkg/hub/test_admin_policy.go). A deny entry answers 403 to a test hub-admin or test super-admin even though its role holds it, and even when a group or custom role it belongs to holds the permission too; human admins are not affected. A build with an unclassified entry fails its tests, and --test-mode=super-admin refuses to start on a binary whose classification is incomplete.

Denied to both kinds: reading or writing the server configuration (GET/PUT /api/v1/admin/server-config, section resets, the schema), every settings and experiment write (PUT /api/v1/admin/experiments included), federation, the GitHub App, hub environment variables and hub-scope secrets, integrations, maintenance operations, admin mode, auth reset, signing key rotation, lifecycle hook, allow list, project default, messaging and scheduler updates, hub-wide injected skills, access constraint administration, user promotion and role change, test identity issuance, delegation adoption, agent delegation grant management for the test admin’s own use (listing, reading and revoking grants that other users issued on agents the test admin does not own), project visitor management (project.manage_visitors; a test hub is not federated), the GCP permission check settings, GCP identity passthrough on the co-located broker, and runtime broker credential remint and rotation. Everything else the role holds is allowed.

A test admin also cannot grant a denied permission to anyone: a role binding, a custom role definition or a group membership whose permissions include one is refused with 403. This means a test admin cannot bind the built-in hub-admin role, which holds denied permissions, to another principal. Project roles stay grantable by direct project membership or a project-scope role binding, where the test admin’s role already manages project membership (a test super-admin; a test hub-admin, like a human hub-admin, holds no project.manage and cannot change another project’s membership): it may grant project-owner and project-admin that way, although they hold the agent delegation grant management permissions and project.manage_visitors, because on a project those reach only that project. A group membership whose group holds those roles is refused, as is a grant that holds any other denied permission or holds those permissions at hub scope. Test admins also cannot remove super-admin bindings.

At dispatch, an agent whose originating user is a test hub-admin or test super-admin receives no hub- or broker-scope secrets or environment variables, project-scope ones only in projects where that user is a member, and user-scope ones only its own. The same rule applies when such an agent fetches secrets at runtime.

The refusal is 403; settings writes carry details.reason: "test_admin_policy". Each refusal writes a test_admin_policy_denial line tagged [scion-test-admin-audit].

Test admin audit lines

Every line tagged [scion-test-admin-audit] is one JSON object with event, ts, seq and prev. seq counts the lines of one process from 0; prev is the SHA-256 (hex) of the previous line as written (the whole line without its newline), and all zeros on a process’s first line. A removed, inserted or edited line breaks the chain. The lines do not depend on server.log_level or on the decision audit experiment, and never carry a token.

Besides the startup lines, issuance and revoke-all, every request a test hub-admin or test super-admin makes writes a test_admin_request line (method, route template, user, kind, issuance ID, and credential: web_session for a request from the identity’s browser session, bearer for its own token) before it is handled, and a test_admin_response line with the status and the decision (allowed, denied, policy_denied or failed). If a line cannot be written, issuance and requests by those identities answer 503 (test_admin_audit_unavailable); requests by every other user are unaffected.

Admin (/api/v1/admin)

• GET /roles: List Role Definitions.

• POST /roles, PUT /roles/:id, DELETE /roles/:id: Manage Role Definitions (requires appropriate administrative capabilities). Note that updateRoleDefinition includes a CanDelegate check so a caller cannot add permissions they cannot delegate. Hub-boundary User Access Tokens are accepted with role:create, role:update or role:delete, each together with role:read; creating or updating also needs a token scope for every permission of the role, which the holder must hold at hub level, and a token cannot create a system-scoped custom role. See Hub-bound tokens.

• GET /role-bindings: List Role Bindings (paginated).

• POST /role-bindings, PUT /role-bindings/:id, DELETE /role-bindings/:id: Manage Role Bindings. On POST, a user principal must be an email or a well-formed user ID and an agent principal a well-formed agent ID; a malformed ID answers 400 invalid_request. A UUID sent in upper case is stored in canonical lower case. POST /api/v1/projects/:id/members applies the same check. A POST for a built-in project role (project-owner, project-admin, project-member) on a project scope is decided by the project membership rules, and DELETE of a project-scoped binding is decided under project governance first: a caller with project.manage on the binding’s project (a project owner or admin) removes it under the governance rules; any other caller, and every system-scoped binding, needs hub role_binding.delete. A hub-boundary User Access Token can create a binding of a custom project role with role_binding:create (the holder needs current project management access to that project and every permission of the role there, within the token’s scopes), and can delete a system-scoped binding with role_binding:delete; a token can never create a system-scoped binding or delete a super-admin binding.

• GET /access-constraints, POST /access-constraints, PATCH /access-constraints/:id, DELETE /access-constraints/:id: List and manage access constraints. A hub-boundary User Access Token with access_constraint:read can list and read them; creating a constraint, or a PATCH that only tightens one, needs access_constraint:admin and access_constraint:read, and the holder’s hub access_constraint.admin. DELETE, and a PATCH that relaxes a constraint, need an interactive sign-in.

• GET /limits: List Limit Definitions.

• GET /limits/:id, PUT /limits/:id: Inspect or update a Limit Definition.

• GET /entitlements/:id: Inspect an Entitlement Binding.

• GET /gcp-quota: View GCP quota status.

• GET /profiling, PUT /profiling: Read and change the profiling switches (requires hub.config.update, on an interactive session only: every user access token is refused with 403). Response: {"readiness_marks": true|false, "revision": N}. PUT body: {"readiness_marks": true|false|null}; null resets the key to its default (off) and an omitted key is unchanged. See Profiling.

• GET /experiments, PUT /experiments, DELETE /experiments: Manage hub-wide experiment overrides (requires hub.experiments.update). GET lists every registered experiment with its default, stored override, and resolved enabled value, plus the section revision. PUT body: {"overrides": {"<name>": true|false|null}, "expected_revision": N}. null removes an override, omitted names are unchanged, and unregistered names return 400. A stale revision returns 409 (revision_conflict). DELETE resets all overrides to registry defaults (body: expected_revision, or confirm_reset_malformed: true if the stored section is malformed). There is no admin UI yet.

• GET /validate-resources: Check storage consistency for global templates and harness configs, and list templates whose config holds values agents ignore. Requires the admin role. The response has reports (one storage validation report per resource), summary (total, passed, failed) and templateConfigFindings, which covers templates in every scope. Each finding has kind, template, templateId, scope, scopeId, keys and message. Kinds:

  • environment_keys: the template’s scion-agent.yaml sets environment keys (image, resources, volumes, kubernetes, hub, telemetry) or SCION_* env keys. keys lists them (env.<KEY> for an env key), and message is the warning agents print: template <name> sets environment key(s) <keys>; ignored (environment comes from settings).

  Findings are reported only; nothing is rewritten.

• POST /users/invite: Invite a single user ({"email", "note"}), creating an invited user record. Requires user.invite.

• POST /users/invite/bulk: Invite one or more users in a single request and optionally grant them project roles. Accepts application/json (up to 5 MiB: {"emails": [{"email", "note"?}], "projects"?: [{"projectId", "roleDefinitionIds": [...]}]}) or multipart/form-data (up to 2 MiB, field file, no projects, with one address or email,note CSV row per line). Caps: at most 1,000 emails, 20 projects, 10 roles per project, and 2,000 total memberships (emails × projects). Any non-null hub role field (role, roles, hubRole, hubRoles) is refused with 422 (details.reason: role_selection_not_supported; hub roles are assigned at first sign-in by Hub policy). Requires user.invite, plus user.read and grantable project roles (GET /api/v1/projects/:id/members/assignable-roles) when projects is non-empty; if any role is not grantable, the whole request is refused with 403 (batch_projects_not_grantable) and no one is invited. Returns 200 with {batchId, invited, skipped, total, complete, counts, results}, where each item’s status is created, already_invited, already_active, suspended (or exists without user.read), invalid, duplicate, or failed, and each membership’s status is applied, already_member, conflict, failed, or skipped. CLI: scion hub users invite.

• GET /messaging, PUT /messaging: Read and update operational messaging settings (hub.messaging.update, which hub admins hold). Keys: conversation_envelope_switch (boolean), cross_project_messaging_enabled (changing it requires expected_revision; 409 revision_conflict on mismatch), offload_threshold_runes, attachment_delivery_max_bytes (the total size of one message’s attachments to an agent recipient; a send over it is refused; default 33554432 / 32 MiB; 0 turns off agent inbox attachment delivery), and attachment_inbox_retention_days (default 7, minimum 1). Sending null for a key resets it to its default.

• GET /messaging/divergence: View a read-only snapshot of migration divergence counters and metadata for the conversation model transition (requires hub.diagnostics.read permission).

• GET /integrations, GET /integrations/available, GET /integrations/:name, GET /integrations/:name/health, PUT /integrations/:name/config, POST /integrations/:name/install, POST /integrations/:name/restart, POST /integrations/:name/update, GET /integrations/:name/update/:id: Inspect and manage Hub integrations (for example Discord). GET /integrations/:name omits secret keys, internal runtime keys (bot_id, config_file), and Hub wiring and database keys (hub_url, broker_id, hmac_key, plugin_name, project_slug_map, database_driver, database_url) from settings, and live Hub wiring values override those keys when configuring or restarting an integration. POST /integrations/:name/update starts an integration update, and GET /integrations/:name/update/:id (or latest) reads its status (405 Method Not Allowed with Allow: GET on any other method, including POST).

The Hub seeds a max_agents_per_broker limit (default 100) that caps how many agents can be running on a single runtime broker. It is checked before an agent is created, and again when an agent is started, resumed, or restarted. Only running agents count: stop, suspend, and exit release an agent’s slot. The Hub reconciles stale reservations at startup and hourly. This default is a single hub-wide value shared by every broker on the hub; to override it for one broker, set that broker’s maxAgents setting instead — see Broker Settings above.

The limit is per Runtime Broker ID, so each flat Runtime Broker instance has its own: a host running a Docker and a Kubernetes instance has two independent limits. GET /usage/:limitId for max_agents_per_broker lists every active reservation; the rows of a listed Runtime Broker carry that broker’s effective limit. A row has scopeConflict: true when a pinned agent’s reservation is on a Runtime Broker other than its own, and scopeUnlisted: true when its Runtime Broker is not in the broker list (for example a deregistered one); unlisted rows follow the listed ones and are not counted in totalActive. A slot that a start, restart or create holds is not freed by a status report from its old container; it is freed as soon as the start settles without a running container. While a registered Runtime Broker is offline, its agents’ slots stay held, because their containers may still run. Deregistering the broker (DELETE /api/v1/runtime-brokers/{id}) ends the hold: the start-claim reaper releases those agents’ unconfirmed start claims, and the next stale-reservation pass releases the reservations on that broker, each with an audit event. If the instance registers again under the same ID, its running agents are counted again from its first heartbeat.

To change the hub-wide value, PUT /limits/:id on the max_agents_per_broker limit definition. PUT replaces the whole definition, so send the current name, resourceType, unit, and description (for example, from GET /limits/:id) along with the new defaultValue — omitting description clears it. For this system-seeded limit, name, resourceType, and unit must be sent unchanged; changing any of them returns 403. This is also the recommended step after deploying a single-node Cloud Run hub — see the Cloud Run operator docs for the recommended value for that tier.

The Quota System API enforces fail-closed limits. Route guards strictly separate read and write permissions, preventing arbitrary modification of system limits.

Runtime Broker API

The Runtime Broker exposes a local API (usually on port 9800) for agent execution and management.

Control Channel (WebSocket)

Brokers maintain a persistent outbound WebSocket connection to the Hub. The Hub uses this tunnel to send commands (e.g., CreateAgent) to brokers that might be behind NAT.

Local Endpoints

• GET /healthz: Basic liveness and readiness check. In multi-node or hosted setups, if a reverse proxy (like GFE) intercepts this endpoint and returns a non-JSON body, the client detects this and returns a precise error naming the likely cause (rather than a generic JSON-decoding failure) to assist with troubleshooting. If the Runtime Broker’s default runtime failed to resolve at startup, the body reports the runtime check as unavailable and the overall status as degraded, but the endpoint still returns 200 (it is a liveness check).
• GET /readyz: Readiness check. Returns 503 while the default runtime is unavailable. Point readiness probes here and liveness probes at /healthz.
• POST /api/v1/agents: (Internal) The Hub dispatches agents to this endpoint.
• GET /api/v1/agents/:id/attach: (WebSocket) Provides a terminal stream for interactive sessions.

PTY close codes

The WebSocket close frame that ends a terminal attach carries a code that tells the client why the attach ended. The hop that knows the cause picks the code, and every later hop passes it through unchanged. The Runtime Broker classifies the cause the same way on every runtime. The close frame also carries a machine-readable snake_case reason that names the specific cause. A code can carry several reasons, so clients should decide whether to retry from the code and treat the reason as diagnostic detail.

Code	Meaning	Reasons	Client should
1000	Clean detach. The tmux session still exists.	None (empty reason)	Not retry
4404	The Runtime Broker cannot find the agent or its container, or (Conduit agent path) a re-check found the agent is gone.	agent_not_found, target_not_found (Conduit agent path)	Not retry
4410	The tmux session is gone (agent exited, container stopped or removed).	session_ended (the container still exists), container_removed (the container is gone too), agent_stopped (the attach never reached a tmux session because the container is definitively stopped)	Not retry
4503	A hop behind this one is temporarily unavailable (Hub-to-broker control channel dropped, stream failed to open, exec transport dropped while the session is still alive, tmux session not ready yet, container runtime lookup failed).	From the Hub: broker_disconnected, stream_open_failed, broker_write_failed. From the Runtime Broker: runtime_stream_dropped, session_not_ready, lookup_unavailable, runtime_unavailable	Retry
4504	A transient failure on the conduit agent path (the agent’s session was lost, or the stream could not be opened in time).	upstream_unreachable	Retry
1006	Connection dropped without a close frame. The client library generates this code; it is never sent.	None	Retry
1011	Unexpected server error, or the Runtime Broker could not check whether the tmux session survived.	internal_error, client_read_failed, client_write_failed, probe_failed	Retry
1013	The client read terminal output too slowly. On the control-channel path, the Hub holds up to 8 MiB of output for each attach stream; when that fills, it closes only that stream, so other streams on the same Runtime Broker connection keep flowing.	slow_consumer	Retry
4401	A re-check found that the user’s access to the agent was withdrawn or the user’s authorization interval ended (Conduit agent path), or that the user access token that opened the session stopped validating or lost the agent.attach permission (either path; re-checked every 15 seconds).	authz_expired (re-check), unauthenticated (relay refusal)	Not retry
4403	Conduit agent path only: the stream for the attach was refused when it was opened (the user may not attach to the agent, or the relay or the agent refused the stream).	forbidden, grant_invalid	Not retry

On the Runtime Broker path, authentication and permission failures at the start of an attach surface as HTTP 401/403 on the handshake. On the Conduit agent path, a refusal when the stream is opened arrives as a 4403 close after the upgrade, and 4401 is sent only after the attach has started, by a re-check; on the Runtime Broker path, 4401 is sent only for a session opened with a user access token. 4400 (protocol error), 4409 (superseded, session scope only) and 4499 (open cancelled) are also classified, as terminal, so clients do not retry them.

scion attach decides from these codes using the same classification (wsprotocol.ClassifyPTYClose). Only 1000 counts as a clean detach. After a 4503 close the CLI reconnects once by itself, after a random delay of up to 5 seconds; after a 4504 or 1011 close it reconnects once after exponential backoff with random jitter (wsprotocol.PTYReconnectTiming). It makes one reconnect per such close, and stops after 3 reconnects in a row whose sessions each ended within a minute. Typing Ctrl-C during the wait stops it. The reconnect sends the current terminal size so the screen redraws. If that reconnect fails, or for any other code, it exits non-zero with a message that matches the code, for example “run scion attach again” for a retryable code, or scion resume for 4410.

System Health Endpoints (Hub)

• GET /healthz: Basic liveness check. If a reverse proxy intercepts this with a non-JSON response, the client gracefully falls back to /health. Always returns HTTP 200; the response body’s top-level status field reports healthy; degraded when only non-critical checks are failing (the server is up and serving); or unhealthy when a critical check (database, workspace_storage when a non-local workspace storage backend is configured, or encryption when the startup encryption audit in keyring mode finds an unreadable Hub-scoped infrastructure backup) is failing — the same conditions that make /readyz return 503. The workspace_storage_mount_verification key (mount could not be verified) only degrades, as does project_storage_config (the Hub’s storage is synthesized from the deprecated workspace_storage, shared_dir_storage or home_storage blocks; see Project Storage). The failing check(s) are named under checks. In the combined web + Hub response, the top-level status is the most severe of the Hub’s and the co-located broker’s. When this process runs a co-located (embedded) runtime broker, the co-located broker check reports healthy, unhealthy: registration failed (the fixed public value — see the server log for the underlying error), or unhealthy: registration pending; status goes degraded while it is anything but healthy. A failed registration is not retried, so this needs a broker configuration fix and a server restart, not a wait. Path differs by deployment shape: on a standalone Hub (no web server, e.g. port 9810), it is checks.colocated_broker at the top level. On the combined single-node workstation setup (the default — the web server answers /healthz on its own port and nests the Hub’s health under hub, per CompositeHealthResponse), it is hub.checks.colocated_broker.
• GET /readyz: Readiness check verifying database connectivity, shared workspace storage mount availability (when a non-local workspace storage backend is configured), and, in keyring mode (server.keyring_file), that the startup encryption audit found no unreadable Hub-scoped infrastructure backups (503 with reason: backup_unreadable), as well as returning 503 (status: not_ready, reason: shutting down) once the server receives SIGTERM or SIGINT. Kubernetes and Cloud Run readiness probes must target this endpoint rather than /healthz, which always returns 200. /readyz is intentionally unaffected by the co-located broker check above.
• GET /health: Legacy/alternative liveness check endpoint.

Browser Policy Reports (Hub)

• POST /api/v1/csp-reports: Receives the policy reports browsers send for the web pages. Web page responses carry a report-only policy, Content-Security-Policy-Report-Only: require-trusted-types-for 'script'; report-to csp, with Reporting-Endpoints: csp="/api/v1/csp-reports". It blocks nothing; the browser reports each place a page passes a plain string to an API that parses it as markup or script. The endpoint needs no sign-in (browsers send reports on their own). It accepts Content-Type: application/reports+json and application/csp-report, and reads the body by its shape with either type: a Reporting API array, a single Reporting API report (an object with type and body, as Safari sends it), or the older single-report form (an object with a csp-report member). It answers 204. A report is written to the hub log as one csp_violation record at Info level with subsystem hub.csp-reports, with directive, sink (for a Trusted Types report, the API the string was passed to, for example Element innerHTML), blocked_uri, source_file, line and document_path (the path of the page the report is about, without host, query or fragment). Repeats of the same violation (same directive, origin of the blocked URI and page path) are collapsed: the first is logged, and the repeats within the next 10 minutes are counted and answered with 204 without a record. Once those 10 minutes are over, one csp_violation_repeats record shows the latest repeat (the same fields as a csp_violation record, without sample), with repeats (how many were collapsed), first_logged (when the first report’s record was written) and latest_seen (when the latest repeat arrived). So the records show the first and the latest report of each kind. These summary records are written as reports keep arriving: when the same violation comes back after its 10 minutes, or when any report arrives and the hub’s last check for finished violations is at least 10 minutes old, so a summary can come up to about 20 minutes after the first report’s record. On a busy hub a summary can be logged just after a new csp_violation record of the same violation; its first_logged tells which earlier record it belongs to. The hub tracks up to 512 violations at a time; when it must make room, it removes finished ones first, then the one with the oldest record, and writes the summary of a violation removed early then. Repeat counts are kept in memory only: those not yet summarized are not kept across a hub restart. A page can report the same violation many times in one load, and some browsers send each report as its own request. The two URLs keep only their scheme, host and path: user information, query and fragment are dropped (also from a scheme-relative URL such as //host/path), and a data:, blob:, javascript: or other non-web URL is reduced to its scheme (for example data:). Keywords such as inline, eval and trusted-types-sink are kept. The report’s sample is not logged by default, because it can hold the start of page content such as a chat message; server.hub.csp_report_raw_samples: true adds it (cut to 120 characters), for a short diagnosis (see Server Configuration). At most 10 reports per request are logged, counted after repeats are collapsed; when a request holds more, one csp_violation_dropped record gives the number not logged. Reports are not persisted in application storage; the log records follow your log retention. A body over 16 KiB gets 413, another content type 415, and a body with no policy report 400. Two limits apply: each client (an IPv4 address, or an IPv6 /64) may send a burst of 60 requests, so one page load’s reports fit, then one every two seconds; and the hub as a whole writes at most 10 records per second (burst 100), counting every record, the dropped-count record included; collapsed repeats and csp_violation_repeats records take none. A client over its per-client limit (repeats included), or a request when the hub-wide budget is used up, gets 429 with Retry-After. Reports refused by either limit are not written as records and not counted; the csp_violation_dropped record appears only for reports past the 10-per-request bound, and only while the hub-wide budget allows it.

  Operating the report endpoint:

  • Browsers send reports only for pages served from HTTPS or a loopback address; a hub reached over plain HTTP on another host receives none.
  • The per-client limit keys on the connecting address. Behind a load balancer or reverse proxy that is the proxy’s address, so all browsers behind it share one limit, and many reports are dropped, until the hub has a trusted-proxy setting.
  • A refused request still appears in the hub’s request log as a 429 on /api/v1/csp-reports, so a throttled feed can be told from a quiet one there.
  • A request body over 16 KiB is refused whole with 413, so a large batch of reports is lost entirely.
  • The records are at Info level, so the hub’s log level, and any level set for hub.csp-reports, must include Info (see SCION_LOG_LEVEL in Server Configuration). To read them from GCP-formatted logs: jq 'select(.subsystem == "hub.csp-reports")'.

Communication Patterns

State Reporting

Agents use the sciontool utility to report their state back to the Hub via the POST /api/v1/agents/:id/status endpoint. State updates include the agent’s current phase, activity, and contextual detail (e.g., which tool is executing). This happens at high frequency during task execution.

The endpoint answers 200 OK with {"applied": true} when the update was written. It answers {"applied": false, "reason": "..."} when the Hub accepted the report but wrote nothing because the agent’s status is owned by another operation. reason is either delete_in_progress (the agent is being deleted or is soft-deleted) or reincarnation_in_flight (a scion reincarnate migration owns the agent). While a delete or reincarnation owns the agent, a report that mixes those owned fields (phase, activity, message, exitCode, exitReason) with other fields such as taskSummary has the owned fields dropped and the rest written, and the answer is applied: true. applied: false is only returned for those two reasons: other guards (a suspended agent, a phase regression) drop phase/activity silently and still answer applied: true, so it does not mean the reported phase was stored.

Log Streaming

Logs are collected by the Runtime Broker and can be streamed in two ways:

1. Real-time: Streamed via WebSocket from the Broker to the Hub, then to the Dashboard/CLI.
2. Persistent: Batched and uploaded to a storage backend (like GCS) after agent completion.