Server Configuration (Hub & Runtime Broker)
This document describes the configuration for the Scion Hub (State Server) and the Scion Runtime Broker.
Configuration Location
Section titled “Configuration Location”Server configuration is defined in the server section of your settings.yaml file.
- Primary:
~/.scion/settings.yaml(Global settings) - Legacy:
server.yamlin~/.scion, in the--configpath, or in the working directory (Deprecated, but supported as a fallback whensettings.yamlhas noserverkey)
Structure
Section titled “Structure”schema_version: "1"server: env: prod log_level: info
hub: port: 9810 host: "0.0.0.0" public_url: "https://hub.scion.dev"
broker: enabled: true port: 9800 broker_id: "generated-uuid"
database: driver: sqlite url: "hub.db"
auth: dev_mode: falseSection Reference
Section titled “Section Reference”Hub Settings (server.hub)
Section titled “Hub Settings (server.hub)”Controls the central Hub API server.
| Field | Type | Default | Description |
|---|---|---|---|
port |
int | 9810 |
HTTP port to listen on (standalone mode). In combined mode (--enable-web), the Hub API is served on the web port instead and this setting is ignored. |
host |
string | "0.0.0.0" |
Network interface to bind to. |
public_url |
string | The externally accessible URL of the Hub (used for callbacks). Also editable in the admin UI; resolved at startup as described in Hub Endpoint Resolution: --base-url, then SCION_SERVER_HUB_ENDPOINT, then SCION_SERVER_BASE_URL, then a value saved in the database, then this file (the web server’s base URL uses only operator inputs and never reads the database; see Web Base URL). See server.hub.public_url: environment first. |
|
content_url |
string | Optional content host: a second public address, apart from public_url, on which shared artifact links and HTML artifact views are served. https://host[:port] only (http only for localhost or a loopback address), with no path, query, fragment or user info. Refused on save when it is on the same host name as public_url or agent_endpoint, at any scheme or port, and ignored at runtime in that case. A value on the same registrable domain as public_url (for example content.example.com for a Hub at hub.example.com) is refused on save and ignored at runtime unless content_url_allow_same_site is set; with the override it is accepted and logs a warning, because it is not a separate security origin for cookies. The supported setup is a content host on a different registrable domain, such as example-content.net. On a database-backed Hub, the GET /api/v1/admin/server-config response and the Admin > Server Config > Hub page report content_url_status (state: unset, in_effect with effective, ignored_invalid, refused_same_host, ignored_same_site with override_key: "server.hub.content_url_allow_same_site", or ignored_agent_endpoint_host). On a request whose Host is this value the Hub serves only the artifact share and view routes and answers 404 to everything else. Admins set it; user access tokens cannot. Applied without a restart; when it changes or is cleared, the previous host is no longer confined and is served like any other host, so remove its DNS record or proxy route at the same time (the Hub’s cookies stay host-only). See server.hub.content_url and Content host. Env: SCION_SERVER_HUB_CONTENTURL. |
|
content_url_allow_same_site |
bool | false |
Override that lets content_url be on public_url’s registrable domain. Without it such a value is refused on save and ignored at runtime, because a content host on the Hub’s registrable domain is not a separate security origin for cookies: the browser treats the two hosts as same-site, and a cookie scoped to that domain reaches both. The supported setup is a content host on a different registrable domain. With the override the value is accepted and the Hub logs a warning. A value on public_url’s own host name stays refused. Admins set it; user access tokens cannot. Applied without a restart, and every change is audited like content_url. Env: SCION_SERVER_HUB_CONTENTURLALLOWSAMESITE. |
agent_endpoint |
string | Optional override of public_url used only for the Hub URL injected into agents (SCION_HUB_ENDPOINT). Use when agents reach the Hub on a different address than users — e.g. an internal VPC URL — while invite links, chat-bridge links, the OIDC issuer default, and the cloudrun_invoker audience default keep using public_url. Must be scheme://host[:port] only: http or https, an IP literal or a hostname of letters, digits, _, -, and ., no path, query, fragment, or credentials (a trailing / is stripped); the Hub fails to start otherwise. When unset, agents receive the Hub’s regular endpoint (public_url, or the endpoint the Hub resolves when public_url is unset). Scope: injected into agents on every broker attached to this Hub, including remote brokers — see Splitting the agent endpoint from the public URL. Security: an http:// value sends agent bearer tokens and fetched secrets unencrypted; prefer https:// unless the network is trusted and isolated. |
|
internal_url |
string | Optional Hub URL that hub-managed broker plugins (Slack, Discord, Telegram, Teams) call the Hub API on: the hub_url they receive. Use it when public_url goes through an auth proxy or reverse proxy the plugins cannot pass, for example http://127.0.0.1:8080 on the Hub host. It is not a public endpoint, and nothing else reads this key. Plugins also receive public_url, but no plugin reads it yet: Slack, Teams, Telegram and Discord still build registration links from hub_url, so setting internal_url puts this address into those links (see the warning under Plugin API URL and public URL). When unset, plugins receive public_url as hub_url too, as before. Same form as agent_endpoint (scheme://host[:port]); the Hub fails to start otherwise. Read once at startup. Settable only in the settings file or with SCION_SERVER_HUB_INTERNALURL: refused by PUT /api/v1/admin/server-config in every mode, and GET leaves it out. See Plugin API URL and public URL. |
|
monitoring_dashboard_url |
string | Optional link to an external monitoring dashboard for this Hub, such as a Cloud Monitoring or Grafana dashboard. Must be an absolute http or https URL with a host, at most 2048 characters, with no user credentials, a port (if any) from 1 to 65535, and no whitespace (including Unicode spaces such as U+00A0), control characters (C0, DEL or C1), bidirectional formatting characters, invisible format characters (U+00AD, U+180E, U+200B to U+200D, U+2060, U+FEFF) or U+FFFD; a path, query and fragment are allowed. Any other value is rejected with 422 when saved through the admin API. When set, the Health page header shows Open monitoring dashboard, which opens the URL in a new tab; when unset, no link is shown. Editable in Server Config and applied without a restart. Only callers with hub.health.read receive it (as links.monitoring_dashboard in the health summary); it is not part of /api/v1/settings/public. Env: SCION_SERVER_HUB_MONITORINGDASHBOARDURL (seed: SCION_SEED_SERVER_HUB_MONITORINGDASHBOARDURL). |
|
gcp_project_id |
string | The Hub’s GCP project. Layer-0, read once at startup: set it in settings.yaml (server.hub.gcp_project_id) or with SCION_SERVER_HUB_GCPPROJECTID, which wins over the file. A legacy server.yaml hub.gcpProjectId is read only when settings.yaml has no server key: a settings.yaml with a server block replaces server.yaml as a whole (see Which GCP project the Hub uses). Used as the project for service account minting and as the Policy Troubleshooter quota project; when it is unset, both use the project of the GCP metadata server on GCE, GKE or Cloud Run. It is also the export project of the Hub’s own metrics and traces, and turns metrics export on when observability.metrics_enabled is unset. Not editable in the admin UI, and refused by PUT /api/v1/admin/server-config in every mode, workstation included (GET leaves it out), because it selects the service account minting project. See Which GCP project the Hub uses. |
|
observability |
object | Switches for the Hub’s own metrics and trace export. See Observability export. | |
gcp_iam_check_mode |
string | "off" |
Controls whether IAM actAs permission is checked when binding a GCP service account to an agent. Supported values: "off" (no check; default) or "enforce" (uses Policy Troubleshooter to enforce iam.serviceAccounts.actAs). "enforce" is strongly recommended for any Hub where agents receive GCP identities; see the caution under GCP IAM Check Mode for what "off" permits. See the security/permissions reference for details on roles and caches. |
gcp_iam_deny_unknown_policy |
string | "fail-open" |
Behavior when Policy Troubleshooter cannot evaluate deny policies (e.g. if the Hub lacks org-level reviewer roles). Supported values: "fail-open" (allow if no explicit deny is found; default) or "fail-closed" (treat as indeterminate and deny). |
read_timeout |
duration | "30s" |
HTTP read timeout. |
write_timeout |
duration | "60s" |
HTTP write timeout. |
admin_emails |
list | [] |
List of emails granted super-admin access. Listed users are always admins: they are promoted on sign-in. When the list is non-empty, an admin whose email is removed from it is demoted to default_user_role at the next hub restart or their next sign-in, whichever comes first. At restart, both admin_emails and the default role come from settings.yaml or the environment, so a change made only in the Admin UI (Postgres mode) takes effect at the user’s next sign-in. If the default role was set only in the Admin UI, a user demoted at restart becomes Member. Two exceptions: admins promoted from Admin > Users (or the users API) stay admins, and nobody is demoted if the startup safety check failed (for example, no existing user matched the list at startup and there were no UI-promoted admins); demotions resume only after the configuration is fixed and the hub is restarted. Roles set from the admin UI for users who were never config admins (member, viewer) are not changed by this list. |
auto_suspend_stalled |
bool | false |
Suspend an agent when the Hub marks it stalled, unless its harness does not support session resume. See Auto-Suspend of Stalled Agents. Editable in Server Config. Env: SCION_SERVER_HUB_AUTOSUSPENDSTALLED (seed: SCION_SEED_SERVER_HUB_AUTOSUSPENDSTALLED). |
stalled_threshold |
duration | "5m" |
How long a running agent with a recent heartbeat may go without activity events before the Hub marks it stalled. A value under 2m falls back to the default. Editable in Server Config. Applied without a restart when settings are stored in the database; in file mode, at the next restart. Env: SCION_SERVER_HUB_STALLEDTHRESHOLD (seed: SCION_SEED_SERVER_HUB_STALLEDTHRESHOLD). |
soft_delete_retention |
duration | Duration to retain soft-deleted agents (e.g., "72h"). The same window applies to agent directories that stale-directory cleanup keeps because their workspace may hold work: once it has passed, the next agent create or start that uses the same agents directory removes them, and the removal is logged. That check reads this value only from ~/.scion/settings.yaml or SCION_SERVER_HUB_SOFTDELETERETENTION on the machine that holds the directory (the CLI machine, or the Runtime Broker). A value set in a scion server start --config file or in the Hub’s admin settings is not seen. Unset or 0 keeps such directories until they are deleted. |
|
soft_delete_retain_files |
bool | false |
Preserve workspace files during the soft-delete period. When the retention window passes, the Hub asks the agent’s Runtime Broker to remove the kept files before it purges the record. While that removal runs, the Runtime Broker refuses a start of the agent with a retryable 409 retained_purge_in_progress. A Runtime Broker that cannot be reached, or whose removal keeps failing, is retried on every purge run for up to 30 days past the window; after that the record is purged and the files are reported as left on that Runtime Broker. |
launch_timeout |
duration | "5m" |
Whole-launch budget for an asynchronous create or start, from the moment the Hub begins the launch until the agent reaches a terminal Hub state. A launch that has not reached one by this deadline is ended and the agent is set to error (launch_timeout). A Go duration string such as "15m". There is no upper limit. A non-zero value below 30s is replaced by the default (5m) with a warning in the Hub log. In settings.yaml, a value that is not a valid duration is ignored and the default applies. Raise it for clusters with slow pod starts. Startup-only: restart required to change. Env: SCION_SERVER_HUB_LAUNCHTIMEOUT. |
launch_keepalive_seconds |
int | 15 |
Keepalive interval, in seconds, that the Hub sends to the Runtime Broker with each asynchronous create or start. A launch whose broker sends no report for 8x this interval (120s at the default) is ended and the agent is set to error (broker_lost). Values of 0 or less use the default. Startup-only: restart required to change. Env: SCION_SERVER_HUB_LAUNCHKEEPALIVESECONDS. |
missing_agent_grace |
duration | "3m" |
How long a running (or stopping) agent may be absent from its Runtime Broker’s heartbeat before the Hub marks it error with exit reason container_missing (an existing preempted or evicted exit reason and its message are kept). Applies only when the broker is online, reported a complete runtime inventory, and sent a recent previous heartbeat; agents with a lifecycle operation in progress are skipped. Values below "1m" fall back to the default. Env: SCION_SERVER_HUB_MISSINGAGENTGRACE. |
start_claim_lease_ttl |
duration | "90s" |
Lease of the claim the Hub takes before dispatching any agent start; the Hub process running the start renews it every third of this. Allowed 30s to 5m; other values fall back to the default. Hot-reloaded. Env: SCION_SERVER_HUB_STARTCLAIMLEASETTL. |
start_max_duration |
duration | "12m" |
Hard deadline on any agent start, including a wait for another Hub node to dispatch it. Minimum 11m (the broker’s pod-ready bound plus a minute). Hot-reloaded. Env: SCION_SERVER_HUB_STARTMAXDURATION. |
start_unconfirmed_hold |
duration | "17m" |
Longest time a start whose outcome is unknown (for example a dispatch timeout) keeps other starts of the agent waiting, until the runtime shows whether it created anything. Minimum 16m (the Runtime Broker’s 15-minute bound on a synchronous start, plus a minute). Hot-reloaded. Env: SCION_SERVER_HUB_STARTUNCONFIRMEDHOLD. |
start_create_unconfirmed_hold |
duration | "16m" |
start_unconfirmed_hold for a new agent’s create-and-start. The Runtime Broker may go on starting the agent for up to 15 minutes after the Hub stops waiting for it, so the allowed range is 16m up to start_unconfirmed_hold. A value from an earlier release below the minimum (for example 5m) is raised to 16m with a warning, and a lower start_unconfirmed_hold (for example 13m) to 16m; such values still save. Hot-reloaded. Env: SCION_SERVER_HUB_STARTCREATEUNCONFIRMEDHOLD. |
perf_trace |
bool | false |
Turns on per-request performance tracing for diagnosis. Observe only. See Request performance tracing and the developer guide. Startup-only: restart required to change. Env: SCION_SERVER_HUB_PERFTRACE. |
csp_report_raw_samples |
bool | false |
Adds each browser policy report’s sample to its csp_violation log record. Off by default: a sample can hold the start of page content (for a Trusted Types report, the text passed to the API, such as a chat message), and log records follow your log retention. Turn it on only for a short diagnosis, then off. See Browser Policy Reports. Startup-only: restart required to change. Env: SCION_SERVER_HUB_CSPREPORTRAWSAMPLES. |
membership_sweep_report_only |
bool | false |
Puts the membership-standing sweep in report-only mode: it logs and audits (agent_hold_would_set) each agent it would hold and stop, and holds and stops none. Off by default (the sweep enforces). Holds from membership changes made while it is on still apply. Set it on every replica: the sweep runs on whichever replica takes its lock, and a replica left enforcing holds the listed agents at its next sweep. For the first boot after an upgrade; see Upgrading: report-only first boot. Startup-only: restart required to change. Env: SCION_SERVER_HUB_MEMBERSHIPSWEEPREPORTONLY. |
cors |
object | CORS configuration (see below). | |
conduit |
object | Conduit relay settings (see Conduit). | |
port_proxy |
object | Agent port proxy settings (see Port proxy). | |
sse |
object | Browser events stream settings (see Events streams). | |
platform_request_timeout |
duration | Request timeout of the platform in front of the Hub, for example the Cloud Run service timeout. No default. When set, the events stream maximum age and the conduit lifetime_cap are lowered to 0.8 of it (see Platform request timeout). Between "150s" and "24h". Startup-only: restart required to change. Env: SCION_SERVER_HUB_PLATFORMREQUESTTIMEOUT. |
|
chat |
object | Web chat limits (see Chat search). |
CORS (server.hub.cors)
Section titled “CORS (server.hub.cors)”| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | true |
Enable CORS. |
allowed_origins |
list | ["*"] |
Allowed origins. |
Conduit (server.hub.conduit)
Section titled “Conduit (server.hub.conduit)”Settings for the in-process conduit relay and its stream grants. They take effect only when the hub.conduit experiment is on. All of them are read at startup, so a change needs a restart. An invalid value is a startup error, not silently ignored.
| Field | Type | Default | Description |
|---|---|---|---|
grant_key_activation |
duration | "15m" |
Delay between publishing a new grant signing key and signing with it. Minimum "1m". Targets must refresh their keys at least this often. Flag: --conduit-grant-key-activation. Env: SCION_SERVER_HUB_CONDUIT_GRANTKEYACTIVATION. |
tcp_allowed_ports |
list of int | [] |
Additional agent-local ports a TCP stream grant may target, on every agent, besides that agent’s exposed ports. The reserved ports (9810, 18380) are always refused. Empty means exposed ports only; this setting never narrows access to exposed ports. Flag: --conduit-tcp-allowed-ports. Env: SCION_SERVER_HUB_CONDUIT_TCPALLOWEDPORTS (comma-separated). |
disable_ssh |
bool | false |
Turn off the agents’ embedded SSH server: scion ssh, scion tunnel --ssh, scion ssh-config and VS Code Remote-SSH (shell, exec, sftp, and SSH port and socket forwarding). TCP tunnels, the port proxy and terminal attach keep working. The Hub refuses SSH tunnels as soon as it restarts with the setting, and agents started after the change do not serve SSH at all. Env: SCION_SERVER_HUB_CONDUIT_DISABLESSH. |
deny_ports |
list of int | [] |
Agent-local ports no tunnel may reach, on every agent: TCP tunnels, the port proxy (over Conduit or the port-forward tunnel), and SSH direct-tcpip forwarding. It wins over exposed ports and tcp_allowed_ports. The Hub refuses TCP streams and proxy requests to these ports, and agents started after the change refuse them as well, including SSH forwarding, which the Hub cannot see. Env: SCION_SERVER_HUB_CONDUIT_DENYPORTS (comma-separated). |
disable_ssh_socket_forwarding |
bool | false |
Refuse SSH Unix socket forwarding (direct-streamlocal) to every socket in the agent. Applied by agents started after the change. Env: SCION_SERVER_HUB_CONDUIT_DISABLESSHSOCKETFORWARDING. |
internal_listen |
string | host:port of the internal relay API listener, used by multi-node deployments. It serves only the internal relay API and must be reachable only inside the cluster or VPC, never publicly. Without internal_tls_cert_file and internal_tls_key_file it must be a loopback address (a loopback IP literal: 127.0.0.0/8 or ::1, not localhost); see TLS on the internal hop below. Setting internal_advertise without it is a setting error: an HA hub refuses to start, and a single-node hub logs it and runs without the relay. Flag: --internal-listen. Env: SCION_SERVER_HUB_CONDUIT_INTERNALLISTEN. |
|
internal_advertise |
string | Base URL (http(s)://host:port) other hub nodes use to reach this node’s internal listener. It must be https:// unless its host is loopback (a loopback IP literal: 127.0.0.0/8 or ::1, not localhost). Default: POD_IP with the listen port, else the listen host if it is not a wildcard, with https:// when the listener serves TLS. Flag: --internal-advertise. Env: SCION_SERVER_HUB_CONDUIT_INTERNALADVERTISE. |
|
internal_tls_cert_file |
string | Path of the PEM certificate chain the internal listener serves TLS with. Set together with internal_tls_key_file. Re-read when the file changes (at most once a minute), so a rotated certificate needs no restart. Flag: --internal-tls-cert-file. Env: SCION_SERVER_HUB_CONDUIT_INTERNALTLSCERTFILE. |
|
internal_tls_key_file |
string | Path of the PEM private key of internal_tls_cert_file. Flag: --internal-tls-key-file. Env: SCION_SERVER_HUB_CONDUIT_INTERNALTLSKEYFILE. |
|
internal_tls_ca_file |
string | Path of the PEM CA bundle every internal hop client verifies peers with. When set it replaces the system roots. A private or internal CA requires it: the system roots do not validate such a certificate, and the hop then fails. Read at startup. Flag: --internal-tls-ca-file. Env: SCION_SERVER_HUB_CONDUIT_INTERNALTLSCAFILE. |
|
internal_tls_server_name |
string | Name internal hop clients verify in the peer certificate instead of the dialed host, so one certificate (for example with the name scion-relay.internal) can serve every node dialed by its pod IP. Set the same value on every node. Default: the dialed host. Flag: --internal-tls-server-name. Env: SCION_SERVER_HUB_CONDUIT_INTERNALTLSSERVERNAME. |
|
peer_auth |
string | "auto" |
Relay-peer authentication: auto, oidc or hmac. Requests between relays are always signed with a key derived from peer_secret, the server.keyring_file signing ring (when peer_secret is unset, carrying X-Conduit-Peer-Key-Id and X-Conduit-Peer-Signature-Version: 2 when signing.emit is "versioned"), or the hub’s signing secret, which is required in every mode. oidc also requires a Google OIDC ID token, auto adds the ID token on GCP, and hmac uses the signature alone. Used only when this node serves the internal relay API (internal_listen set, or any HA hub): a single-node hub without internal_listen starts its relay with no relay-peer authentication and refuses requests to and from other relays. Env: SCION_SERVER_HUB_CONDUIT_PEERAUTH. |
peer_service_accounts |
list | own service account | With OIDC peer auth, the service-account emails allowed to call the internal relay API. The hub logs a warning at startup when the default resolves to a Compute Engine default service account. Env: SCION_SERVER_HUB_CONDUIT_PEERSERVICEACCOUNTS (comma-separated). |
peer_audience |
string | "scion-conduit-relay-peer" |
With OIDC peer auth, the ID token audience. It must be identical on every hub node. Env: SCION_SERVER_HUB_CONDUIT_PEERAUDIENCE. |
reconnect_window |
duration | "5s" |
Jitter window sent with a planned close: targets redial after a random delay within it. Between "0s" and "5m". Flag: --conduit-reconnect-window. Env: SCION_SERVER_HUB_CONDUIT_RECONNECTWINDOW. |
authz_recheck_interval |
duration | "60s" |
Period of the authorization re-check sweep of open user streams: port proxy streams (including user requests on the legacy port-forward tunnel), terminal attach streams on the agent’s Conduit session, and scion tunnel streams. User streams are also re-checked when the user’s access changes and when the hub’s event connection is re-established; Runtime Broker placement lookups are shared across streams evaluated in the same sweep. A stream whose user lost the permission is closed with 4401 authz_expired within one interval even if the revocation event is missed; revocations normally close it at once. Between "1s" and "10m". Flag: --conduit-authz-recheck-interval. Env: SCION_SERVER_HUB_CONDUIT_AUTHZRECHECKINTERVAL. |
lifetime_cap |
duration | "3500s" |
Lifetime cap of a conduit session. Set it below the platform’s connection limit (Cloud Run request timeout, load balancer or proxy timeout). 60 seconds before the cap the relay sends GoAway: the session takes no new streams, the agent reconnects, and streams still open are closed with 4503 relay_restart within 30 seconds and never later than the cap. A PTY client sees that close as 4503 and reconnects. Server-sent events are not affected. Between "90s" and "24h". When platform_request_timeout is set, the cap is lowered to 0.8 of it (never below "90s"). Each session gets its own jittered cap so sessions opened together do not all reconnect together: between 80% and 120% of the cap when platform_request_timeout is set, and between 80% and 100% of it otherwise, because the cap is then the platform limit itself. Flag: --conduit-lifetime-cap. Env: SCION_SERVER_HUB_CONDUIT_LIFETIMECAP. |
shutdown_grace |
duration | Time the platform allows between SIGTERM and killing the hub process. No default; the Cloud Run deployment profiles set "10s". While the conduit relay runs, the hub handles SIGTERM (and SIGINT) by sending GoAway to every conduit session before its listeners close, with a drain deadline of at most shutdown_grace minus 2 seconds when this key is set (a value of 2 seconds or less gives a 1 ms deadline); when it is unset there is no cap and the drain deadline stays 30 seconds. Agents open a replacement session and streams still open at the deadline are closed with 4503 relay_restart. In external relay mode with the ingress proxy serving there is no relay in the hub: the hub stops proxying conduit dials, asks the relays to drain the sessions it proxied, and closes the proxied connections still open when that drain step ends (see Separate Conduit Relays). When set, it is also the whole hub process’s shutdown budget: the co-located Runtime Broker’s shutdown, the HTTP drain and the background teardown each get a share of shutdown_grace minus 2 seconds (half of shutdown_grace when it is 4 seconds or less), and the process exits within shutdown_grace even if a step has not finished (exit status 1 in that case). When it is unset, the drain stays 30 seconds and each shutdown step keeps its own bound, so one stop can take a minute or more (up to 35 seconds for the co-located broker, then up to 30 seconds of HTTP drain): on a hub run by systemd or Kubernetes, set it below the stop timeout (for example "20s" for a 30-second stop timeout). From the first SIGTERM or SIGINT, /readyz answers 503 and new requests other than health checks and Runtime Brokers’ own requests get 503 with Retry-After; a second signal forces the shutdown (exit status 1). Must be a duration greater than 0. Read at startup; a change needs a restart. Env: SCION_SERVER_HUB_CONDUIT_SHUTDOWNGRACE. |
|
proxy_session_max_age |
duration | "1h" |
How long a user’s Conduit session admitted through trusted-proxy authentication (no bearer credential, so no expiry of its own) may open new tunnels, counted from admission. After it, a new tunnel request on that session is refused with 401; tunnels already open are unaffected. Between "5m" and "1h". Env: SCION_SERVER_HUB_CONDUIT_PROXYSESSIONMAXAGE. |
stream_authz_max |
object | see description | Authorization interval of open streams, per originating principal kind (user, broker, agent). When a stream reaches its interval, the hub re-checks the principal: if the permission still holds, the stream is renewed for one more interval; if not, it is closed with 4401 authz_expired (4404 target_not_found if the agent is gone). If the check cannot be evaluated (for example a store outage), the stream is never renewed or closed early; it is closed with 4401 authz_expired at the end of its current interval. Revocations and the authz_recheck_interval sweep close a revoked stream sooner, but never extend it. Defaults: user "8h", broker "24h", agent "24h", each between "1m" and "168h". Only user is enforced today (port proxy streams, including user requests on the legacy port-forward tunnel, terminal attach streams on the agent’s Conduit session, and scion tunnel streams); the broker and agent values are validated but not yet applied. Flag: --conduit-stream-authz-max-user. Env: SCION_SERVER_HUB_CONDUIT_STREAMAUTHZMAX_USER, SCION_SERVER_HUB_CONDUIT_STREAMAUTHZMAX_BROKER, SCION_SERVER_HUB_CONDUIT_STREAMAUTHZMAX_AGENT. |
instance_id |
string | see description | This node’s relay instance id. It must be unique among live hub processes: a relay that starts with an id already in use takes it over from the other process. Up to 128 printable ASCII characters, no spaces. Default: POD_NAME when set, else the host name plus a random per-process suffix. Env: SCION_SERVER_HUB_CONDUIT_INSTANCEID. |
relay_mode |
string | "in_process" |
Where conduit sessions are held. in_process runs a relay in every hub process. external runs none: separate relay processes (scion server --role relay, see Relay Settings) hold the sessions, and the hub serves no conduit sessions itself. External mode requires peer_secret, internal_listen (the relays call the hub internal API on it) and the hub.conduit experiment; startup fails without them. Layer 0 (bootstrap); read at startup, restart required. Env: SCION_SERVER_HUB_CONDUIT_RELAYMODE (or SCION_SERVER_HUB_CONDUIT_RELAY_MODE). |
ingress |
string | "proxy" |
How a hub in external relay mode serves a conduit dial. proxy reverse-proxies the dial to the relay chosen by selection, over a relay-peer leg; it requires HMAC peer authentication: under any other resolved peer_auth (OIDC, including auto on GCP) the hub starts with the proxy off and conduit dials get 503 conduit_not_serving (cause ingress_requires_hmac). redirect answers with the chosen relay’s public URL. Ignored with relay_mode: in_process. Layer 0 (bootstrap); read at startup, restart required. Env: SCION_SERVER_HUB_CONDUIT_INGRESS. |
target_stream_deadlines |
string | "off" |
Whether targets enforce user streams’ authorization deadlines locally. off leaves enforcement with the hub or relay node that holds the stream. With enforce, a target closes a user stream at its authorization deadline unless it is renewed. The hub publishes the value to targets with the grant keys (GET /api/v1/conduit/grant-keys). User streams to targets that support renewal carry their deadline in the open grant, and the node holding the stream renews it at about 80% of the interval, in both modes (in external relay mode, hub-held streams renew over the bridge to the separate relay, and tunnels held by a separate relay are re-checked and renewed by that relay’s stream scheduler when the forwarding relay advertises tunnel_recheck, carrying route relay_tunnel in refusals and unstamped counts). With enforce, an open stream that the hub or relay cannot keep renewed is refused with 503 unavailable (reason stream_renewal_unavailable) on tunnels, the port proxy and PTY attach. Today that covers a relay-held tunnel forwarded by an older relay that does not support tunnel re-check, and users whose identity has no renewal path (federated users, whose grants carry no deadline under off). The hub logs a warning at startup when enforce is set with relay_mode: external. Also with enforce and relay_mode: external, a user stream to an agent whose sciontool does not support stream renewal is refused before it opens: 409 agent_upgrade_required (reason stream_renewal_unsupported) on tunnels and the port proxy, and 503 runtime_attach_unsupported with the same reason on PTY attach. Restart such an agent on an image whose sciontool supports stream renewal. In that mode a user PTY attach never uses the runtime broker’s terminal path, and goes over the agent’s own session instead. In that mode user requests never fall back to the older routes: when the agent has no conduit session, or conduit is not available on the hub, the port proxy and PTY attach answer with a retryable 503 (unavailable), and exposed ports that only the port-forward tunnel can serve (a non-loopback host) are unavailable. Because streams are renewed in both modes, streams opened under off keep being renewed after a switch to enforce. A stream that stops being renewed, for example one held by a node that is no longer running, ends at its deadline (at most the configured user stream_authz_max). Requires the hub.conduit experiment. Set it in the settings file or the environment; scion server start has no flag for it. Env: SCION_SERVER_HUB_CONDUIT_TARGETSTREAMDEADLINES (or SCION_SERVER_HUB_CONDUIT_TARGET_STREAM_DEADLINES). |
peer_secret |
string (secret) | Dedicated relay-peer secret: the HMAC key of every call between the hub and separate relays, and between relays, is derived from it (unless peer_key_file and peers are configured; see below). Raw octets, used as given: no decoding. Exactly one trailing newline is removed, then the value must hold at least 32 octets, at least 16 of them distinct. Generate one with openssl rand -base64 32 and use the output as it is. Required in external relay mode and on --role relay when peer_key_file is not set, where startup fails without it; in in_process mode it is validated but not used; the hub also refuses to start if it equals the session secret. It is independent of the session secret: a relay never holds the session secret. Never stored in the database or logged, and never returned by the server-config API; a non-empty value in a server-config PUT is refused, and an empty or null value is ignored. It is never passed into agent or workload environments. Prefer a file-mounted settings.yaml over the environment. Layer 0 (bootstrap); read at startup, restart required. Env: SCION_SERVER_HUB_CONDUIT_PEERSECRET (or SCION_SERVER_HUB_CONDUIT_PEER_SECRET). |
|
peer_key_file |
string | Opt-in per-peer Ed25519 private key file (0600 owner-only JSON {"keyId": "<id>", "privateKey": "<base64 32-byte seed or 64-byte Ed25519 key>"}) used by this hub or separate relay to sign relay-peer requests with X-Conduit-Peer-Signature-Version: 3 (scion-conduit-peer-v3) and X-Conduit-Peer-Key-Id. Set together with peers (if either peer_key_file or peers is set, both are required, and this node’s own instance_id must appear in peers with its role — "hub" or "relay" — and a matching keyId and public key). When configured, this node signs its outbound relay-peer calls with its Ed25519 key instead of a shared secret and verifies each inbound v3 call only against the enrolled public keys and role of the caller named in X-Conduit-Peer-Caller (hub: calls require role: "hub", relay: calls require role: "relay"), so a compromised relay’s key cannot forge calls from another relay or from a hub. Never returned by the server-config API (PUT refuses a non-empty peer_key_file or peers). Layer 0 (bootstrap); read at startup, restart required. Env: SCION_SERVER_HUB_CONDUIT_PEERKEYFILE. |
|
peers |
list | Enrolled relay-peer public keys for per-peer Ed25519 authentication (peer_key_file): each entry has id (the peer’s server.hub.conduit.instance_id), role ("relay" or "hub"), and keys (one or more {"keyId": "<id>", "publicKey": "<base64 32-byte Ed25519 public key>"} entries so keys can rotate with overlap). Layer 0 (bootstrap); read at startup in settings.yaml, restart required. |
Restricting what users reach in agents. disable_ssh, deny_ports and disable_ssh_socket_forwarding help when agents share their container with local services that users of the Hub should not reach through it, such as a credential proxy or a control socket. The Hub sends them to each agent at create, start and restart as SCION_CONDUIT_POLICY, replacing any value from project, template or agent env, and sciontool enforces them where it makes the connection. Only agents running a sciontool that knows the setting enforce it; the Hub-side checks (SSH tunnels, TCP ports, the port proxy) apply to every agent. An agent that cannot read the value, including one that meets a field it does not know, refuses SSH and every port until its image is updated. deny_ports and disable_ssh_socket_forwarding restrict forwarding only. They are not a boundary against a user who can run commands in the agent: an SSH shell, or a terminal attached with agent.attach (the tmux session), can connect to any local service itself. disable_ssh removes the SSH shell but not terminal attach; a service that users with agent.attach must not reach should not run in the agent’s container.
Rotating grant keys. POST /api/v1/admin/conduit/grant-keys/rotate (unscoped Hub administrators only, hub.conduit_grant_keys.execute) rotates in a new grant signing key with an overlap window: grants signed with the outgoing key stay valid until the earlier of their own expiry and the outgoing key’s NotAfter. Once the new key activates, the outgoing key can no longer sign; the Hub keeps only its public half. After NotAfter the Hub stops publishing the key, but keeps its public half for the stream renewal check for one stream_authz_max.user interval plus 30 seconds. The response lists only key IDs and timestamps.
Revoking a grant key. To withdraw a grant key at once, rotate first, wait until the new key is active (activate_at in the rotate response), then call POST /api/v1/admin/conduit/grant-keys/revoke with {"kid": "<old key id>"} (same permission as rotate). The key stops verifying anything at once. Streams whose latest grant it signed are not renewed again and end at their current authorization deadline. The key that signs now cannot be revoked.
Upgrading hub replicas. Upgrade all hub replicas to this version before the next grant key rotation, and not within one overlap window of a rotation (the activation delay plus the overlap, about 1 hour 15 minutes with the defaults). An upgraded hub does not rewrite a stored key ring that has one active key and no successor.
TLS on the internal hop. The internal hop (between hub nodes, and between hubs and separate relays) carries user and agent credentials and session content, so it is never plaintext off the host. With internal_listen set, the hub or relay refuses to start when:
internal_listenis not a loopback address (a loopback IP literal: 127.0.0.0/8 or::1, notlocalhost) andinternal_tls_cert_fileandinternal_tls_key_fileare not set;internal_advertise, or on a separate relayserver.relay.hub_internal_url, ishttp://to a host that is not loopback.
Hub and relay clients also refuse to send anything to a peer endpoint that is http:// to a host that is not loopback. Only loopback IP literals count; a host name, localhost included, is never resolved for this decision and is refused. There is no override. To fix a refusal, give every node a certificate and set internal_tls_cert_file and internal_tls_key_file. If that certificate comes from a private or internal CA, also set internal_tls_ca_file on every node. Alternatively, bind internal_listen to 127.0.0.1 behind a TLS terminator on the same host, and advertise that terminator’s https:// address. When upgrading multi-node hubs or separate relays, set these keys on every node in the settings file or the environment (not as --internal-tls-* flags, which older binaries refuse) before upgrading, then upgrade all hubs and relays together; see Upgrading.
Hosted HA. In an HA deployment each hub node runs a relay that other nodes must reach directly, so the hub refuses to start when:
- the grant key ring is not stored with the shared at-rest key;
- it runs on Cloud Run (
K_SERVICEis set), whose instances are not individually addressable; internal_advertiseuses the public hub host or a*.run.apphost;- the relay is not addressable at its internal endpoint, or answers its self-check as another instance.
Outside HA, a relay that cannot start is logged and the hub serves without it.
Port proxy (server.hub.port_proxy)
Section titled “Port proxy (server.hub.port_proxy)”Settings for the agent port proxy (/api/v1/agents/{id}/ports/{port}/...) when it runs over the agent’s Conduit session (the hub.conduit experiment). Read at startup, so a change needs a restart. An invalid value is a startup error.
| Field | Type | Default | Description |
|---|---|---|---|
response_header_timeout |
duration | "60s" |
How long the proxy waits for the service on the agent port to send its response headers, counted from when the request has been sent. This includes the WebSocket handshake. If the headers do not arrive in time, the proxy cancels the upstream request, answers 504 (code runtime_error; a browser gets the proxy error page) and counts it in scion.hub.port_proxy.upstream_timeout. The response body has no time limit: server-sent events, downloads, chunked responses and upgraded WebSockets keep flowing once the headers have arrived. Between "5s" and "10m"; 0 is not accepted. Env: SCION_SERVER_HUB_PORTPROXY_RESPONSEHEADERTIMEOUT. |
view_ttl |
duration | "2h" |
With the hub.port_view experiment on, how long a page link stays valid. The hub creates one when a user opens an exposed port from the hub. Opening the port’s link from the hub again creates a new one. The lifetime is fixed when the link is created; using the link does not extend it. Between "15m" and "24h". Env: SCION_SERVER_HUB_PORTPROXY_VIEWTTL. |
Events streams (server.hub.sse)
Section titled “Events streams (server.hub.sse)”Settings for the browser events stream (/events, server-sent events), which the web UI keeps open for live updates (agent heartbeats that change no visible state do not emit status events on /events). Read at startup, so a change needs a restart. An invalid value is a startup error. Both are Layer 0: a hosted Hub’s server-config API refuses them.
| Field | Type | Default | Description |
|---|---|---|---|
max_connection_age |
duration | "10m" |
Maximum age of an events stream. At that age the Hub sends a reconnect event and closes the stream, and the browser reconnects at once. Each stream gets its own age between 80% and 120% of this value, so streams opened together do not all reconnect together. When platform_request_timeout is set, the value is first lowered to 0.8 of it. Between "1m" and "24h". Env: SCION_SERVER_HUB_SSE_MAXCONNECTIONAGE. |
heartbeat_interval |
duration | "30s" |
Interval between the heartbeat events of an events stream. The web client treats a stream that has been silent for 75 seconds as dead, so the maximum is "1m". Between "5s" and "1m". Env: SCION_SERVER_HUB_SSE_HEARTBEATINTERVAL. |
Each stream’s maximum age is recorded on the stream and appears as max_age_ms in its events stream closed log line.
Platform request timeout
Section titled “Platform request timeout”Many platforms and proxies end a request after a fixed time, for example the Cloud Run service request timeout (300 seconds by default, at most 3600). Long-lived requests that run into that limit are cut by the platform instead of being ended by the Hub, and while they stay open each one holds a request slot (on Cloud Run, one unit of containerConcurrency).
Set server.hub.platform_request_timeout to the platform’s timeout. The Hub then ends events streams and conduit sessions before it:
- Events stream maximum age:
min(sse.max_connection_age, 0.8 × platform_request_timeout), then the per-stream jitter of ±20%. With a 900-second timeout and the defaults, streams end after 8 to 12 minutes, well before 900 seconds. - Conduit lifetime cap:
min(conduit.lifetime_cap, 0.8 × platform_request_timeout), never below 90 seconds, then the per-session jitter of ±20%.
server: hub: platform_request_timeout: "900s" # the Cloud Run service timeoutWhen the Hub runs on a Cloud Run service (K_SERVICE is set) and platform_request_timeout is not set, it logs one warning at startup that names the setting. Startup continues.
Chat search (server.hub.chat.search)
Section titled “Chat search (server.hub.chat.search)”Bounds on web chat search, so one search cannot stall the hub. Each search runs with a time limit that starts when it takes its slot and covers finding the caller’s visible projects, the database query and the direct-message filters; a search that reaches it is stopped in the database and answered with 503 (code SEARCH_TIMEOUT), never 500. Each user runs at most one search at a time, and a hub-wide cap applies; a search over either limit is answered at once with 429 (code SEARCH_BUSY, Retry-After: 1), never queued. On a SQLite hub, searches read through a separate read-only connection, so a slow search does not hold the connection other requests use. On a Postgres hub, searches share the main connection pool, so the hub-wide cap is lowered to a quarter of database.max_open_conns (at least 1). A search that takes 500 ms or more, times out or fails writes one chat search slow log line with its shape (duration, backend, scope, term length, result count), never the search text, a request id or any other identifier; a search the client abandons writes no line at the default log level (only a debug line). Every search that runs is counted in the histogram hub_chat_search_duration_seconds (attributes backend, path, scope, outcome: ok, timeout, canceled or error), and refused searches in the counter hub_chat_search_busy_total. Read at startup; restart required to change.
| Field | Type | Default | Description |
|---|---|---|---|
timeout |
duration | "3s" |
Time limit of one search; a positive Go duration. The admin settings update refuses an invalid value with 422; at startup an invalid value is logged and the default used. Env: SCION_SERVER_HUB_CHAT_SEARCH_TIMEOUT. |
max_concurrent |
int | 2 |
Searches running hub-wide at once. 0 means the default. On SQLite it is also the size of the read-only search connection pool. On Postgres, where search shares the main connection pool, the hub lowers it to a quarter of database.max_open_conns (at least 1; 1 with the default pool of 5) and logs one startup line with the configured and effective values. Env: SCION_SERVER_HUB_CHAT_SEARCH_MAXCONCURRENT. |
Observability export (server.hub.observability)
Section titled “Observability export (server.hub.observability)”Switches for the Hub’s own metrics export (Cloud Monitoring) and trace export (Cloud Trace). These switches are separate from the agent telemetry.* settings. Both are Layer-1 settings in the hub_observability section. You can set them on the Server Config page (Hub Server tab), through PUT /api/v1/admin/server-config as a Hub admin, in settings.yaml, or with the environment variables below. The Hub creates its exporters once at startup, so a change applies after a restart. A save lists both keys under pending_restart (“applies after restart”).
| Field | Type | Default | Environment variable | Layer | Restart | Description |
|---|---|---|---|---|---|---|
metrics_enabled |
bool | unset | SCION_SERVER_HUB_OBSERVABILITY_METRICSENABLED |
1 (hub_observability) |
required | true: the Hub exports its metrics. false: it does not. Unset: metrics export runs when server.hub.gcp_project_id is set, as before this switch existed. |
tracing_enabled |
bool | unset | SCION_SERVER_HUB_OBSERVABILITY_TRACINGENABLED |
1 (hub_observability) |
required | true: the Hub exports traces. false: it does not. Unset: trace export runs when SCION_TRACING_ENABLED and server.hub.gcp_project_id are both set, as before this switch existed. |
Precedence. The general rule for Layer-1 keys applies (see Precedence). A database row an admin has saved wins over settings.yaml and the environment, and the environment and settings.yaml only seed the row and act as the fallback. A key set by its environment variable gets the “Overridden by environment on this node” badge on the page.
Export project. When a switch is true, the exporter sends data to server.hub.gcp_project_id if the operator set it (environment, else the server settings file: settings.yaml, or a legacy server.yaml when settings.yaml has no server key). Otherwise it uses the project of the GCP metadata server, the same fallback service account minting uses; off GCP the metadata server is not queried. If no project resolves, the Hub logs a warning that names SCION_SERVER_HUB_GCPPROJECTID and starts without that exporter. Startup never fails for this. An unset switch keeps the earlier rule, which only considers an operator-set project ID. The export project, endpoint and credentials cannot be edited in the admin UI. The page shows the export project and its source (env, file or metadata) read-only, together with each exporter’s state at startup and the reason for it.
Which GCP project the Hub uses
Section titled “Which GCP project the Hub uses”The Hub reads every one of these sources once, at startup. A change applies after a restart.
| Use | Project, first match wins |
|---|---|
| Hub metrics export (Cloud Monitoring) and trace export (Cloud Trace) | 1. server.hub.gcp_project_id: SCION_SERVER_HUB_GCPPROJECTID, else the server settings file (see below). 2. The project of the GCP metadata server (on GCE, GKE or Cloud Run only). The page shows the source as env, file or metadata. |
| Metrics dashboard (the Hub queries Cloud Monitoring for agent usage) | 1. telemetry.cloud.gcp_project_id from settings.yaml or SCION_SERVER_TELEMETRY_CLOUD_GCPPROJECTID. 2. The project_id of the scion-telemetry-gcp-credentials Hub secret. 3. server.hub.gcp_project_id as configured above. The metadata server is not asked on this path. |
| Service account minting and the Policy Troubleshooter quota project | 1. server.hub.gcp_project_id as configured above. 2. The project of the GCP metadata server. |
Server settings file. The Hub reads its server settings from one file, not key by key: when settings.yaml has a server key, that block is used as a whole and a legacy server.yaml is ignored (the Hub logs a deprecation warning when both exist). So a server.yaml hub.gcpProjectId is not read when settings.yaml has a server block, even one without gcp_project_id; move it to server.hub.gcp_project_id. Only when settings.yaml has no server key is server.yaml read, and then its hub.gcpProjectId applies.
server.hub.gcp_project_id is settable only in the settings file or the environment: the server-config API refuses it in every mode.
Upgrade note: earlier releases ignored server.hub.gcp_project_id in settings.yaml. A Hub whose settings.yaml already sets it now uses that value at the next start: Hub metrics export turns on while server.hub.observability.metrics_enabled is unset, metrics and traces go to that project, and service account minting and the Policy Troubleshooter use it instead of the metadata server project. SCION_SERVER_HUB_GCPPROJECTID still wins. Remove the key, or set it to the project you want, before upgrading. A legacy server.yaml hub.gcpProjectId behaves as before: it applies only when settings.yaml has no server key.
Startup log. The Hub logs one line for each exporter. The line says whether the exporter is on or off and why, and names the export project when it is on. For example:
Hub trace export: off (server.hub.observability.tracing_enabled unset; off because SCION_TRACING_ENABLED is not set)Hub metrics export: on (project: my-project (source: metadata); server.hub.observability.metrics_enabled=true)WARNING: Hub metrics export: off (server.hub.observability.metrics_enabled=true but no export project resolved (...); starting without the exporter. Set SCION_SERVER_HUB_GCPPROJECTID or run on Google Cloud)Metrics are exported every 60 seconds and every trace is sampled. Neither can be configured.
Asynchronous agent create
Section titled “Asynchronous agent create”A synchronous Hub agent create holds the create request open while the Runtime Broker provisions the workspace and starts the agent. Each dispatch to the broker is bounded by the Hub’s fixed 180-second request timeout. The Hub extends that request’s write deadline to cover it, and scion start and scion create wait up to 4 minutes for the answer, so the synchronous path is still not suited to agents that take longer than that to start.
Asynchronous create is the hub.async_agent_launch experiment, on by default. An admin can turn it off, or back on, in Admin → Server Config → Experiments; no restart is needed, and the Hub checks it each time it sends a create to a broker. With it on, the Hub answers as soon as the broker accepts the create. The agent is returned in a pre-running phase with an active launch, and the broker finishes the start in the background:
- Opt-in per request. A create is asynchronous only when the experiment is on and the request opts in (
acceptAsyncLaunch). Provision-only creates and reprovisioning are always synchronous.scion startandscion resumeopt in on every Hub create and then poll the agent until it isrunning,error, orstopped. By default they wait for the Hub’s remaining launch budget plus 30 seconds (5 minutes when the Hub does not report a budget);--wait-timeoutoverrides this and--no-waitreturns once the Hub has accepted. Seescion start. The web UI’s create form also opts in: the agent page shows the current launch step while the agent provisions, and if the create does not complete, its failure code and reason with Delete and Create again. Scheduled agent creates opt in server-side. The Discord integration’s/scion threadopts in unless itsasync_createsetting isfalse; it reports each launch step in the thread and binds the thread once the agent is running. Requests that do not opt in are synchronous. - The wait is not the launch. If the CLI stops waiting (the wait budget runs out, or Ctrl-C), the launch continues on the Hub and broker, and the agent still comes up. Run
scion start <agent>again to resume waiting. - Bounded by
launch_timeout. A launch that has not reached a terminal Hub state by its deadline is ended: the broker stops it shortly before the deadline, and the Hub sets the agent toerrorwith launch errorlaunch_timeoutshortly after. Starting such an agent is refused withagent_create_incomplete; delete it and create it again. - Broker did not take the launch. If the broker refuses the create, cannot be reached, or acknowledges a different launch, the Hub ends the launch with launch error
dispatch_failed(“the runtime broker did not accept the launch”), soscion startand other clients following the launch stop waiting and report the failure. A create usually removes the agent in that case. An agent kept after a failed env submission or workspace finalize stays inprovisioning: submit the env again once the Hub has released the failed start’s claim, afterstart_create_unconfirmed_hold(16 minutes by default). If such an agent later stops or fails, starting it is refused withagent_create_incomplete; delete it and create it again. - Turning it off. Creates sent after the experiment is turned off are synchronous. Launches the broker has already accepted are not cancelled: they finish, or are ended by
launch_timeoutor broker loss, as before. - Rolling back. Turn the experiment off and wait until every Hub node has applied the change: a node usually applies it at once, and at the latest at its next settings poll (every 60 seconds, plus up to 10 seconds of jitter). Then wait at least
launch_timeoutplus 15 seconds more (the Hub’s launch reaper runs every 15 seconds) before you downgrade any Hub node or Runtime Broker to a version without asynchronous create, so launches already in flight end under the newer code. Turning the experiment off needs no restart. - Starts during a launch. While a launch is in progress and before its deadline, a start or restart of that agent returns the launching agent with HTTP 200 instead of starting it again (a restart adds the warning
agent is launching; restart not performed), andscion startkeeps waiting. Restore, reincarnate, and wake are refused withagent_launching.
Requirements. The Hub, the Runtime Broker, and the CLI must all run a version that includes asynchronous create. A broker that reports no async launch support, or that answers the create synchronously, gets the synchronous create, so mixing versions is safe but slow starts on an older broker keep the synchronous limits. A broker runtime that cannot serve an asynchronous launch (currently substrate) also falls back to the synchronous create.
Broker restarts during a launch. The Runtime Broker keeps each asynchronous launch it has accepted in a launch journal on disk, so a launch is not left waiting for the Hub’s broker_lost or launch_timeout when the broker stops or crashes:
- Graceful stop (SIGTERM). The broker first refuses new creates, starts and restarts with HTTP 503, error code
broker_drainingandRetry-After: 5. It stops each launch in progress and reports it failed with launch errorbroker_restarted(“the runtime broker shut down during the launch”). When the Hub confirms, the broker removes what the launch created. It waits up to 10 seconds for this, out of its shutdown budget (30 seconds, or its share ofshutdown_gracewhen that is set). A report that is not delivered in that time stays in the journal for the next start. The Hub treatsbroker_drainingas a start the broker did not act on: it releases the agent’s start claim at once, so a retry after the drain is not blocked, and the start and restart actions (POST /api/v1/agents/{id}/startand/restart) answer with a retryable HTTP 503, error codebroker_draining, and the broker’sRetry-After. Stops, deletes and launch reports are still served while the broker drains. When the Hub and the broker run in one server process (for example the workstation server, or a combined Hub and broker deployment), the drain runs while the in-process Hub still accepts the brokers’ launch reports: the Hub refuses new user requests once the shutdown begins, but keeps its listener open until the co-located broker and every flat broker instance in the process have stopped, and only then begins its own HTTP drain. A report that is not delivered in time stays in the journal and is sent when the server starts again. - Crash or kill. When the broker starts again with the same state directory, it reports each unfinished launch failed with
broker_restarted(“the runtime broker restarted during the launch”). The report goes out under the old broker process’s identity. When the Hub confirms, the broker removes the launch’s runtime resources (pods, secrets, containers). It also removes the agent’s files if the launch was a create and still owns them. A launch whose success was decided but not yet delivered is reported as succeeded, and nothing is removed. If the Hub gives no definitive answer in time (from a running Runtime Broker or from recovery), the Runtime Broker keeps the journal entry, asterminal_pending, and the launch marker, so the next Runtime Broker start resends the report; after 5 unanswered attempts or 24 hours since the first, recovery logs a warning and removes the record with no cleanup, leaving the rest toscion delete. - Failed cleanup. If removing what a launch created fails, the journal keeps the launch as
cleanup_pending, and the next start of the Runtime Broker retries the cleanup it recorded (only the launch’s own resources, when that was the scope). After 5 failed attempts, or 24 hours after the first failure, the Runtime Broker logs it, drops the record, and leaves the rest toscion delete. - What the journal holds. It lives under
<state-dir>/launches/, where the state directory is<server.broker.state_dir>/<broker-id>(by default~/.scion/runtime-broker-state/<broker-id>). Directories are mode 0700 and files 0600. Entries hold only identifiers: launch, agent, run and project IDs, the agent name and project path, the runtime name, the latest step and the created resource handles. They never hold environment variables, secrets, tokens or the request. - Local filesystem only. Each broker process locks its own journal with
flock. Broker replicas that share one broker ID must each use a local filesystem or a distinct state directory, becauseflockis not reliable on NFS or other shared filesystems. - Ephemeral homes. Crash recovery needs the state directory to survive the restart. The Hub chart keeps the broker home on an
emptyDirby design: a container restart in the same pod keeps it, but a new pod does not. A graceful stop still reports in-flight launches. After a crash that ends in a new pod, or a reschedule that does not stop the pod gracefully, nothing reports those launches: the Hub ends each one asbroker_lostonce the broker has sent no report for 8 timeslaunch_keepalive_seconds(120 seconds by default), or aslaunch_timeoutif the launch deadline comes first. Runtime resources the launch had already created (pods, secrets) are not removed then;scion deleteremoves them with the agent. To recover such launches instead, choose a persistent state directory (next item). - Choosing a persistent state directory. To recover launches after a crash or a new pod, set
server.broker.state_dir(orSCION_SERVER_RUNTIMEBROKER_STATEDIR) to a directory that survives the restart: on Kubernetes, a volume of your choice mounted into the broker container, for example aPersistentVolumeClaimper replica. The chart does not add one; that is an operator decision. The directory must be on a local filesystem, not NFS or another shared filesystem (see above), and each replica that shares a broker ID needs its own. Keep the path the same across restarts. Changing it starts the broker with an empty state directory: launches journalled under the old path are not recovered, andscion runtime-broker deregister --purge-localremoves the directory under the current setting only. It readsserver.broker.state_dirfrom the global settings and environment of the shell that runs it, so run it where the broker’s setting applies (on Kubernetes, inside the broker container). If it cannot resolve the setting, it removes nothing and exits non-zero. - Grace period. The Hub chart sets
terminationGracePeriodSeconds: 45on its combined Hub and broker pod (hub.terminationGracePeriodSeconds), so the drain and the rest of the shutdown fit before the pod is killed. For broker pods you deploy another way, set 45 yourself: the Kubernetes default of 30 seconds leaves little margin.
Slow pod starts. On clusters with slow node provisioning, for example GKE Autopilot cold starts where a new node is added and the agent image takes several minutes to pull, raise launch_timeout so the launch is not ended first:
server: hub: launch_timeout: "15m"launch_timeout and launch_keepalive_seconds are read only at Hub startup; restart the Hub after changing them. They cannot be set through the admin server-config API (see Layer 0).
Retired setting. The server.hub.async_agent_launch setting (env SCION_SERVER_HUB_ASYNCAGENTLAUNCH) has been retired and is ignored. A Hub that still has it set logs this warning at startup: “server.hub.async_agent_launch is retired and ignored; use the hub.async_agent_launch experiment (on by default)”. Remove it. The experiment is on by default, so asynchronous create stays on unless an admin turns the experiment off. An admin server-config save (PUT /api/v1/admin/server-config) that still carries the key is refused with 422 and the same message.
Brokers on a Cloud Run service. An asynchronous launch keeps running in the Runtime Broker after the create response. A broker hosted on a Cloud Run service therefore needs CPU allocated outside requests: deploy the service with --no-cpu-throttling, and keep its minimum instances above zero.
- Symptoms. If the service allocates CPU only during requests (the Cloud Run default), the launch stalls after the create response. The broker stops reporting, so the Hub ends the launch and sets the agent to
errorwith launch errorbroker_lost(orlaunch_timeout). With minimum instances at zero, idle scale-in can also end a broker mid-launch, with the same result. - Startup warning. The server cannot see the service’s CPU setting. When the Runtime Broker is enabled and the process runs on a Cloud Run service (
K_SERVICEset,CLOUD_RUN_INSTANCEunset), it logs this warning once at startup: “If this Cloud Run service allocates CPU only during requests, async agent launches will stall after the create response and be reaped. Deploy with –no-cpu-throttling and set SCION_CLOUDRUN_CPU_ALWAYS_ALLOCATED=true to silence this.” - Silencing it. Set the environment variable
SCION_CLOUDRUN_CPU_ALWAYS_ALLOCATED=trueon a service deployed with--no-cpu-throttling. It is an assertion, not a check: remove it if you redeploy with CPU throttling. The Cloud Run deploy script (scripts/cloudrun/deploy.sh) sets both.
Brokers using the cloudrun-sandbox runtime. On the single-node Cloud Run Instance tier, the Runtime Broker runs on the Cloud Run Instance and starts each agent as a Cloud Run Sandbox. With the hub.async_agent_launch experiment on, eligible creates that opt in are asynchronous:
- Launch timeout. Set
launch_timeoutto at least"60s". A sandbox start may not finish within a shorter budget, and the launch is then ended withlaunch_timeout. - CPU after the create response (not verified). Async on this runtime assumes the Cloud Run Instance keeps CPU allocated while the broker finishes the launch in the background after the create response. This has not been verified on a live Instance. If the Instance does not keep CPU allocated, launches stall after the create response and the Hub ends them with
broker_lostorlaunch_timeout; the synchronous create (the experiment off) is not affected. To return to it, turn the experiment off; no restart is needed. The startup warning above covers Cloud Run services only and is not logged on an Instance.
Runtime Brokers using the cloudrun runtime. The cloudrun runtime starts each agent as a Cloud Run Instance. Each Instance declares container port 18381, where sciontool init opens a startup listener, and its startup probe checks that port (every 10 seconds, up to 24 failures) instead of the platform default port 8080. With the hub.async_agent_launch experiment on, eligible creates that opt in are asynchronous, as on other runtimes. If a launch fails or is ended, the Runtime Broker deletes the Instance the launch created, including one whose create request was cut off but which the platform created anyway.
- Restarts. The runtime does not set a restart policy on the Instance, so the platform’s default restart behavior applies: an agent that exits with code 0 may not be restarted. Start it again with
scion start. - Launch timeout. Set
launch_timeoutto at least"120s"; the default ("5m") is above this. The figure comes from measurements on the live Instances API: an Instance whose image was already cached was ready about 8 seconds after the create was accepted (worst of the cached runs), and the single start measured with an image not yet pulled took about 46 seconds. The broker waits for the create to finish and stops the launch 20 seconds before its deadline, so an uncached start needs a deadline of about 66 seconds. Allowing about 54 seconds more, close to double, for the work the broker does before the create and for larger images and slower pulls than the single uncached start measured, gives 120 seconds. Raise it further for large agent images. - Resources. The agent’s resolved
resourcesset the Instance container’s CPU and memory limits. Cloud Run has limits only, so each takeslimits, orrequestswhen no limit is set; with neither, the platform default applies (1 vCPU, 512Mi). Unlessruntime.enforce_resource_defaultsis false,limits.cpualways has a value, because the built-in defaultlimits.cpu: "2"applies here as on other runtimes, sorequests.cputakes effect only when it is above 2 (it then raises the limit to match); with the defaults off,requests.cpuapplies whenlimits.cpuis unset. CPU must be 1, 2, 4, 6 or 8 vCPU, memory 512Mi to 32Gi in wholeMiorM, and memory must fall in the range the vCPU count allows (1 vCPU up to 4Gi; 2 up to 8Gi; 4 from 2Gi to 16Gi; 6 from 4Gi to 24Gi; 8 from 4Gi to 32Gi). A resource left unset is checked at its platform default, solimits.cpu: "4"also needslimits.memoryof at least2Gi. Values outside these limits fail before anything is provisioned, with a message naming the field, the quoted value and the platform rule: a synchronous create, start or restart answers HTTP 400 (validation_error), and an asynchronous launch ends with launch errorcontainer_config. Errors name the CPU limit as the resolvedresources.limits.cpu, which the built-in default or a largerrequests.cpumay have set. Cloud Run Instances keep CPU allocated for their whole lifetime; the runtime does not set the request-basedcpu_idleoption.resources.diskis not applied; the Runtime Broker logs the effective CPU and memory, and any unapplied disk, when it creates the Instance. - Clone-per-agent. A clone-per-agent agent gets its own workspace on the NFS share,
<subpath_root>/<project-id>/agents/<agent>/workspace, next to its home, and the Instance mounts it at/workspace. The project’s shared workspace is neither cloned nor mounted for it.sciontool initclones the repository into the empty workspace inside the container on the agent’s first start; a later start, resume or restart finds it populated and keeps the agent’s work (it only refreshes the git credential helper). Deleting the agent removes its workspace. An agent created before this layout keeps working in the project’s shared checkout, in shared-plain mode, until it is recreated. The workspace is on the share, but everything else the agent writes, to the container’s root filesystem or/tmp, is held in memory and counts against the Instance’s memory limit. Empty-per-agent is still not supported on this runtime. Moving a clone-per-agent agent to another Runtime Broker with its workspace (anexportmove) is refused on every runtime other than Kubernetes, by theworkspace_on_exportmove check. - Project shared directories. Each project shared directory is mounted into the Instance as an inline NFS volume from
<subpath_root>/<project-id>/shared-dirs/<name>on the workspace share, at/scion-volumes/<name>(or<workspace>/.scion-volumes/<name>whenin_workspaceis set, honoringread_only). The Runtime Broker must have the share mounted undernfs.mount_rootso it can create and harden each shared-directory leaf before the Instance starts; starts with shared directories on a non-NFS backend, on the deprecated splitserver.shared_dir_storageexport, or without a broker mount of the share fail closed before any Instance is created. After upgrading to a version with shared-directory mounts, an existing Instance on NFS project storage whose project has shared directories is recreated once on its next start because its volume specification changes; its NFS-backed workspace and home survive the recreation.
Asynchronous agent start
Section titled “Asynchronous agent start”Asynchronous start, resume and restart of an existing agent is the hub.async_agent_start experiment, on by default. An admin can turn it off, or back on, in Admin → Server Config → Experiments; no restart is needed. When it is on, the Hub answers a start that opted in as soon as the Runtime Broker accepts it, as it does for an asynchronous create. The answer shows the agent in phase starting with an active launch (kind start, or restart for a restart). The broker then reports the outcome, and the agent moves to running, or to its failure phase with launch.error set. With the experiment off, every start, resume and restart is synchronous, as before.
- Needs both experiments. It applies only while
hub.async_agent_launchis on too. - Who opts in. The
scion startandscion resumecommands ask for it, so with the default they are asynchronous wherever the Runtime Broker supports it, and they wait for the launch to finish. Lifecycle API callers opt in with the body{"acceptAsyncLaunch":true}onPOST /api/v1/agents/{id}/startor/restart. Messages that wake an agent (--wake),scion reincarnateand starts the Hub schedules itself stay synchronous. - Which starts. A start of an agent at rest (
stopped,suspended,haltedorerror, including anerroragent that remembers an earlier phase; see resume memory below), and the start leg of a restart. The restart’s stop leg stays synchronous. The first start of acreatedorprovisioningagent stays synchronous, and so does a start of an agent that is already running. The Runtime Broker must support it too (it advertisesasyncStart). An older broker answers synchronously, and the start then completes as before.launch_timeoutmust be more than 20 seconds. - Every Hub node must run a version that supports it. Each such node advertises
async_start_v1in its hub-instance registry row (the one the health summary lists). The Hub checks this before every asynchronous start: while any live Hub instance lacksasync_start_v1, starts stay synchronous, so a rolling upgrade from a version that writes the hub-instance registry stays synchronous until the last node is upgraded. The check sees only nodes that write the registry. A node on a version from before the hub-instance registry writes no row and is not counted. In the health summary’s Hub instances table such a node does not appear, and when it is the node serving the page, the table is hidden; if no node writes the registry (every node is on such a version), the table is hidden whichever node serves the page. When you upgrade a multi-node Hub from such a version, turnhub.async_agent_startoff (an admin override) before the upgrade and back on once every node runs a version that advertisesasync_start_v1. The Hub logsasync start disabled: hub instance <label> (version <version>) lacks async_start_v1, once per change of the answer or its reason. While the experiment is on, each Hub re-reads the registry in the background every 15 seconds, so the answer stays current however rarely starts arrive, and a start never waits for the read. While the experiment is off, the Hub does not read the registry for this check. A freshly started Hub process, or one on which the experiment was just turned on, runs starts synchronously until its first read, within about 15 seconds. If registry reads fail, or none has succeeded for a minute, starts run synchronously, and the Hub logs a failed read at most once every 5 minutes. A node whose registry row has not been written for 45 seconds (stale) or that stopped cleanly is not counted, so a node whose registry writes fail stops blocking after 45 seconds. - Failure phase. A failed asynchronous restart ends
stopped. A failed start fromstoppedorerrorreturns to that phase. A failed start fromsuspendedorhaltedendserrorwith resume memory (below), so a client that waits for a phase still sees the failure. A failed asynchronous restart never revokes the agent’s credential. - Rolling back. Turn the experiment off (an admin override; it stays off across restarts and upgrades), then wait at least
launch_timeoutplus 15 seconds before you downgrade any Hub node, so start launches already in flight end under the newer code. The all-nodes check does not cover launches that were already in flight when a node was downgraded. - Resume memory. When an asynchronous start from
suspendedorhaltedfails, the agent showserrorand remembers the phase it started from (launch.resumePhasein the API). A later plainscion startorscion resume, the web Start button, or a message with--wakethen behaves as it would have for that phase. Turning the experiment off does not clear this memory; any later phase other thanstartingorerrordoes. - Runtime Broker side. A Runtime Broker that can run a start in the background reports
asyncStartin its capabilities (registration, heartbeat and/api/v1/info). The Hub’s start must carry the launch fields and the agent’s Hub ID; otherwise, or on a runtime that cannot launch in the background (currentlysubstrate), the broker starts the agent synchronously as before. The broker keeps a background start in the launch journal like a create (a failed launch report also records the launch step it failed at), and a stop, delete or newer start of the agent ends it (if one arrives during admission or before handoff, the start answers409 start_cancelled). A failed or abandoned start removes the pods, secrets or containers it created but never the agent’s files; this holds for a restart too, which the broker records as a start. The Runtime Broker also includes the Hub-mintedrunIdon its launch reports: if a launch fails on the Runtime Broker after the agent was already observedrunning(running_observed), the Hub still applies the failed launch report (moving the agent toerroror its start-kind failure phase, recordinglaunch.errorandlaunch.step, settling the start claim, releasing quota, and answeringappliedso the Runtime Broker cleans up) whenrunIdmatches the agent’s current run, the agent is inrunningorerrorwith no live activity, no other start, delete or reincarnation claim holds it, and the error code is notbroker_restarted.
Agent lifecycle events
Section titled “Agent lifecycle events”Agent lifecycle events are the hub.lifecycle_events experiment, off by default. With it on, every authoritative agent transition (create, launch, start, stop, halt, delete, restore, and configuration and label changes) records one versioned agent.lifecycle event in the same database transaction that commits it. The Hub then delivers the event to the Cloud Logging log scion-analytics. An admin turns it on or off in Admin → Server Config → Experiments; no restart is needed.
- Requirements. Events are recorded only when the Hub ID is pinned (
server.hub.hub_id, or a single-process SQLite hub’s persisted ID file) and the server’s Cloud Logging is enabled (SCION_CLOUD_LOGGINGor the telemetry setting). A workstation server bound only to loopback, on SQLite with dev auth, writes the events to stdout instead. Otherwise the Hub records nothing and reports why. - Enable it on every replica together. A replica with the experiment off records nothing. The next recorded event for an affected agent is then preceded by an explicit gap marker (
resynced), so the loss stays visible. - Turning it off stops new events. Events already recorded are still delivered; turning it off discards nothing pending.
- Diagnostics are Hub metrics plus a server-log line on every change:
lifecycle_observer_state(enabled,disabled_experiment_off,disabled_unpinned_hub_id,disabled_no_sink),lifecycle_sink_kind(cloud_logging,stdout,none),lifecycle_drainer_state, and the delivery and loss counters (lifecycle_outbox_dropped,lifecycle_resyncs,lifecycle_build_failures,lifecycle_ambiguous_transitions,lifecycle_unconfirmed_starts,lifecycle_attribution_stale). - Bounded retention. Undelivered events wait in a database outbox for up to 72 hours or 100,000 rows. Anything trimmed is counted and logged with its event IDs.
- Run attribution. Each event names the launch it belongs to (
launch_id) only when the Hub can prove the pairing of that run with the agent’s generation. The Hub keeps that state in the lifecycle cursor (a schema migration adds its columns; it is additive, with no backfill). The counterlifecycle_attribution_stale(exported aslifecycle_attribution_stale_total) records cursors whose attribution could not be trusted and was reset to unattributed.- Absent
launch_id. On a lifecycle event, an absentlaunch_idmeans the run attribution is unknown or unverified: the producer could not prove which launch the transition belongs to. It does NOT mean zero usage, that no run existed, or a launch-to-generation mapping, and it does not open or close a matched usage interval. Consumers must not join usage to such an event by launch, must not pair launch-less events by adjacency, and must not infer a launch from neighbouring events. On a closure withoutlaunch_id,incarnation_idandsegment_metadata_revisiondescribe an unattributed segment; they also must not be used to infer the launch. The transition itself is real and counts in coverage accounting. Usage that cannot be attributed stays in totals and in coverage accounting as unmatched; it is never treated as zero and never dropped. - Closures after a started. A closing event without
launch_idcan follow an earlierstartedthat named a run. The producer then could not prove the pairing at close time, for example after a run-mismatch delete. This does not retract the earlier event, and it is not zero usage. Do not infer the run from the earlier event or fromincarnation_idandsegment_metadata_revision. The matched interval is simply not available. - Malformed segments. A segment whose stored identity is malformed (only reachable through corruption or an unsupported writer) is discarded without a closing event. The stream marks the loss with one
resyncedevent (cause=unobserved_change,gap=true, nolaunch_id), and thelifecycle_attribution_stale_totaldiagnostic and a log line record it too. Thatresyncedsignals incomplete transition history. It is not a replacement terminal, a proven run mapping, a matched interval or evidence of zero usage. - Unknown history. Agents that existed before attribution tracking was enabled, whose tracking state was lost, or that went through many delete-and-restore cycles keep an “unknown history” mode. For them, a failed start that reverts to the previous run leaves the following events without
launch_iduntil the next qualifying fresh mint observed by the hub, which restores normal attribution for the newly minted run. A successful start alone is not that trigger, and it is not evidence that observation was continuous. Do not delete agents to clear this mode. - Continuity. Run attribution in lifecycle events assumes the lifecycle observer ran continuously, on compatible hub builds, since a known baseline. After the observer was off, after a downgrade, or with mixed hub versions writing agents, attribution for the affected agents is not guaranteed for that period: changes that returned to the same run while unobserved cannot be detected. Treat such periods as unverified. A
resyncedevent with causeno_cursormarks an agent whose earlier history the Hub never observed; its existing run stays unattributed until the next qualifying fresh mint.
- Absent
Request performance tracing
Section titled “Request performance tracing”perf_trace: true (env SCION_SERVER_HUB_PERFTRACE=true) records where Hub API requests and SSE connections spend their time, and how many authorization store reads and decision-audit records each request causes. It is for diagnosing slow agent lists, mainly on a test or staging Hub. On a production Hub, turn it on only for a short diagnosis window: the log volume grows by one line per request. It is off by default. For a how-to (reading the log lines, running the perf/bench harness, and using the counts as regression budgets), see Hub Performance Tracing and Benchmarks.
server: hub: perf_trace: trueObserve only. Tracing never changes an authorization decision, a decision-audit record (content, count or delivery; records are delivered asynchronously and can be lost, see Log delivery guarantees), a response body, filtering, sorting, redaction or lineage. The only change a client can see is the extra response headers described below, and only an unscoped local platform admin who asks for them can see it.
What it records. Each Hub API request writes one perf_trace log line (subsystem hub.perf-trace); an SSE connection writes one at connect and one at close (sse_stage). A line holds:
endpoint: a fixed endpoint class, for exampleagents.global.legacy,agents.global.sorted,agents.project.legacy,agents.project.sorted,agents.project.sorted_agent,sse.events, orother.phase_<name>_usandphase_<name>_n: time (microseconds) and count per phase. Phases arelist_scope_authz(list-level authorization before rows are read),list_db_read(agent row and member reads; database time only),list_read_authz(per-row read decisions),enrich,capabilities(per-item capabilities and the env-view decision),messageability,scope_capabilities,serialize(encoding and writing the body), and for SSEsse_expand,sse_authorizeandsse_write.store_<method>_nandstore_<method>_us,authz_store_calls,authz_store_us: reads the authorization service makes to prepare its inputs (groups, role bindings, role definitions, access constraints, delegation edges, and user, agent, project and membership rows), counted after request-local reuse.audit_records,audit_allow,audit_deny,audit_other,audit_emit_us: decision records counted in memory. With the default sampling setting, each authorization decision emits one record, soaudit_recordsis the request’s decision count.audit_emit_usmeasures the emitter call. This path performs no database write.db_wait_count,db_wait_us,db_in_use,db_open: connection-pool waits during the request and pool use when the line is written (the end of the request), when the database driver reports them. In the response headers they are taken when the response starts. The pool is shared, so waits include concurrent requests.elapsed_us,method,request_id(for correlation only), and for SSEsse_events.
Counted scope. The store counts cover the authorization service’s request-path reads of the methods above. Relationship progeny lookups and reads inside a store transaction are not counted, and work handed to a detached background context after the request ends records into a trace that is no longer logged. capabilities also includes the rare re-list read decision for a sorted-list row whose authorization inputs changed between reads.
Response headers. Only an unscoped local platform admin (a local user with the admin role, not using a scoped access token, not federated) who sends X-Scion-Perf-Trace: 1 also gets X-Scion-Perf-Endpoint, X-Scion-Perf-Phases, X-Scion-Perf-Phase-Counts, X-Scion-Perf-Store-Calls, X-Scion-Perf-Store-Us, X-Scion-Perf-Decisions and, when available, X-Scion-Perf-DB. Values are name=integer pairs; durations are microseconds. The headers are set when the response starts, so they omit serialize. No other caller gets them: not unauthenticated endpoints, agents, brokers, scoped tokens, federated identities or non-admin users. Their requests are still traced and logged. The perf/bench API benchmark records the headers with --want-perf-trace, and for its non-admin caller joins the hub’s perf_trace log lines by request ID with --hub-perf-log. Counts (store calls, decisions, phase counts) do not depend on machine speed, so they suit CI budgets; durations do not.
What the counts reveal. The counts are not just timing. A deny count is the number of candidate rows hidden from the caller (on a filtered list, whether one specific agent exists). Store-call counts hint at owner, delegator and group structure behind the rows. DB pool figures describe process-wide load. That is why the headers go only to unscoped local platform admins, and why the log line belongs with other operator logs.
Cardinality and privacy. Phase names, store methods, endpoint classes and audit outcomes are fixed sets. No path, query, ID, name, email, token, secret or configuration value is logged or used as a label.
Overhead. With tracing off, no middleware or decorator is installed, and each recording point in the list handlers is one context lookup that finds nothing. With tracing on, each recorded phase or store read adds two clock reads and an uncontended lock, plus a few hundred bytes and one log line per request.
Relay Settings (server.relay)
Section titled “Relay Settings (server.relay)”Settings of a separate relay process, scion server --role relay (external relay mode, server.hub.conduit.relay_mode: external). Settings the relay shares with the hub’s relay, such as internal_listen, peer_auth, peer_secret and lifetime_cap, stay under server.hub.conduit. All of them are Layer 0 (bootstrap) and read at startup, so a change needs a restart.
A separate relay requires targets (agents and Runtime Brokers) on version X or later. Until then, and until its other production gates are met, the separate relay runs on development and test hubs only.
| Field | Type | Default | Description |
|---|---|---|---|
hub_internal_url |
string | Base URL (http(s)://host:port) of the hub internal API the relay calls for admission, RPCs and re-checks. Any hub replica serves it, so it may be a load-balanced internal address; it must be reachable only inside the cluster or VPC. It must be https:// unless its host is loopback (a loopback IP literal: 127.0.0.0/8 or ::1, not localhost; see TLS on the internal hop in the conduit settings above). Layer 0; restart required. Env: SCION_SERVER_RELAY_HUBINTERNALURL (or SCION_SERVER_RELAY_HUB_INTERNAL_URL). |
|
public_url |
string | The relay’s own public URL, recorded as its relay_instances.public_endpoint. A user authenticated by Google IAP is admitted through a relay only with the hub’s own IAP configuration (auth.proxy.iap with its audience, Google’s issuer and key endpoint), so clients must reach the hub’s IAP backend; trusted-proxy identity headers are never accepted from a relay. Layer 0; restart required. Env: SCION_SERVER_RELAY_PUBLICURL (or SCION_SERVER_RELAY_PUBLIC_URL). |
|
singleton |
bool | false |
Experimental. Asserts a non-HA singleton relay. Required on Cloud Run (K_SERVICE set), where the relay refuses to start without it; set it together with max_instances=1, min_instances=1 and cpu_idle=false. Allowed elsewhere, where it labels the relay non-HA. The relay lists the feature singleton in its relay_instances row, retries its startup self-check for up to 60s, and reports other live relays sharing its internal endpoint in the gauge scion.relay.singleton_endpoint_peers. Upgrade the hubs, which migrate the database, before the relays. Layer 0; restart required. Env: SCION_SERVER_RELAY_SINGLETON. |
Relay database access. A relay reads its own configuration file, never the hub’s, and connects with the database.url in it. Give the relay a relay-scoped, least-privilege database role with exactly these rights: read and write on relay_instances and conduit_sessions; read, insert and update on conduit_principal_epochs (no delete); and read on the hub settings table, of which it reads only the experiments row to follow hub.conduit. On PostgreSQL the relay LISTENs on the hub’s dedicated scion_ev_conduit channel, never on the hub’s other event channels. The hub notifies that channel, in the same transaction as its own notice, of each revocation and settings change a relay acts on, with the subject and identifiers only. Grant the role table rights only (PostgreSQL does not restrict LISTEN by role). It needs nothing else, and never reads token, credential or secret tables. At startup the relay logs the configuration files it loaded and its database driver, never the connection string or its credentials.
Broker Settings (server.broker)
Section titled “Broker Settings (server.broker)”Controls the Runtime Broker service.
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Whether to start the broker service. |
port |
int | 9800 |
HTTP port to listen on. |
write_timeout |
duration | "210s" |
HTTP write timeout of the broker’s server. Keep it above the Hub’s fixed 180-second request timeout to the broker (the default leaves 30 seconds of margin); a lower value can cut off an answer the Hub is still waiting for. Env: SCION_SERVER_RUNTIMEBROKER_WRITETIMEOUT. |
broker_id |
string | Unique UUID for this Runtime Broker. It names the Runtime Broker’s state directory (<state_dir>/<broker-id>), so it must be a single path element: an empty ID, ., or an ID containing .., /, \ or NUL is refused on join and on credential save. A value written by hand (for example with scion config set) is caught when the server starts, which refuses to start with it. |
|
broker_name |
string | Name the broker registers under on the Hub. Default: the hostname (Hosted Broker for a broker run with a co-located Hub). |
|
broker_nickname |
string | Same as broker_name, and takes precedence over it. Set by scion runtime-broker register --broker-name. |
|
hub_endpoint |
string | The Hub URL this broker connects to. | |
container_hub_endpoint |
string | The Hub URL this broker gives its Docker and Podman agents in place of the agent Hub endpoint, whatever that endpoint is. Use it when this broker’s containers reach the Hub at another address than remote brokers do, for example http://scion-hub.internal:8080 for Docker agents on a broker colocated with the Hub, an internal https:// address on a broker in the Hub’s private network, or http://host.containers.internal:8080 for local development. A plaintext http:// value must be a host alias of the agent’s runtime (see Hub Endpoint Resolution); any other address needs https://. Kubernetes, Cloud Run and Apple container agents keep the agent Hub endpoint (Apple container agents still get this value in place of a localhost endpoint). |
|
state_dir |
string | ~/.scion/runtime-broker-state |
Base directory of the broker’s runtime state: the launch journal, dispatch attempts and pending env-gather. The broker uses <state_dir>/<broker-id>. Must be an absolute path or start with ~/; a relative path refuses startup. Read at startup. Flat Runtime Broker instances (instances) do not use it and keep their default directories; setting it on a process that hosts them logs a warning. |
broker_token |
string | Authentication token for the Hub. | |
auto_provide |
bool | false |
Automatically add as provider for new projects. |
allow_container_script_harnesses |
bool | true |
Allow the Runtime Broker to run container-script harnesses (provision.py) and stage fallback capture_auth.py assets. When false, dispatches whose harness-config uses scripted provisioning (or whose staged hook wrapper cannot be evaluated cleanly) fail closed with 403, and fallback capture_auth.py is not staged. Env: SCION_SERVER_RUNTIMEBROKER_ALLOWCONTAINERSCRIPTHARNESSES. |
cors |
object | Broker CORS configuration; off by default (see Broker CORS). | |
instances |
list | Experimental. Flat Runtime Broker instances this process hosts; see Flat Runtime Brokers. Empty or absent means the regular Runtime Broker. Several entries are accepted, each with a unique key; the Hub may run in the same process or remotely (see the link). Read only from the global settings.yaml (not server.yaml or project settings). Each entry has key (immutable local key: lower-case letters, digits and -, starting and ending with a letter or digit, up to 63 characters), name (the name registered with the Hub), and runtime_target (see Flat instance runtime target). An entry may also set agent_defaults, the workload defaults of the agents it runs (see below). |
Single-process requirement for co-located flat instances. When server.broker.instances runs co-located with the Hub in the same scion server process, each flat Runtime Broker instance ID must be hosted by at most one server process at a time. Every co-located start replaces that instance’s stored HMAC secret in the Hub database, and startup refuses to activate a co-located flat instance (instance_in_use) when another live server process already holds its local lock or has a fresh online heartbeat for that instance ID. Multi-replica Hub deployments sharing a database must run flat Runtime Broker instances on separate remote Runtime Broker hosts rather than co-locating the same server.broker.instances across Hub replicas.
Broker CORS (server.broker.cors)
Section titled “Broker CORS (server.broker.cors)”Nothing in Scion calls the Runtime Broker from a browser: the Hub reaches it with signed server-side requests or over the control channel. CORS on the broker is therefore off by default, and the broker sends no CORS headers.
CORS headers are added only to requests that pass broker authentication. Broker authentication runs first, and preflight (OPTIONS) requests are not exempt from it, so a broker with HMAC keys refuses an unsigned preflight before CORS applies. Allowing an origin therefore helps only a caller that can sign its requests.
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable CORS on the broker API. A missing server.broker.cors block means off. |
allowed_origins |
list | [] |
Origins that get CORS headers on authenticated broker requests. Enabling CORS without listing origins allows none. "*" allows every origin; the broker logs a warning at startup when it is set. |
allowed_methods |
list | GET, POST, PUT, PATCH, DELETE, OPTIONS |
Methods sent in Access-Control-Allow-Methods for an allowed origin. |
allowed_headers |
list | Authorization, Content-Type, X-Scion-Broker-Token, X-API-Key |
Headers sent in Access-Control-Allow-Headers for an allowed origin. |
max_age |
int | 3600 |
Preflight cache lifetime in seconds. |
Env: SCION_SERVER_RUNTIMEBROKER_CORS_ENABLED, SCION_SERVER_RUNTIMEBROKER_CORS_ALLOWEDORIGINS (comma-separated), and the matching names for methods, headers and max age.
Flat instance runtime target (server.broker.instances[].runtime_target)
Section titled “Flat instance runtime target (server.broker.instances[].runtime_target)”The runtime target of one flat Runtime Broker instance, and the instance’s own infrastructure defaults. A flat instance never uses Runtime Broker Profiles, so these fields take the place that a profile and its runtime entry hold for a regular Runtime Broker. Each default keeps its usual precedence: an agent’s own value still wins over it, and an unset field falls back to the same global setting as before. All fields are local operator policy and are read only from the global settings.yaml; only the publishable subset (type, display_name, Cloud Run region, and the Kubernetes namespace, defaults, and placement fields) is reported in runtimeTargetPresentation, while kubeconfig paths and contexts, service-account policy, and Cloud Run project_id, service_account, network, subnetwork, and credentials_file never leave the host. Unknown keys and invalid values refuse startup with an error that names the settings path. A field for another target type (for example a Kubernetes or Cloud Run field on a docker instance) is refused with field not valid for runtime target type <type>.
| Field | Type | Applies to | Description |
|---|---|---|---|
type |
string | all | docker, kubernetes, or cloudrun (experimental: requires hub.flat_cloudrun_target and a co-located Hub in the same process; on a remote host without a co-located Hub, cloudrun instances are not activated). Required. |
display_name |
string | all | Non-sensitive name shown in the UI. Default: the type. |
context |
string | kubernetes | Context in the loaded kubeconfig. Default: its current context. |
namespace |
string | kubernetes | Namespace for the instance’s agents. Default: SCION_K8S_NAMESPACE, POD_NAMESPACE, the service account’s namespace, else default. |
kubeconfig |
string | kubernetes | Absolute path of the instance’s own kubeconfig file, then its only configuration and credential source. Default: the process’s normal kubeconfig loading. |
kubernetes_block_service_account |
string | kubernetes | Kubernetes ServiceAccount for pods whose GCP identity mode resolves to block. |
kubernetes_service_account_mappings |
map | kubernetes | GCP service account email to the Kubernetes ServiceAccount bound to it through Workload Identity, for GCP identity mode assign. |
shared_dir_storage_class |
string | kubernetes | Default storage class for shared directory PVCs (must support ReadWriteMany). An agent’s own kubernetes.shared_dir_storage_class wins. |
shared_dir_size |
string | kubernetes | Default size of each shared directory PVC, a positive Kubernetes quantity such as 40Gi. Default: 10Gi. An agent’s own kubernetes.shared_dir_size wins. |
safe_to_evict |
bool | kubernetes | false adds the cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation to agent pods (on GKE Autopilot this requests extended run duration). true or unset adds nothing. An agent’s own kubernetes.safeToEvict, true or false, wins. |
priority_class_name |
string | kubernetes | Default PriorityClass of agent pods (DNS-1123 subdomain). The PriorityClass must already exist on the cluster. An agent’s own kubernetes.priorityClassName wins. |
gke |
bool | kubernetes | true turns on GKE mode for the instance. false or unset detects a GKE cluster automatically. |
node_selector |
map | kubernetes | Node labels (key to value) every agent pod of the instance must match. See Flat instance placement. |
tolerations |
list | kubernetes | Tolerations of every agent pod of the instance, each with key, operator (Equal or Exists; empty means Equal), value and effect (NoSchedule, PreferNoSchedule, NoExecute, or empty for any). An empty key requires Exists, and Exists requires an empty value. |
runtime_class_name |
string | kubernetes | RuntimeClass of every agent pod of the instance (DNS-1123 subdomain). The RuntimeClass must already exist on the cluster. |
project_id |
string | cloudrun | GCP project ID (6–30 characters: lowercase letters, digits, hyphens) hosting the Cloud Run instances. Required for cloudrun. Execution scope identity resolves the project’s numeric project number (cloudrun:<projectNumber>/<region>). Never published in runtimeTargetPresentation. |
region |
string | cloudrun | GCP region (such as us-central1). Required for cloudrun. Published as-is in runtimeTargetPresentation. |
service_account |
string | cloudrun | Dedicated user-managed IAM service account (<name>@<project>.iam.gserviceaccount.com) that Cloud Run agent instances run as. Required for cloudrun; project default compute (<number>-compute@developer.gserviceaccount.com) and App Engine (<project_id>@appspot.gserviceaccount.com) service accounts and Google-managed service agents (service-<number>@..., @gcp-sa-...) are refused. Never published in runtimeTargetPresentation. |
network |
string | cloudrun | Direct VPC egress network (resource name or projects/<project>/global/networks/<name>). At least one of network or subnetwork is required for cloudrun so instances reach the NFS workspace share. Never published in runtimeTargetPresentation. |
subnetwork |
string | cloudrun | Direct VPC egress subnetwork (resource name or projects/<project>/regions/<region>/subnetworks/<name>). At least one of network or subnetwork is required for cloudrun. Never published in runtimeTargetPresentation. |
credentials_file |
string | cloudrun | Clean absolute path (not /, no ~, environment variables, path list, or relative segments) to this instance’s own GCP credentials file. Required for cloudrun. Checked at startup (must be a regular file, not a symlink or directory, and not group- or world-writable), and used as the sole credential source for both Cloud Resource Manager project-number resolution (cloudrun:<projectNumber>/<region>) and the Cloud Run runtime (ambient ADC is never used). Never published in runtimeTargetPresentation. |
When any server.broker.instances entry has type: cloudrun, server.project_storage must set both backend: nfs (with at least one share in nfs.shares) and homes: nfs. Cloud Run rewrites label keys to GCP’s label rules (sanitizeGCPLabelKey, turning dots into underscores), so ownership checks on a cloudrun instance read scion.runtime_broker_id and the identity labels (scion.project_id, agent_id, scion.name) under their GCP-sanitized label keys (runtime.LabelKey), while checkOwnerLabel refuses a launch whose scion.runtime_broker_id, scion.project_id, agent_id, or scion.name label value would be rewritten by sanitizeGCPLabelValue and ownership checks require exact value equality.
A runtime target does not choose where its agents’ storage lives: server.project_storage does, for every instance of the process. The former shared_dir_storage_backend, shared_dir_storage_backends, home_storage_backend and home_storage_leaf fields are removed. A flat instance that sets them refuses startup with an error that names the field, server.project_storage and volumes:. On runtimes and profiles, startup reports them and agent start refuses them; see Removed storage overrides. For a node-local directory, use a volumes: entry.
Flat instance placement
Section titled “Flat instance placement”node_selector, tolerations and runtime_class_name place the instance’s agent pods on the nodes the operator chooses, whatever template the agent uses, including default. Unlike the other defaults in the table, the operator owns them, so an agent’s own value does not win:
- An agent’s
kubernetes.nodeSelectormay add keys the runtime target does not set. It cannot change the value of a key the runtime target sets, including the pool keyscion.dev/pool. A differing value refuses the start. - Each of an agent’s
kubernetes.tolerationsmust match one of the runtime target’s tolerations exactly: the same key, operator (empty counts asEqual), value and effect. A broaderExiststoleration does not cover a different entry. When the runtime target sets no tolerations, any agent toleration is refused, including one with an empty key andExists. While placement is set, an agent toleration with an empty key andExistsis refused even when the runtime target has the same entry. - An agent cannot choose or change the runtime class. An agent
kubernetes.runtimeClassNamemust be empty or equal to the runtime target’s value.
A refused start is reported as a validation error that names the field and whether it comes from the request or the agent’s stored configuration. The check runs before the start creates, changes or stops anything. Node selector entries combine: an agent-added key narrows placement further, and a key with no matching nodes leaves the pod pending. An agent-added key can also steer a pod onto any pool whose nodes carry no taint, because only the runtime target’s keys are fixed. Tolerations stay limited to the runtime target’s. A node selector and a toleration are independent: Scion never derives one from the other, so a tainted pool needs both.
The values are read from the instance’s settings when it starts, so a change takes effect when the Runtime Broker restarts. Existing pods keep their placement. The Brokers page shows the instance’s reported placement read-only, as defaults for new agents. In GCP identity block mode the Workload Identity node label is still added after this placement.
server: broker: instances: - key: agents name: agents runtime_target: type: kubernetes namespace: agents node_selector: scion.dev/pool: agents tolerations: - key: scion.dev/pool operator: Equal value: agents effect: NoSchedule runtime_class_name: gvisorgvisor is an example: Scion does not create RuntimeClasses or check that one exists.
At agent launch (create, start, and restart), both flat Kubernetes instances (server.broker.instances[].runtime_target) and regular (legacy) Kubernetes Runtime Brokers (runtimes.<name> and profiles.<name>) check and enforce these rules before provisioning or stopping anything. On a regular Runtime Broker:
- Every Kubernetes runtime entry in
runtimesthat targets the same cluster (context) andnamespacemust agree on itsnode_selector,tolerationsandruntime_class_name; a disagreement refuses the launch (and is rejected when savingruntimesin admin settings). - A profile’s placement layers over its runtime entry’s placement:
node_selectormerges per key with the profile winning (exceptscion.dev/pool, which a profile may repeat with the same value or omit when the runtime sets it, but cannot change),tolerationsmerge by exact tuple (key,operatorwith empty normalized toEqual,value,effect) preserving order, andruntime_class_namefrom the profile wins when set. Even when neither the runtime entry nor the profile sets any placement field, the resulting empty policy is operator-owned, so an agent’s own tolerations orruntimeClassNameare still refused unless the operator policy includes them. - On a restart, if the Hub does not supply the project path on a legacy Kubernetes Runtime Broker (for example a call from an older Hub or for an orphaned agent) and the project settings therefore cannot be loaded, the restart is refused before the stop leg runs so the running pod is not torn down.
Saving or resetting runtimes or profiles in admin settings checks the Kubernetes placement rules across both sections together; see Legacy host placement check.
Before upgrading
Section titled “Before upgrading”An agent that sets its own tolerations or runtime class while the instance’s runtime target (or, on a regular Runtime Broker, the resolved runtime entry and profile) sets none is refused at its next start or restart. Running agents keep their pods until then. Before upgrading a host with flat instances or a regular Kubernetes Runtime Broker:
- Run
scion doctorwith the new release. On a host with flat instances, theflat-placement-preflightcheck scans the stored configuration of every agent each flat instance owns; on a host without flat instances,legacy-placement-preflightscans the host’s settings and agents (see Legacy host placement check). Neither check changes anything, and both report counts only: agents with tolerations while the operator policy sets none (agent_tolerations_without_operator_policy), agents with a runtime class while the operator policy sets none (agent_runtime_class_without_operator_policy), and agents they could not check (incomplete). They show no agent names or configuration values. An agent can be counted in both categories. - Treat
incompleteas unfinished: fix unreadable agent records or configurations, then run the check again. - For agent tolerations, decide whether each belongs to the instance or profile/runtime entry. If it does, add it to the runtime target’s (or runtime entry’s / profile’s)
tolerations; otherwise remove it from the agent’s configuration. Tolerations match by key, operator (empty counts asEqual), value and effect; a broadExiststoleration on the operator policy does not cover a different agent entry. A toleration added to a runtime target, runtime entry, or profile applies to every agent that uses it. - For an agent runtime class, either set
runtime_class_nameon the runtime target (or runtime entry / profile) or remove the agent’s class. An agent cannot choose a class on its own; repeating the operator policy’s class is accepted. - Restart the Runtime Broker (or save the updated
runtimes/profilesin admin settings) to activate the changes, then runscion doctoragain until every count andincompleteare zero.
The check always shows these two counts. It also counts agents the operator policy would refuse for another reason: a toleration that is not among the policy’s (agent_toleration_not_in_policy), a runtime class that differs from the policy’s (agent_runtime_class_differs), a node selector key the policy sets to another value (agent_pool_selector_conflict for scion.dev/pool, agent_selector_conflicts_operator otherwise), a toleration that matches all taints (agent_match_all_toleration) and an invalid placement field (agent_invalid_placement_field). These appear only when counted.
Legacy host placement check
Section titled “Legacy host placement check”On a host without flat instances, scion doctor reports a legacy-placement-preflight line. It changes nothing and shows counts only, never names or values. It covers this host’s global settings file and the agents in the global project, the hub-managed projects and the project you run it in.
- Settings: runtime entries and profiles whose placement the Hub refuses when they are saved:
settings_malformed_placement(a misspelt key such asnodeSelector, or a wrong shape),settings_invalid_placement_field,settings_placement_on_non_kubernetes,settings_profile_runtime_missing(a profile with placement whose runtime entry does not exist),settings_sibling_placement_disagreement(entries with the same context and namespace but different placement) andsettings_pool_selector_conflict(a profile that changesscion.dev/pool). The line names the sections to repair. - Agents: the same agent categories as the flat check, against the runtime entry and profile each agent uses. With no Kubernetes placement configured, agent tolerations and runtime classes count as
agent_tolerations_without_operator_policyandagent_runtime_class_without_operator_policy. - Incomplete coverage:
incompletecounts agents whose record, configuration, profile or runtime cannot be read or resolved here.not enumeratednames sources the check cannot list: linked projects other than the current one (runscion doctorinside each). Neither is a clean result.
Settings stored in the Hub are reported by placement_preflight in the server config view (see admin settings). Repair the named settings first, then the counted agents, and run the check again.
Flat instance agent defaults (server.broker.instances[].agent_defaults)
Section titled “Flat instance agent defaults (server.broker.instances[].agent_defaults)”An optional block of workload defaults for the agents a flat Runtime Broker instance runs. A flat instance does not use Runtime Broker profiles, so these take the place of the matching profile settings. Each value sits exactly where the profile value sat in its own precedence order: a template, the agent’s own configuration or the request still wins wherever it won over a profile, and an unset value falls through to the same lower settings. The block is read from settings.yaml only and is never sent to the Hub. Unknown keys are refused at startup, naming the settings path.
| Key | Type | Precedence |
|---|---|---|
default_harness_config |
string | Below the template’s and the agent’s stored harness config; above the top-level default_harness_config. |
image_registry |
string | Above the top-level image_registry of the settings file; below a saved Hub top-level value and the node’s SCION_IMAGE_REGISTRY (see Image registry order). |
volumes |
list of volume mounts | Appended after the harness config’s volumes. |
resources |
resource spec | The agent’s own resources win over it; it wins over the Hub agent_defaults.resources and default_resources. |
harness_overrides.<harness-config> |
image, image_pull_policy, user, auth_selected_type, env, volumes, resources |
Applied over harness_configs.<harness-config>, as a profile’s harness override is. An explicit image outranks the template and inline images (only a request image beats it). Its resources rank above the agent’s own. An empty env value in any override marks a key the user must supply. Validated like a profile’s harness override. |
secrets |
list of secret declarations (no values) | Required secrets, between the harness config’s secrets and the template’s. |
clone_depth |
"full" or an integer from 1 |
Below the template’s or agent’s clone_depth; unset keeps the default shallow clone. |
server: broker: instances: - key: docker-1 name: docker-1 runtime_target: type: docker agent_defaults: image_registry: registry.example.com/team resources: requests: memory: 2Gi harness_overrides: claude: auth_selected_type: vertex-aiRuntime Broker environment variables
Section titled “Runtime Broker environment variables”These environment variables are read by the Runtime Broker process when launching Docker or Podman agent containers:
| Variable | Default | Description |
|---|---|---|
SCION_FORCE_HOST_NETWORK |
unset | When set to any non-empty value, forces agent containers to run with --network=host whenever a Hub endpoint is configured. Without it, only Docker uses host networking, and only for a loopback Hub endpoint (localhost, 127.0.0.1, ::1, or host.docker.internal). |
SCION_HOST_NETWORK_BLOCK_POLICY |
auto |
Controls how the Runtime Broker handles an agent whose GCP identity metadata mode is block when the container runs with --network=host (where in-container iptables interception of 169.254.169.254 cannot be installed). auto (or unset) refuses agent start on a detected Linux cloud host (GCE, AWS EC2, Azure, Oracle Cloud, Alibaba Cloud, OpenStack, DigitalOcean, or Hetzner, detected from local DMI, cloud-init, or GCE guest-agent files) and allows it with a warning on other hosts; strict always refuses block under host networking; allow allows agent start even on a detected cloud host and logs a warning. Any unrecognized value is logged at ERROR and treated as strict. See Host networking on Docker and Podman. |
Database (server.database)
Section titled “Database (server.database)”Persistence settings for the Hub.
| Field | Type | Default | Description |
|---|---|---|---|
driver |
string | "sqlite" |
Database driver: sqlite or postgres. |
url |
string | "hub.db" |
Connection string or file path. |
max_open_conns |
integer | sqlite: 1; postgres: 5 |
Maximum number of open connections. For sqlite this is always 1, whatever is configured, to serialize writes. For postgres a value of 1 or less takes the default. Settings file only: no environment variable sets it. |
max_idle_conns |
integer | sqlite: 1; postgres: 2 |
Maximum number of idle connections in the pool. For postgres a value of 1 or less takes the default; for sqlite, a value of 0 or less. Settings file only: no environment variable sets it. |
conn_max_lifetime |
duration | sqlite: unlimited; postgres: "30m" |
Maximum time a connection may be reused (Go duration, for example "30m"). Empty takes the default; "0" means unlimited. Settings file only: no environment variable sets it. |
conn_max_idle_time |
duration | sqlite: no limit; postgres: "5m" |
Maximum time a connection may sit idle in the pool before it is closed (Go duration). Set it shorter than the server-side or proxy idle timeout so idle connections are recycled first. Empty takes the default; "0" means no idle limit. Settings file only: no environment variable sets it. |
allow_shadowed_schema |
bool | false |
PostgreSQL only. Lets the Hub start, and the maintenance commands that migrate its database run, when the schema migration would create a new table set that hides existing Hub tables in a later schema on the search_path. See PostgreSQL search_path check. Env: SCION_SERVER_DATABASE_ALLOWSHADOWEDSCHEMA. |
tx_diagnostics |
object | see below | Transaction-hold and lock-wait diagnostics. See Transaction diagnostics. Settings file only. |
Transaction diagnostics
Section titled “Transaction diagnostics”The Hub measures how long each database transaction and connection checkout holds its connection, and how long each request waits to get one. SQLite runs one writer at a time, and the Hub’s SQLite pool has a single connection, so one slow holder makes every other database request wait behind it. These diagnostics name the holder and the waiters.
Field (server.database.tx_diagnostics.*) |
Type | Default | Description |
|---|---|---|---|
enabled |
bool | true |
Install the diagnostics. false opens the database exactly as without them. |
hold_warn_threshold |
duration | "2s" |
Log a transaction or connection checkout held at least this long. "0" turns these lines off; metrics are still recorded. |
lock_wait_warn_threshold |
duration | "1s" |
Log a request that waited at least this long for a connection. "0" turns these lines off. |
warn_interval |
duration | "1m" |
At most one line per call site and kind per interval. Must be positive; "0" or a negative value fails startup. Lines dropped by this limit are counted in the next line’s suppressed. |
The lines are WARN records from the hub.db subsystem:
db_tx_long_hold:kind(tx, or on SQLiteconnfor a held connection with no transaction line),op(the calling function),site(the first three calling frames, extended past internal store frames when needed to name the caller),hold_ms,begin_ms,stmts,stmt_ms(time in SQL),commit_ms(time in the transaction’s COMMIT; on SQLite that includes writing and syncing the WAL, so a slow disk shows up here; on akind=connline,begin_msandcommit_msadd up the transactions run during that checkout),idle_ms(time the connection was held outside BEGIN, SQL and COMMIT, such as Go work inside a transaction),max_stmt_ms,rows_affected,outcome, and on SQLitewaiters_max(requests from the ent client that waited during the hold) andbusy.db_lock_wait:op,site,wait_ms,req(tx,execorquery) andoutcome(acquired,canceled,deadlineorerror). On SQLite it also names who held the connection during the wait:holders(how many connection checkouts overlapped the wait; many short ones mean a queue rather than one slow holder),top_holder_op,top_holder_siteandtop_holder_ms(the holder with the most time in the wait), and, when the waiter gave up,holder_open_op,holder_open_siteandholder_open_ms(the checkout still holding the connection at that moment).
A write made outside an explicit transaction commits inside its own statement, so its commit time is part of that statement’s stmt_ms and is not reported as commit_ms. Both carry suppressed and dropped (lines lost because the log queue was full). Per-waiter lines cover requests made through the ent client. Waits of the raw-SQL stores appear in aggregate, in the scion.db.pool.connections.wait_duration metric. Startup schema migrations run through the same path, so a long migration can log one or more lines at boot; they are expected and are not a stall.
When metrics export is on, the Hub also records the histograms scion.db.tx.hold.duration (labels op, kind, outcome, pool), scion.db.tx.lock_wait.duration and scion.db.tx.commit.duration (labels op, outcome, pool), the counter scion.db.pool.connections.wait_duration (ms), and scion.db.diag.warn.dropped. The op label is capped at 256 values; further call sites are reported as other.
PostgreSQL search_path check
Section titled “PostgreSQL search_path check”On PostgreSQL, the Hub’s schema migration looks up and creates its tables in current_schema(), the first existing schema on the connection’s search_path. If that schema has no Hub tables but a later schema on the search_path does, the migration would create a new, empty table set in the first schema. That set would hide the existing data, and the Hub would start up looking like a new install. This can happen with:
- a custom
search_path, such ascustom,public, wherecustomhas no Hub tables andpublicholds them; - the default
"$user", public, when a schema named after the database user exists and is empty.
So before migrating, the Hub server and every maintenance command that migrates the Hub database check the schemas on the search_path. A schema counts as holding Hub tables when it has both the agents and runtime_brokers tables. If the current schema has none and a later schema does, the Hub refuses to start (and the command refuses to migrate), and the error names both schemas. To fix it, change the connection’s search_path so the schema with the Hub tables comes first, or drop the current schema if nothing else uses it.
If you do want a new table set in the current schema, set server.database.allow_shadowed_schema: true (or SCION_SERVER_DATABASE_ALLOWSHADOWEDSCHEMA=true). The Hub, or the command, then logs a warning that names both schemas and continues.
A fresh database (no Hub tables in any schema on the search_path) and an existing install whose current schema holds the Hub tables start as before. SQLite is not affected. The same check and override apply to the maintenance commands that migrate the database: server migrate (on its PostgreSQL destination), server backfill and server migrate-dm-keys (including their default dry-run mode), server recover-authz, server test-infra mark and hub secret migrate-names (except with --dry-run, which does not migrate). These commands read the override from the server settings or the environment, as the Hub does. (hub secret migrate is disabled and opens no database; see Hub server: Migrations.)
Authentication (server.auth)
Section titled “Authentication (server.auth)”| Field | Type | Default | Description |
|---|---|---|---|
mode |
string | Human auth mode. "proxy" is the only value the server checks: it then uses the proxy authenticator configured under proxy and offers no OAuth providers. Any other value, including unset (the default), "oauth" and "dev", leaves the hub handling authentication itself. |
|
dev_mode |
bool | false |
Enable insecure development authentication. This (or the --dev-auth flag) is the dev auth switch; mode does not enable it. Requires a loopback listen host (127.0.0.1, ::1, or localhost), both when the web server is enabled and when the Hub API runs standalone with the web server disabled. Browser sign-in uses single-use launch links minted by POST /auth/dev-launch and consumed at GET /auth/dev-signin (both return 404 when dev_mode is false). |
dev_token |
string | Static token for dev mode. | |
dev_token_file |
string | ~/.scion/dev-token |
File the dev mode token is read from, or written to if it does not exist. A relative path is resolved against the global config directory (~/.scion). |
authorized_domains |
list | [] |
Limit access to specific email domains. On a database-backed Hub, SCION_SERVER_AUTH_AUTHORIZEDDOMAINS on a node ranks above a saved list, and clearing the saved list returns to the startup list; see server.auth.authorized_domains: environment first. |
user_access_mode |
string | "open" |
Who may sign in: "open" (any verified email, subject to authorized_domains if set), "domain_restricted" (email domain must be in authorized_domains), or "invite_only" (the email must belong to an invited, allow-listed or existing user). Users in admin_emails are always allowed. |
default_user_role |
string | "member" |
Hub role given to a user when their account is first created or activated: first sign-in, including the first sign-in of an invited or allow-listed user. Values: "member" or "viewer" ("admin" is rejected; use admin_emails). Users in admin_emails are always admins. Changing it does not affect existing users. It is also the role given to an admin who is removed from admin_emails. See Hub roles. Environment: SCION_SEED_SERVER_AUTH_DEFAULTUSERROLE (recommended) or SCION_SERVER_AUTH_DEFAULTUSERROLE (per-node, deprecated for Layer-1 keys). |
agent_run_scope |
string | "off" |
How the Hub treats the run an agent token was issued for. Agent tokens carry a run_id claim. "off" ignores it; "observe" compares it with the agent’s current run and logs and counts mismatches (metric scion.hub.agent_token.run_scope) without refusing any request. Enforcement cannot be selected yet. Read at startup, so a change needs a Hub restart. Environment: SCION_SERVER_AUTH_AGENTRUNSCOPE. |
agent_run_scope_legacy_until |
string | RFC 3339 time after which agent tokens issued without a run_id claim are no longer accepted by the run check (in observe mode, counted as mismatches). Empty means no cut-off. Environment: SCION_SERVER_AUTH_AGENTRUNSCOPELEGACYUNTIL. |
Proxy Auth (server.auth.proxy)
Section titled “Proxy Auth (server.auth.proxy)”Proxy authentication configuration (consulted when server.auth.mode is set to "proxy"). See Proxy Auth (Google IAP) for the full deployment guide, and Generic JWT proxy provider for bespoke auth proxies.
| Field | Type | Default | Description |
|---|---|---|---|
provider |
string | Selects the proxy auth provider: "iap", "jwt", or "header". |
|
require_trusted_proxy_ip |
bool | false |
Enables defense-in-depth IP allowlisting. Uses the trusted_proxies CIDR list. |
driver_principals |
list | [] |
Exact emails (no wildcards) of the browser drivers, such as a test-UI service account, that may carry test identity browser sessions through the proxy. A driver principal never gets a hub account: without a test identity session, page navigations and assertion-only API requests get 403 (driver_principal_has_no_session), and API requests with its ID token never act as it (401 or 403). Startup only; never writable through the settings API and not shown by it. A hub whose mode is not "proxy" refuses to start with a non-empty list. See Browser Sessions as Test Identities. |
driver_token_audience |
string | The IAP OAuth client ID. On /api/v1/ requests, a driver’s Google ID token with exactly this audience is replaced by the test identity session it carries (only when every condition in the how-to holds). Unset: never replaced. Startup only; never writable through the settings API and not shown by it. |
Google IAP Settings (server.auth.proxy.iap)
Section titled “Google IAP Settings (server.auth.proxy.iap)”| Field | Type | Default | Description |
|---|---|---|---|
audience |
string | MANDATORY for IAP. The expected audience claim (aud) in the IAP-signed JWT assertion. Supported formats are Cloud Run native path or GCE/GKE GCLB backend service path. |
|
issuer |
string | "https://cloud.google.com/iap" |
The expected JWT issuer. Override only for mock/testing setups. |
jwks_url |
string | "https://www.gstatic.com/iap/verify/public_key-jwk" |
The URL to retrieve public keys for signature verification. Override only for testing. |
Generic JWT Settings (server.auth.proxy.jwt)
Section titled “Generic JWT Settings (server.auth.proxy.jwt)”Used when provider is "jwt". Exactly one key source — public_key_file, jwks_url, or jwks_file — must be set; the Hub refuses to start otherwise.
| Field | Type | Default | Description |
|---|---|---|---|
header |
string | "X-Auth-Proxy-JWT" |
HTTP header carrying the signed JWT assertion. |
algorithm |
string | Required. The single accepted signing algorithm. Must be asymmetric: RS256, RS384, RS512, ES256, ES384, ES512, PS256, PS384, or PS512. |
|
issuer |
string | If set, must match the JWT iss claim. Empty skips issuer validation. |
|
audience |
string | If set, must be contained in the JWT aud claim. Empty skips audience validation. |
|
public_key_file |
string | Key source: path to a PEM-encoded public key (PKIX, PKCS1, or an X.509 certificate). Loaded once at startup. | |
jwks_url |
string | Key source: JWKS endpoint. Keys are cached and refreshed hourly and on an unknown kid, with the last-good key set served if the endpoint fails. |
|
jwks_file |
string | Key source: path to a local JWKS JSON document. Loaded once at startup; keys are matched by kid. Changes require a restart. |
|
claims.email |
string | "email" |
Claim holding the user’s email. The claim is required on every token. |
claims.subject |
string | "sub" |
Claim holding the stable subject ID. Falls back to the email when absent. |
claims.display_name |
string | "name" |
Claim holding the display name. |
claims.domain |
string | "hd" |
Claim holding the hosted domain. |
Transport Auth (server.auth.transport)
Section titled “Transport Auth (server.auth.transport)”Transport auth configuration for the platform guard (IAP or Cloud Run invoker). See Proxy Auth (Google IAP) for the full deployment guide.
| Field | Type | Default | Description |
|---|---|---|---|
mode |
string | "none" |
Transport auth mode: none, iap, or cloudrun_invoker. |
oidc_audience |
string | OIDC audience for transport tokens. For iap: the IAP OAuth client ID. For cloudrun_invoker: the Hub URL (auto-derived from hub.public_url if empty). |
|
platform_auth_sa |
string | Dedicated service account the Hub impersonates to mint OIDC ID tokens for agents. This account is never provisioned as a Hub user and is never issued user credentials, even if it signs in. |
Agent transport environment variables
Section titled “Agent transport environment variables”When transport auth is configured, the Hub injects these environment variables into agent containers at dispatch time:
| Variable | Description |
|---|---|
SCION_TRANSPORT_TOKEN |
Initial Google OIDC ID token for the transport layer. Bootstrap only: sciontool init moves it to ~/.scion/transport-token and removes it from the child environment. If the file cannot be written, the value stays in the environment and is used until it expires. |
SCION_TRANSPORT_TOKEN_FILE |
Set by sciontool init for child processes. Path of the transport token file, which every refresh rewrites. |
SCION_TRANSPORT_AUDIENCE |
Audience the transport token was minted for. |
SCION_TRANSPORT_TOKEN_EXPIRY |
Expiry of the initial token, in RFC 3339 format. Bootstrap only: removed by sciontool init together with SCION_TRANSPORT_TOKEN. |
SCION_TRANSPORT_MODE |
Transport mode (iap or cloudrun_invoker). Injected alongside the other three transport vars so that in-agent clients can select the correct header placement. |
Broker transport configuration
Section titled “Broker transport configuration”Brokers are long-lived originators that mint their own OIDC tokens (via GKE Workload Identity or ambient GCE SA). Transport settings are configured via environment variables or per-connection credentials-file fields.
Environment variables (for containerized brokers):
| Variable | Description |
|---|---|
SCION_TRANSPORT_MODE |
Transport mode: iap or cloudrun_invoker. |
SCION_TRANSPORT_AUDIENCE |
OIDC audience — the custom OAuth 2.0 Client ID (for iap) or Hub URL (for cloudrun_invoker). |
Credentials-file fields (per hub connection, in ~/.scion/hub-credentials/<name>.json, persisted by scion runtime-broker register or scion runtime-broker join):
| Field | Type | Description |
|---|---|---|
transportMode |
string | Transport mode: iap or cloudrun_invoker. |
transportAudience |
string | OIDC audience for the transport token. |
Environment variables override credentials-file values. Per-connection credentials-file fields support the multi-hub scenario where each hub has a different IAP OAuth client ID.
OAuth (server.oauth)
Section titled “OAuth (server.oauth)”OAuth provider credentials.
server: oauth: web: google: { client_id: "...", client_secret: "..." } github: { client_id: "...", client_secret: "..." } cli: google: { client_id: "...", client_secret: "..." }Storage (server.storage)
Section titled “Storage (server.storage)”Backend for storing templates and artifacts.
| Field | Type | Default | Description |
|---|---|---|---|
provider |
string | "local" |
Storage provider: local or gcs. |
bucket |
string | GCS bucket name. | |
local_path |
string | Local path for storage. |
Secrets (server.secrets)
Section titled “Secrets (server.secrets)”Backend for managing encrypted secrets. The local backend is read-only and rejects secret write operations. Configure gcpsm to enable full secret management.
| Field | Type | Default | Description |
|---|---|---|---|
backend |
string | "local" |
Secrets backend: local or gcpsm. The local backend rejects writes; use gcpsm for production. |
gcp_project_id |
string | GCP Project ID for Secret Manager. Required when backend is gcpsm. |
|
gcp_credentials |
string | Path to GCP service account JSON or the JSON content itself. Optional if using Application Default Credentials. | |
gcp_replication_locations |
list of strings | [] |
GCP regions for user-managed secret replication (e.g. ["us-east1", "europe-west1"]). When non-empty, secrets are created with user-managed replication restricted to these regions instead of automatic global replication. Required for GCP orgs enforcing constraints/gcp.resourceLocations. See User-Managed Replication Locations. |
Project Storage (server.project_storage)
Section titled “Project Storage (server.project_storage)”The one storage block of a Hub and its Runtime Brokers. backend places the workspace and the shared directories of every project, with one value for both. homes places agent home directories, and is the only per-data-class choice. The block replaces server.workspace_storage, server.shared_dir_storage and server.home_storage, which are deprecated and will be removed in a later release (see Migrating from the deprecated blocks).
server: project_storage: backend: nfs # local | nfs | cloudrun-volume | gke-shared-volume homes: local # local | nfs (nfs needs backend nfs) nfs: mount_root: /mnt/scion-nfs subpath_root: projects shares: - id: shared server: 10.0.0.2 export: /scion-projects pv_name: scion-projects-pvc| Field | Type | Default | Description |
|---|---|---|---|
backend |
string | "local" |
Where workspaces and shared directories live: "local" (node-local directories), "nfs" (an NFS export), "cloudrun-volume" (a Cloud Run platform-managed volume) or "gke-shared-volume" (a GKE shared CSI-backed PVC). Names are case-sensitive. |
homes |
string | "local" |
Where agent home directories live: "local" (in the pod, or on the broker host) or "nfs" (on the nfs share of this block). "nfs" needs backend: nfs. See Agent home storage. |
nfs.* |
object | The NFS block, with the same fields and defaults as server.workspace_storage.nfs: mount_root, mount_options, auto_mount, uid, gid, storage_class, subpath_root and shares (id, server, export, pv_name). Used when backend is "nfs". |
|
cloudrun_volume.* |
object | volume_name and subpath_root, as in server.workspace_storage.cloudrun_volume. Used when backend is "cloudrun-volume". |
|
gke_shared_volume.* |
object | volume_name, pv_claim_name and subpath_root, as in server.workspace_storage.gke_shared_volume. Used when backend is "gke-shared-volume". |
|
homes_options.* |
object | The operational knobs of agent homes, with the meaning and defaults they had under server.home_storage: leaf (pod), stop_grace_seconds (30), termination_wait_seconds (15), skeleton_max_bytes (268435456) and allow_incomplete_phases (false). See Agent home storage. |
The operational content of the deprecated sections still applies to the same fields under server.project_storage: startup validation of the backend blocks, subpath_root, NFS mounts on the Runtime Broker, NFS workspaces on Kubernetes, cleanup on project delete, the 503 safety gate, agent groups on NFS shared directories and the cloudrun runtime’s need for the nfs backend.
Where each data class lives
Section titled “Where each data class lives”backend |
Workspaces | Shared directories |
|---|---|---|
local |
Node-local | Node-local, next to the project’s local config |
nfs |
<mount_root>/<share id>/<subpath_root>/<project id>/... on the export |
<subpath_root>/<project id>/shared-dirs/<name> on the same export, on every runtime, including Docker and Podman |
cloudrun-volume, gke-shared-volume |
<subpath_root>/<project id>/... on the volume |
<subpath_root>/<project id>/shared-dirs/<name> on the same volume |
- Docker and Podman follow the backend. With
backend: nfs, shared directories are bind-mounted from the export on Docker and Podman too. Before this block existed, a broker withserver.workspace_storage.backend: nfsand noserver.shared_dir_storagekept Docker shared directories node-local; that is no longer the case. For shared directories already on node-local disk: the Hub’sproject-storage-migratemaintenance migration (Admin → Maintenance, orPOST /api/v1/admin/maintenance/migrations/project-storage-migrate/run; run it withdryRunfirst) moves the Hub’s own trees (and moves stopped (stoppedorhalted, not running orerror) Cloud Run agents’ legacyagents/<agent id>/{home,secrets}directories on the workspace share into the unified<subpath_root>/<project id>/agents/<slug>/{home-,secrets-}<agent id>layout under the project’s provisioning locks; running agents are deferred and a Cloud Run agent whose legacy home has not yet been moved is refused at start); each Runtime Broker moves its own node-local shared dirs (~/.scion/project-configs/<slug>__<id8>/shared-dirs/<name>) into<ProjectRoot>/shared-dirs/<name>on the share on its first start after upgrade with its shared-dir layout on the share (copy-only, never overwriting; the source is kept as<name>.migrated-<ts>). - Hub-seeded
.scionon the share, broker-local agent state. On a shared backend (nfs,cloudrun-volume,gke-shared-volume), the Hub seeds<ProjectRoot>/.scion(project-id,project-slug, and where applicablesettings.yaml) at project create, clone, and register for per-agent git projects (clone-per-agentandempty-per-agent) as well as Hub-managed and shared-workspace projects (and provisions<ProjectRoot>/workspaceat Hub-managed project create). The Runtime Broker runs Hub-managed projects from<ProjectRoot>(<subpath_root>/<project id>) and reads<ProjectRoot>/.scion, while keeping every agent’s state directory (agent-info.json, local Agent home copy, worktree state) broker-local under~/.scion/project-configs/<slug>__<id8>/.scion/agents(falling back to~/.scion/projects/<slug>/.scion/agentsfor pre-upgrade agents). Withbackend: nfs,clone-per-agentandempty-per-agentworkspaces live on the share at<subpath_root>/<project id>/agents/<slug>/workspaceon Kubernetes and on the local container runtimes (Docker, rootful Podman and Applecontainer, which bind-mount that directory from the Runtime Broker’s mount of the share; rootless Podman does not support NFS workspace storage), and a workspace path requested for such aclone-per-agentagent (--workspace) is ignored. On Docker, Podman, and Applecontainer, the NFS container bind check (nfsBindScope) admits only the project workspace (<subpath_root>/<project id>/workspace) or, forclone-per-agentandempty-per-agenton the shared layout, the starting agent’s own existing workspace (<subpath_root>/<project id>/agents/<slug>/workspace), as a clean absolute path with no..and no symbolic links below the share mount point; another agent’s workspace,agents/itself, or any nested path is refused. On Cloud Run Instances (thecloudrunruntime),clone-per-agentworkspaces also live on the share. An agent created before its project moved to the shared layout, whose workspace is not in its directory on the share and whose broker-local workspace holds files other than.scionand.scion-volumes, is refused at start on a local container runtime with409(if the broker-local workspace holds only.scionand.scion-volumes, start on the share proceeds): the Runtime Broker log names the workspace, and deleting the agent removes it, so move any work out first, then recreate the agent. Agent homes on the shared export use one path shape across Kubernetes and Cloud Run:<subpath_root>/<project id>/agents/<slug>/home-<agent id>. - Volume backends open the shared-directory write gate. Because
cloudrun-volumeandgke-shared-volumeput shared directories on the volume, a Cloud Run Hub accepts shared-directory writes on those backends. A Cloud Run Hub without a shared backend refuses them with503and error codeshared_dir_writes_unavailable, whose message namesserver.project_storage. - Removing a shared directory deletes it. Removing a shared directory declaration deletes its directory on the shared backends, as it does on local disk.
- Node-local directories are volumes. A directory that must stay on the node, such as a build cache, is an ordinary
volumes:entry on a profile, harness config or template, not a storage backend.
Configuration source (Layer-0)
Section titled “Configuration source (Layer-0)”server.project_storage is a Layer-0 setting. The Hub and every Runtime Broker read it from their own global settings.yaml at startup; the Hub also reads SCION_SERVER_PROJECT_STORAGE_* environment variables. It is never read from the database or from a project’s settings. On a workstation Hub, a PUT /api/v1/admin/server-config body that sets server.project_storage writes it into settings.yaml and lists the key in the response’s reload.requires_restart; a hosted Hub refuses it with 422 (see Layer-0 edits by deployment mode). The admin UI has no field for it. Either way, a change needs a restart. Configure the same block on the Hub and on every Runtime Broker that serves its hub-native projects (see Layout identity check).
Validation
Section titled “Validation”The Hub refuses to start, and scion config validate reports an error, when the conditions below hold. The error names the field at fault. scion config migrate still converts a file whose storage block breaks a field rule, and reports each such field as a warning, so that you can fix the converted file.
A Runtime Broker with an invalid block still starts, so that you can stop and delete its existing agents, but it has no storage. It does not fall back to the local layout. It logs the error at startup and refuses every agent create, start and restart with 422 and error code storage_invalid, naming the fields at fault. Its GET /healthz status is degraded, with the check project_storage: unavailable. Its storage descriptor is marked invalid, so the Hub also refuses shared-mode dispatches to it (see Layout identity). Stop, delete, logs and status keep working. Fix the block and restart the broker.
On a workstation Hub, the admin settings API (PUT /api/v1/admin/server-config) checks a storage block by the same rules before it writes anything. It refuses an invalid block with 422 (validation_error), and the error’s details.errors lists each field with its rule. Nothing is written, and nothing is reported as needing a restart.
server.project_storageis set together with any ofserver.workspace_storage,server.shared_dir_storageorserver.home_storage. Exactly one storage block configures a deployment; the error names the blocks to remove.- the backend block is invalid, by the same rules as
server.workspace_storage, reported underserver.project_storage; - a field of the selected backend block breaks one of the field rules below;
homesis notlocalornfs, or isnfswhilebackendis notnfs;- a
homes_optionsvalue is invalid; - (Hub) the Hub cannot build its storage layout: for example, the share’s host base is not an absolute path, because
nfs.mount_rootis empty or relative. The Hub serves files from its own mount, so it refuses to start rather than refuse every shared-directory request later; - (Hub) the global settings file, in any format, has a storage block key (
project_storage,workspace_storage,shared_dir_storageorhome_storage) as a parsed mapping key, but the block was not loaded: for example a block not underserver:, a missingschema_version: "1", or a server section that fails to decode. The Hub refuses to fall back to the local backend. A comment, or a key such asshared_dir_storage_classthat only contains the name, never triggers this.
Field rules
Section titled “Field rules”These rules apply to the backend block that backend selects. They apply to a block synthesized from the deprecated blocks too, and its errors name the deprecated keys.
| Field | Rule |
|---|---|
nfs.mount_root |
An absolute, clean path (no trailing /, no . or .. components). |
nfs.shares[].id |
Required, at most 63 bytes, and unique. It is used as one directory name under mount_root, so it must not contain /, \, whitespace, NUL or control characters, and must not be . or ... Existing ids such as scion-hub1 stay valid. |
nfs.shares[].server |
Required. An IPv4 address or an RFC 1123 hostname. No leading -, whitespace, , or /. IPv6 literals are not supported. |
nfs.shares[].export |
Required. An absolute, clean path of at most 1024 bytes, without .. components, whitespace, , or NUL. |
nfs.mount_options |
Comma-separated options. Each is a lower-case name (letters, digits, _), optionally followed by = and a value made of letters, digits and ._:/+-. Empty options are not allowed. suid and dev are refused, as is nosuid, nodev or rw with a value. Mount options are hardened separately. |
nfs.shares[].pv_name, nfs.storage_class, gke_shared_volume.pv_claim_name |
Empty, or a DNS-1123 subdomain: lower-case letters, digits, - and ., at most 253 bytes. |
cloudrun_volume.volume_name, gke_shared_volume.volume_name |
One directory name, by the same rule as nfs.shares[].id. |
subpath_root, nfs.uid, nfs.gid |
As described for server.workspace_storage. |
For the shared-directory export of a transitional split (see below), server and export are optional, because that export is never mounted by the broker. If you set them, they must follow the same rules.
Migrating from the deprecated blocks
Section titled “Migrating from the deprecated blocks”Without server.project_storage, the Hub and the Runtime Brokers synthesize it from the deprecated blocks at startup:
server.workspace_storagegivesbackendand the backend blocks.server.home_storage.backendgiveshomes; its other fields givehomes_options.server.shared_dir_storagedecides where shared directories go:nfson the same tree as the workspace (same NFS server, export andsubpath_root): nothing changes.nfswhile the workspace is not onnfs: shared directories (andnfshomes) stay on theshared_dir_storageexport and workspaces stay where they are. This transitional split exists only for synthesized configurations; it cannot be written as aserver.project_storageblock.localwhile the workspace is onnfs: shared directories are now placed on the export with the workspace, and a warning says so.- unset while the workspace is on
nfs: shared directories follow the workspace onto the export on every runtime (Docker, Podman and Apple agents kept them node-local before), and a warning says so. For existing data: the Hub’sproject-storage-migratemaintenance migration (Admin → Maintenance, orPOST /api/v1/admin/maintenance/migrations/project-storage-migrate/run; run it withdryRunfirst) moves the Hub’s own trees; each Runtime Broker moves its own node-local shared dirs into<ProjectRoot>/shared-dirs/<name>on the share on its first start after upgrade with its shared-dir layout on the share (copy-only, never overwriting; the source is kept as<name>.migrated-<ts>). - unset, or
local, while the workspace is oncloudrun-volumeorgke-shared-volume: shared directories stay where they are today (their own claims on Kubernetes) untilserver.project_storageis written. The Hub then treats them as non-durable, so a Cloud Run Hub refuses shared-directory writes with503. Writing the block moves shared directories onto the volume, and the contents of the existing per-directory claims are not moved automatically. While such a claim still exists, every agent start that now places that directory on the volume returns a warning naming the claim. Scion never deletes those claims.
Every synthesis logs a startup warning that names the deprecated blocks in use and prints the exact server.project_storage block to write, with defaults filled in. The Hub’s project_storage_config health check reports degraded until the block is written. For the transitional split, the printed block is backend: nfs on the shared-directory export: writing it moves workspaces onto the export too, because one backend places both.
Synthesis is refused, and startup fails, only when the deprecated blocks are ambiguous:
workspace_storageandshared_dir_storageare bothnfsbut name different trees (a different NFS server or export, or a differentsubpath_root; the share id andmount_rootare host-local naming and may differ). One storage block places both, so choosing either would hide the other’s data. Move one tree onto the other’s share, then writeserver.project_storagenaming it.home_storage.backendisnfsbut neitherworkspace_storagenorshared_dir_storageisnfs, so there is no share for the homes.
For example, a deployment with workspaces and shared directories on the same export and the broker home leaf:
# Before (deprecated)server: workspace_storage: backend: nfs nfs: mount_root: /mnt/scion-nfs shares: - id: shared server: 10.0.0.2 export: /scion-projects pv_name: scion-projects-pvc shared_dir_storage: backend: nfs nfs: mount_root: /mnt/scion-nfs shares: - id: shared server: 10.0.0.2 export: /scion-projects pv_name: scion-projects-pvc home_storage: leaf: broker# Afterserver: project_storage: backend: nfs nfs: mount_root: /mnt/scion-nfs shares: - id: shared server: 10.0.0.2 export: /scion-projects pv_name: scion-projects-pvc homes_options: leaf: brokerRemove all three deprecated blocks in the same edit, on the Hub and on every Runtime Broker, and restart. Check the edited file with scion config validate. For data that the new block places elsewhere, such as hub-local workspaces and node-local shared directories: the Hub’s project-storage-migrate maintenance migration (Admin → Maintenance, or POST /api/v1/admin/maintenance/migrations/project-storage-migrate/run; run it with dryRun first) moves the Hub’s own trees; each Runtime Broker moves its own node-local shared dirs into <ProjectRoot>/shared-dirs/<name> on the share on its first start after upgrade with its shared-dir layout on the share (copy-only, never overwriting; the source is kept as <name>.migrated-<ts>). No scion CLI command runs this migration; see Maintenance Panel Operations.
Removing a shared directory deletes its data
Section titled “Removing a shared directory deletes its data”Removing a shared-directory declaration through the Hub deletes that directory and its contents on a shared backend (nfs, cloudrun-volume, gke-shared-volume, and the shared-directory export of the transitional split). The deletion uses the same confined, symlink-refusing walk as project deletion. The local backend is unchanged: the local directory is deleted, as before. The Required scratchpad cannot be removed while project_defaults.default_scratchpad is on (409).
Removed storage overrides
Section titled “Removed storage overrides”Storage placement is no longer chosen per runtime, per profile, per instance, per shared directory or per agent. These keys are removed:
shared_dir_storage_backendandshared_dir_storage_backendsonruntimes.<name>,profiles.<name>andserver.broker.instances[].runtime_target;home_storage_backendandhome_storage_leafon the same three places.
Settings validation (scion config validate), the Hub admin settings API (PUT /api/v1/admin/server-config) and flat Runtime Broker instance loading reject them with an error that names the key, server.project_storage and volumes:.
Startup reports them and agent start refuses them:
- Startup reports. When the settings file, or the runtimes and profiles stored in the Hub, still carry one of these keys on
runtimes.<name>orprofiles.<name>, the Hub, the Runtime Broker and a flat host log an error at startup that names each key, where it was found and the migration below. The process keeps running. The Hub’sstorage_overrideshealth check reportsdegraded, a Runtime Broker’s/healthzreportsstorage_overrides, andGET /api/v1/admin/server-configlists the messages instorage_override_errors. A fix to the Hub’s stored settings clears the Hub’s entry when it is saved; the Hub reads the settings file at startup, so a fix there clears its entry when the Hub restarts. A Runtime Broker re-reads its settings within 30 seconds. - Agent start refuses. The Runtime Broker refuses to start an agent whose profile, or that profile’s runtime entry, carries one of these keys. The refusal comes before anything is created, with
422and error codestorage_migration_required. Agents of other profiles start normally. An agent whose shared-directory backend record from an earlier version (shared-dir-storage.json, or the.retiredcopy of it) pins shared directories tonfsis refused the same way while the shared-directory layout is not shared. The record is kept. If that data was copied elsewhere or is deliberately left behind, removing the record lets the agent start on the current layout.
To migrate a shared_dir_storage_backend: nfs override, edit the settings and remove the key:
- set
server.shared_dir_storage.backend: nfs(transitional: shared directories of every runtime then use that export, and workspaces stay where they are), or - write
server.project_storagewithbackend: nfs(this also moves workspaces onto the export).
Check shares[0].pv_name and subpath_root against the live export. Kubernetes agents mounted that claim at <subpath_root>/<project id>/shared-dirs/<name>, so these two decide where the existing shared-directory data is. Keep shares[0].server and export accurate as well. For a home_storage_backend: nfs override, set server.home_storage.backend: nfs (or homes: nfs in server.project_storage). Then run scion config validate. Stored runtimes and profiles are edited in the Hub admin settings; a save drops the removed keys.
Agents no longer record their own shared-directory backend, and the operation that changed an agent’s recorded shared-directory backend is removed. For a node-local directory, use a volumes: entry instead.
Layout identity check
Section titled “Layout identity check”Every Runtime Broker reports its storage layout in its heartbeat descriptor: the backend, for nfs the share server, export, subpath_root and the export identity marker, and homes. When the Hub’s own backend is shared (not local), it compares a broker’s report with its own layout:
- Dispatch refusal: the Hub refuses to dispatch an agent of a hub-native project in the shared-plain or worktree-per-agent sharing mode to a broker whose layout disagrees, with
409and a message naming each differing field (for examplenfs export: differs) and asking for the sameserver.project_storageon the broker. The refusal never includes the configured values;details.storageLayoutDifflists the field names. The hub log has the full values. A broker that reports no layout (an older version) is refused the same way; upgrade it. Linked projects, which the broker provides from its own path, and per-agent workspace modes are not checked. - Health: the admin health summary shows the differences as
layout_mismatchin the broker’s storage entry and counts the broker as a problem. A hub admin sees the values on both sides; any other reader sees the differing fields by name only. - Broker records:
GET /api/v1/runtime-brokersandGET /api/v1/runtime-brokers/{id}include the NFS server, export, subpath root and export identity marker, and the volume claim, of a broker’s storage descriptor for hub admins only (and for the broker reading its own record). Other readers see the backend, homes and share health; the value fields are omitted and the descriptor carriesredacted: true. The project registration response follows the same rule for the linked broker. Agent-move refusals between brokers on different exports also name the differing fields only.
Workspace Storage (server.workspace_storage)
Section titled “Workspace Storage (server.workspace_storage)”Configures the backend and mount settings for storing and managing agent workspaces. This is a critical setting for high-availability deployments where multiple Hub and Broker replicas need shared, durable access to project workspaces.
| Field | Type | Default | Description |
|---|---|---|---|
backend |
string | "local" |
Storage backend pivot: "local" (node-local directories), "nfs" (Network File System mounts), "cloudrun-volume" (Cloud Run platform-managed volume mounts), or "gke-shared-volume" (GKE shared CSI-backed PVC mounts). Names are case-sensitive; any other value stops the Hub at startup. |
nfs.mount_root |
string | The host base directory under which NFS exports are mounted. If it is a symlink, the Runtime Broker resolves it, so the share paths that the nfs_mounts check of GET /healthz and scion doctor report are under the resolved directory, not the configured one. |
|
nfs.mount_options |
string | "vers=3,hard,nconnect=4,_netdev" |
Standard mount options passed to the mount.nfs utility. The broker always adds nosuid,nodev as the last options; they cannot be turned off. |
nfs.auto_mount |
boolean | false |
Whether the Runtime Broker mounts the shares itself. See NFS Mounts on the Runtime Broker. Requires the broker to run as root. |
nfs.uid |
integer | 1000 |
Node-independent owner UID for NFS-backed workspace trees to ensure consistent container write permissions. Ignored by the Kubernetes runtime, where agents always run as uid 1000; when it is set to another value, the broker logs a warning the first time an agent starts on Kubernetes with that value in each broker process, so it appears again after a broker restart. Must be between 0 and 4294967294; 0 or unset means 1000. |
nfs.gid |
integer | 1000 |
Node-independent owner GID for NFS-backed workspace trees. Must be between 0 and 4294967294; 0 or unset means 1000. |
nfs.storage_class |
string | The Kubernetes StorageClass name used to dynamically allocate volumes on GKE. | |
nfs.subpath_root |
string | "projects" |
The base folder within the share for project workspaces. See subpath_root. |
nfs.shares |
list of objects | [] |
List of NFS share objects. Each share requires: id (stable ID), server (IP address or hostname), export (exported path, e.g., /scion-workspaces), and optional pv_name. With pv_name, Kubernetes pods mount that PersistentVolumeClaim by subPath. Without it, they mount each directory as an inline nfs volume of its server path, with no NFS mount options, and the Runtime Broker must have the export mounted writably to create each directory first; see Inline NFS volumes. |
cloudrun_volume.volume_name |
string | Required when backend is "cloudrun-volume" (the settings schema checks this only for the selected backend). The name of the platform volume declared in the Cloud Run service specification. The Hub resolves workspaces under /mnt/<volume_name>, so the service must mount the volume at exactly that path; see the Cloud Run volume example in the GCP HA setup guide. If it is missing or empty, the Hub refuses to start. |
|
cloudrun_volume.subpath_root |
string | "projects" |
Sub-directory prefix within the Cloud Run volume. See subpath_root. |
gke_shared_volume.volume_name |
string | Required when backend is "gke-shared-volume"; if it is missing or empty, the Hub refuses to start. The K8s volume name referencing the persistent volume claim (PVC). The pod spec must mount that volume at /mnt/<volume_name>: the Hub derives every workspace path from it, and a pod that mounts the PVC elsewhere fails readiness (GET /readyz returns 503) rather than writing workspaces to ephemeral container storage. |
|
gke_shared_volume.pv_claim_name |
string | The name of the GKE-managed PVC bound to the shared storage backend (e.g. Filestore). | |
gke_shared_volume.subpath_root |
string | "projects" |
Sub-directory prefix within the GKE volume. See subpath_root. |
Startup validation
Section titled “Startup validation”The Hub checks workspace_storage (or the backend fields of project_storage) when it starts, and refuses to start with an error naming the bad field when:
backendis not one of the four names above;backendis"nfs"andnfs.sharesis empty;backendis"cloudrun-volume"or"gke-shared-volume"and the matchingvolume_nameis missing or empty;- the selected backend’s
subpath_rootis invalid (see below).
A Runtime Broker that runs without the Hub does not refuse to start. If the block is invalid by these rules or by the field rules, including an incomplete nfs block or an invalid subpath_root (values such as projects/ or ./projects, which earlier versions accepted and normalized, are invalid), the broker logs the error, skips its NFS mount checks and refuses every agent start, as described under Validation.
The Hub’s readiness check (GET /readyz) and its Cloud Run write guard still apply as further safeguards. A volume backend without a mount point fails readiness, and blocks workspace writes on Cloud Run.
subpath_root
Section titled “subpath_root”subpath_root is the directory inside the share or volume that holds project trees, at <subpath_root>/<project-id>/.... It defaults to "projects" for every backend, and for the deprecated shared_dir_storage.nfs. It must be a clean relative path: no leading /, no . or .. components, no empty components and no trailing /. If a value is not clean, the error names the clean value to use instead (for example projects for projects/). It may have several components, for example team/projects. The Cloud Run runtime’s NFS export paths use the same nfs.subpath_root.
NFS Mounts on the Runtime Broker
Section titled “NFS Mounts on the Runtime Broker”When backend is nfs, the Runtime Broker reads this block from its global settings.yaml (never from project settings) and checks each share at <mount_root>/<share id>. The block must follow the field rules: an absolute mount_root, and for every share a unique single-segment id, a server (a hostname or an IPv4 address), and an absolute export. If the block is invalid, the broker logs a warning, skips NFS handling and refuses agent starts (see Validation). With any other backend, the broker does no NFS handling at all.
auto_mount: false(default): The Runtime Broker only checks. It reads the host mount table (/proc/mounts) to confirm each share is mounted from the expectedserver:exportwith thenosuidandnodevoptions, at startup and then every minute. It never mounts or unmounts anything and never refuses agent creation on the/proc/mountscheck alone. Mount each export on the host yourself withnosuid,nodev, for example with/etc/fstab. A share mounted without them is reported unhealthy (reasonmount_flags), and agent creation, start and restart on a Docker, Podman or Apple runtime are refused with503(nfs_unavailable) until it is remounted with them.auto_mount: true: The Runtime Broker mounts missing shares in the background, and remounts a share that is mounted from the wrong source. Every mount it makes carriesnosuid,nodevas its last options. A share mounted from the expected source without them is remounted in place (mount -o remount,nosuid,nodev); containers already running keep their current mount until they restart. The Runtime Broker must run as root: if it does not, it logs a warning at startup and only checks, without runningmount. If the Runtime Broker’s default runtime is Kubernetes or Cloud Run, it also only checks, because the platform mounts the export into the agent. Each mount command times out after 90 seconds, or sooner if the agent-create request ends.
With auto_mount: true, agent creation for a project on the nfs backend is decided by the runtime the agent is dispatched to. This check runs before anything is mounted. Only agent creation is checked by the mount gate; starting or restarting an existing agent is not, apart from the nosuid,nodev check below.
- Docker, Podman, or Apple: The Runtime Broker first makes sure the first share is mounted, because that share holds the workspaces. If it is not mounted, the Runtime Broker returns
503with error codenfs_unavailableand a generic remediation message (share IDs, mount points, and server/export paths are logged on the Runtime Broker and omitted from client responses). - Kubernetes or Cloud Run (verify-only), and shared
project_storagebackends (nfs,gke-shared-volume,cloudrun-volume): On Kubernetes or Cloud Run, the Runtime Broker never mounts and does not refuse the request on the/proc/mountsserver:exportstring match (which is advisory and logged as a warning when the container’s mount source differs in form, such as hostname vs IP or a trailing slash). Wheneverserver.project_storageuses a shared backend (nfs,gke-shared-volume, orcloudrun-volume), the Runtime Broker checks that the configured share mount point and<subpath_root>are visible directories (GET /healthzreportschecks.project_storage: "unhealthy: share_not_visible (…)"with overall statusdegraded, and the Runtime Broker heartbeat sendsproject_storage: unhealthywhen absent), refuses creates, starts, restarts, and wakes on shared-storage projects with503 project_storage_unavailableand a path-free remediation message (forwarded by the Hub as503 project_storage_unavailable), and verifies that<ProjectRoot>/.scion/project-idmatches the dispatched project (409on mismatch).
Whatever auto_mount is set to, creating, starting or restarting an agent on a Docker, Podman or Apple runtime for a project on the nfs backend first reads the mount table again. If the first share, or anything mounted at or below it, lacks nosuid or nodev (whatever its source), or the mount table cannot be read, the broker returns 503 with error code nfs_unavailable. This also applies when the broker’s default runtime is Kubernetes or Cloud Run and the agent runs on another runtime; such a broker still reports the share healthy, so the refusal shows only in the 503 response and the broker log. Kubernetes and Cloud Run agents are not checked: the platform mounts the export. Kubernetes agent pods run as non-root without privilege escalation and drop all capabilities. Cloud Run NFS volumes take no mount options, and Cloud Run instances run sandboxed.
In both modes, NFS problems are logged and reported per share in the nfs_mounts check of GET /healthz. The overall status becomes degraded when a shared server.project_storage directory is not visible (checks.project_storage), or when all of these hold for nfs_mounts: auto_mount is on, the default runtime is not Kubernetes or Cloud Run, the first check has finished, and a share is unhealthy. Neither nfs_mounts nor project_storage affects GET /readyz, and the Runtime Broker keeps serving projects that do not use shared storage. The Runtime Broker also sends its health to the Hub on every heartbeat, and the admin Health page shows it in the Health column of the Runtime brokers card. That copy holds only healthy or unhealthy for nfs_mounts and project_storage. To see which share is failing and the mount error, query GET /healthz on the Runtime Broker itself. scion doctor runs the same check locally, using the same mount-table lookup. It reports whether NFS is configured, whether each share is mounted from the expected source with nosuid,nodev, and whether each server is reachable on TCP port 2049. A share that is not mounted is a warning when auto_mount is off, the share has a pv_name, or the default runtime is Kubernetes or Cloud Run (so the Runtime Broker does not mount it, as in /healthz), and a failure otherwise. One exception: with a Kubernetes default runtime and no pv_name on the first share, that share not being mounted is a failure, because agent Pods then mount the share inline and the Runtime Broker must create each directory through its own mount. A share mounted without nosuid,nodev is a failure when auto_mount is off and a warning when it is on (the broker remounts it); it is not reported when the default runtime is Kubernetes or Cloud Run. When the Hub or Runtime Broker runs as root and creates a new workspace or shared-directory leaf directory on the shared export, it fchowns that newly created leaf to nfs.uid and nfs.gid before applying 2775 and the default ACL, while keeping intermediate directories, <ProjectRoot>/.scion, and <ProjectRoot>/provision owned by the creator.
NFS Workspaces on Kubernetes
Section titled “NFS Workspaces on Kubernetes”With the nfs backend, each Kubernetes agent pod gets a workspace-provision init container, whether the pod mounts the share through a bound PV claim (nfs.shares[].pv_name) or inline (no pv_name). It runs for both git and non-git agents. It creates the per-project subPath (or, if another pod is already provisioning it, waits for that pod to finish) and chowns it to the agent runtime uid (1000) and nfs.gid so the agent can write /workspace; nfs.uid is ignored on Kubernetes (see the nfs.uid row above). For git agents, it also clones the repository. The init container runs as root with only the CHOWN, FOWNER, and DAC_OVERRIDE capabilities and does not follow symlinks. If the chown fails, the agent start fails and the error names the failed init container, so the agent never runs with an unwritable workspace. A chown refused with EPERM on an entry that already has the wanted owner (uid 1000 and nfs.gid) does not count as a failure. That is what an export that maps root to an anonymous user (root_squash or all_squash) looks like to the init container. Such an export works only when it maps root to anonuid=1000 and anongid=<nfs.gid>, with nfs.uid unset or 1000: a Hub or Runtime Broker that runs as root gives each new workspace or shared-directory leaf it creates to nfs.uid (see above), and on such an export the init container cannot change that owner. With the default anonuid=65534, entries are owned by 65534 and the agent start fails.
Pods get fsGroup from nfs.gid (default 1000). For shared directories on the NFS export, Scion also adds each shared directory’s own group to the pod’s supplementary groups (see Agent groups), so nfs.gid does not need to match it.
Cleanup on project delete
Section titled “Cleanup on project delete”On a shared backend (nfs, cloudrun-volume, gke-shared-volume, and the shared-directory export of the transitional split), deleting a project removes its <subpath_root>/<project-id> tree: workspace, project configuration, shared directories, provisioning state, worktrees and agent directories. The Hub is the only remover of this tree: it removes it in the background from its own mount of the share, once the delete has committed. Runtime Brokers never remove it. Every Hub replica must have the share mounted. The removal does not follow a symlink anywhere in the path. A removal that fails part-way (for example while an agent pod is still terminating) is retried once, and if it fails again the Hub’s periodic maintenance pass retries it. If the Hub cannot see the share (the mount point has no <subpath_root> directory, or the check does not answer within 2 seconds), nothing is removed and the tree is kept for manual removal: the delete answers 200 with a storageKept item (code shared-project-storage) instead of 204, the Hub writes an ERROR project storage deletion line with outcome kept that names the project ID, and its health reports the share as missing (on nfs the Hub is not ready while it has projects, see below; on the other shared backends project_storage_tree is degraded). When the deleted project was the last one, the Hub has no projects left and stays ready, so the ERROR log line is the only signal. If the project exists again by the time cleanup runs, the tree is kept.
On nfs, the Hub must see the share, or it is not ready. A state-only Hub (one without the share mounted) on a shared layout is not supported. The critical workspace_storage check of GET /healthz (and with it GET /readyz) checks the share itself, not only the mount point. When the <subpath_root> directory is missing below the mount and the Hub has at least one project, workspace_storage is unhealthy: shared project storage is empty or not mounted (the mount point has no entries at all) or unhealthy: shared project storage has no project tree (the share has entries, but no <subpath_root>). /readyz then answers 503 (reason: workspace storage not available) and /healthz reports unhealthy, so the replica leaves service. A check that does not answer within 2 seconds is unhealthy: shared project storage check timed out, and any other error is unhealthy: shared project storage could not be checked. A Hub with no projects yet stays ready on an empty share, because the <subpath_root> directory appears with the first project; if the Hub cannot read its project list, it counts as having projects. The mount stat, the share check and the project-list read share one 2-second bound, and repeated checks of the same share wait on one check in flight. These values never name a path, and none of them is logged: the check runs on every health call, so its value is the signal. A <subpath_root> that is a symlink is refused (never followed) and reads as could not be checked, so it keeps even a Hub with no projects not ready; make the sub-path root a plain directory on the share. The other shared backends (cloudrun-volume, gke-shared-volume) still report a missing project tree as the non-critical project_storage_tree check (degraded), which does not affect /readyz (on nfs, project_storage_tree is retired and folded into workspace_storage).
A forced project delete (force=true) that finds agent records does not remove the tree at once, because those agents may still be running. The Hub writes provision/.scion-delete-after into the tree, with the time after which it may be removed: now plus server.hub.soft_delete_retention when that is longer than 24 hours, otherwise now plus 24 hours. The Hub’s periodic maintenance pass then removes the tree, the slug-named trees below and the workspace objects once that time has passed. If the project exists again by then, the tree is kept and the file is removed. A forced delete that finds no agent records removes the tree at once, like any other delete. On a backend that is not shared there is no tree to defer: a forced delete with agent records deletes the project’s workspace objects (transfer staging) at once, as any delete does, and touches no Runtime Broker workspace. Two Hubs must not share one <subpath_root> on one share: each Hub removes and defers only by its own project records.
The Hub also removes, on a shared backend, the slug-named trees left from before the project storage layout: hub-projects/<slug> on the share and the Hub’s ~/.scion/projects/<slug>, with their .migrated-<time> copies. A tree is removed only when its identity markers (.scion/project-id, the project ID in .scion/settings.yaml, or, for the Hub’s own ~/.scion/projects/<slug> but not its copies, the Hub’s workspace records) name the deleted project. A tree with no marker, or one that names another project, is kept and logged. When the Hub has object storage, it also deletes the project’s workspace objects under hubs/<hub-id>/workspaces/<project-id>/.
Each provider Runtime Broker removes its own ~/.scion/projects/<slug> directory and project-configs entry when no agents of the project remain on that broker: while the broker still lists any agent of the project in any phase (running, stopped, error, or halted), it keeps ~/.scion/projects/<slug> whole and answers 409 (agents still exist on this broker), which the Hub logs, audits as kept, and reports in the delete response’s storageKept list (code runtime-broker-project-directory, 200 instead of 204) without failing the project delete (when removing an agent directory, the broker removes scion-agent.json first so an interrupted removal is not mistaken for a remaining agent). A Runtime Broker with Kubernetes runtimes also deletes, in its runtimes’ default namespaces, the project’s per-directory shared-directory PersistentVolumeClaims. It deletes only claims that carry the project-ID label of the deleted project, a scion.shared-dir label, and a scion-shared-<project>-<dir> name, and that no existing pod in the namespace references. A claim a pod still references is kept and logged as in use. A claim without the project-ID label is never deleted: when it has the project’s name, it is listed in the Runtime Broker’s log for an operator to check. The share’s own claim (pv_name) is never deleted. A project’s shared directory never reuses a claim of the same name that is labelled for another project. Known limitations: claims are not released in a namespace that agents were placed in through a scion.namespace or namespace agent label, in the namespace of a Kubernetes profile the Runtime Broker has not resolved since it started, by the Hub’s embedded Runtime Broker, or for a git project linked at a local path on the Runtime Broker. Such claims are left in place, never deleted wrongly.
Deleting an agent also removes what the Hub owns of its storage on the share and in object storage: the agent’s home-<agent-id> and secrets-<agent-id> under <subpath_root>/<project-id>/agents/<slug>, then that directory itself once it is empty, and the agent’s objects under hubs/<hub-id>/workspaces/<project-id>/<agent-id>/. The agent’s own workspace (and its worktree) is removed by its Runtime Broker, never by the Hub. While the workspace is still there (for example after a forced delete whose Runtime Broker could not be reached), the directory is kept and logged as pending removal by the Runtime Broker. This happens on a delete that removes the agent’s files and is not a soft delete, or when the retained files of a soft-deleted agent are purged. A soft delete that keeps the files keeps it all. On Cloud Run (cloudrun), if a start fails because the instance create was refused before an instance could appear, the runtime also removes the agent directories (home-<agent-id>, workspace, and agents/<slug> if empty) that that start created on the share, while keeping any directories that already existed before the start.
Every removal is best-effort. A failure is logged and never fails or rolls back the delete; only a kept shared tree or a Runtime Broker 409 that keeps its project directory changes the delete’s answer (see above). Each removed or kept tree, claim and object prefix is logged as one project storage deletion line. The line names the deleter (hub or broker), the tree, the project, the path and the outcome.
Ephemeral Storage & 503 Safety Gate
Section titled “Ephemeral Storage & 503 Safety Gate”To protect deployments from silent data loss, the Hub implements a strict 503 Safety Gate:
- If the Hub is deployed on serverless environments like Google Cloud Run with the
localbackend selected, its local workspace paths map to ephemeral, non-durable container storage. - The Hub detects this non-durable state and refuses the project workspace file endpoints that change files (upload, inline file editing, and delete) and the shared-workspace
git pullendpoint. - These endpoints return
503 Service Unavailablewith error codeworkspace_writes_unavailableand a descriptive message rather than allowing writes to persist ephemerally on the container’s scratch space, enforcing the transition to a durable backend (nfs,cloudrun-volume, orgke-shared-volume) for production. - Shared-directory writes follow the same rule: on a Cloud Run Hub whose shared directories are not on a shared backend, they return
503with error codeshared_dir_writes_unavailable, and the message namesserver.project_storage.
Shared Directory Storage (server.shared_dir_storage)
Section titled “Shared Directory Storage (server.shared_dir_storage)”Selects where project shared directories are stored. With the nfs backend, every Runtime Broker resolves a project’s shared directories to the same path on one NFS export, so agents on different brokers — Docker, Kubernetes, or Cloud Run Instances (on server.project_storage) — see the same files.
This setting is global-only: each broker process reads it from its own global settings.yaml, never from project settings. It is a Layer-0 setting and requires a restart to take effect.
| Field | Type | Default | Description |
|---|---|---|---|
backend |
string | "local" |
"local" (shared directories live next to the project’s local config) or "nfs". Any other value, including a different case, is rejected. |
nfs.mount_root |
string | Required for nfs. Host directory under which the share is mounted, at <mount_root>/<shares[0].id>. Docker, Podman, and Apple runtimes bind-mount from here. |
|
nfs.shares |
list of objects | [] |
Required for nfs. Only the first entry is used. id is required. pv_name names the static PersistentVolumeClaim that Kubernetes pods mount by subPath, and is required for Kubernetes brokers. |
nfs.subpath_root |
string | "projects" |
Directory within the share that holds per-project trees. Must be a relative path. |
nfs.gid |
integer | Optional. The only shared-directory group that agents may be given as a supplementary group. When set, a leaf owned by any other group is skipped with a warning. See Agent groups. |
Shared directories resolve to <mount_root>/<share id>/<subpath_root>/<project id>/shared-dirs/<name>. On Kubernetes, pods mount the pv_name claim with the matching subPath instead of creating a per-directory PVC.
The nfs backend fails closed. Agent start is refused when the block is incomplete, the host base directory does not exist, the runtime cannot mount that export (for example, cloudrun-sandbox, or cloudrun on the split server.shared_dir_storage export), or a shared-directory path resolves through a symlink. The NFS export itself must be provisioned and mounted before agents start. The uid, mount_options, storage_class, and auto_mount fields of the nfs block are ignored here.
With the nfs backend, the Hub and brokers also apply the following:
- Symlink-safe access: Every Hub operation on an NFS shared directory goes through the same confined resolver. This covers the web file browser, archive downloads, and shared-dir deletion. The resolver walks each path component with
O_NOFOLLOW, anchored on the inode of the project’s tree, and refuses any symlink in the path. A missing or incompletenfsblock, or an unusable host base directory, fails closed on the Hub as well as on agent start. - Leaf modes and ACLs: A newly created shared directory gets mode
2775(setgid, group-writable) and a minimal default POSIX ACL, so files agents create inside it inherit group write access regardless of umask. If the export does not support POSIX ACLs, a warning is logged once and the directory stays plain2775with no ACL. Files created inside such a directory follow each writer’s umask (usually022), so they are not group-writable. Directories that already existed are never modified. See the hybrid tier guide for the manual fix-up recipe. - Chat attachments: Attachments are delivered into each recipient agent’s inbox (
~/.scion/inbox/<message-id>/) whenhub.attachment_artifactsis on. It is on by default wherever the artifact service is ready, including the workstation server and other Hubs with local storage (no object store is needed), and on Hubs whose resource storage is an object store; a stored override still applies. For a message in a project topic thread, whose readers are the whole project, the Hub also copies each attachment into the project’sscratchpadshared directory at.attachments/_webchat/<id>/<name>, and the message lists that inbox path first. It resolves the directory the same way as the file browser, so on NFS the copy lands where agents mount the scratchpad, and every write goes through the same confined resolver. The message tags the copy’s container path with its shared directory (attachment_mountsin the delivered envelope). Attachments in direct conversations and participant-list group conversations are delivered only into the recipients’ inboxes, with no shared copy, on every storage backend. An attachment that is not stored as an artifact (for example whilehub.attachment_artifactsis turned off) has no inbox delivery. On shared storage (NFS) its scratchpad copy is likewise made only for project topic threads; a direct or group conversation’s attachment is not copied there. On the workstation server (local storage) the copy is how an agent reads the file, so it is made for every message, direct conversations included. The other agents of the project can then read a direct message’s attachment. This is a known limitation of that configuration, and it goes away once the Hub stores attachments as artifacts. Files an agent attaches withscion message --attachare read from the same directory. Scratchpad copies older thanmessaging.attachment_inbox_retention_days(default 7) are removed by the Hub’s hourly maintenance pass, on every backend. - Ownership on an export that does not squash ids: the broker creates the project chain as its own user and never changes ownership. Upper directories get
2755and the leaf2775, and each inherits the group of a setgid parent. Pods create nothing on this export; they mount the existing leaf bysubPath. When agents with different uids share a directory, for example Docker agents and Kubernetes pods, give the share directory (<mount_root>/<share id>) a shared group with the setgid bit (for examplechgrp <gid>andchmod 2775) so every leaf inherits it. Scion then adds that group to each agent that mounts the leaf; see Agent groups. - Existing upper directories: the expected layout for an upper directory that already exists on the export is setgid, group
nfs.gid, not writable by others, and owned by the broker user ornfs.uid. An existing upper directory that is group- or other-writable or owned by another user, and does not have that layout, is left as it is and logged once per process as a warning. - Cleanup on delete: Deleting a project removes its whole
<subpath_root>/<project id>tree from the export (see cleanup on project delete). Removing a single shared directory declaration deletes that directory. Both are best-effort: failures are logged and never block or roll back the database change. - Startup summary: At startup the server logs one
server.shared_dir_storage resolved layout: …line, plus a warning if any ignorednfsfields are set.
Agent groups on NFS shared directories
Section titled “Agent groups on NFS shared directories”Different kinds of agents can write to the same NFS shared directory (leaf): Docker or rootful Podman agents on brokers, and Kubernetes pods. They usually run with different uids, so each one can modify the others’ files only through the leaf’s group. For that to work:
-
New files must be group-writable. Where the export supports POSIX ACLs, the leaf’s default ACL makes new files group-writable whatever the writer’s umask. Where it does not (for example NFSv4.1 exports, where Linux clients cannot use POSIX ACLs), Scion logs a warning once, and new files follow the writer’s umask. Agents that were given NFS shared-directory groups (below) run with the group bits of their umask cleared (
022becomes002; a stricter077becomes007), so files they create are group-writable without an ACL. Other writers, and agents on images without this support, use their own umask (usually022), and their files cannot be modified by the other kinds of agents. -
Every writer must be in the leaf’s group. At each agent start, the broker reads the group of every NFS shared directory the agent mounts, from the leaf itself, and adds it to the agent:
- Kubernetes: added to the pod’s
supplementalGroups.fsGroupis not changed, and a group equal tofsGroupis not repeated (the pod already holds it). - Docker and rootful Podman: added with
--group-add. The agent image’ssciontoolmust keep these groups when it switches from root to the agent user (for the harness, services, lifecycle hooks and commands run through the substrate exec endpoint); older images drop them and keep the previous behaviour. - Rootless Podman and Apple containers: not supported; the broker logs a warning and starts the agent without the group. Docker with
userns-remapor a rootlessdockerdalso gets no effect from the group, because the leaf gid is not mapped into the container’s user namespace; the broker does not detect this case.
For safety, the broker skips a group (with a warning) when it is below
1000, when it is an overflow id (65534or4294967294, which NFSv4 id mapping reports for unmapped groups), or whennfs.gidis set and does not match. If the group cannot be read, the agent starts without it. Agents without an NFS shared directory are unchanged. On anall_squashexport the server maps every client to one identity, so the added group has no effect there. - Kubernetes: added to the pod’s
With umask 002, other files the agent creates later (for example in its home directory) are group-writable for its primary group too. OpenSSH refuses a group-writable ssh config or key file (such as ~/.ssh/config). Under umask 002, a file created without an explicit mode is group-writable; secrets and ssh tools create such files with mode 0600.
Some files are not upgraded:
- Files created by a writer that passes an explicit restrictive mode (for example
open(..., 0644)) stay non-group-writable; neither an ACL nor the umask can add permissions the creator did not request. - Files copied or moved in with their modes preserved (for example
cp -p,rsync -a,tar -xas the owner, ormvwithin the export) keep those modes. - Leaves created outside Scion, or before Scion added the leaf ACL, keep their existing mode and ACL. Scion only sets modes and ACLs on leaves it creates. Fix them by hand (see the hybrid tier guide).
Known limit, mixed writers: when agents with different uids write to the same nfs directory and neither POSIX ACLs nor the cleared umask group bits apply, subdirectories and files they create follow each writer’s umask, usually 022, and one kind of agent cannot write into subdirectories the other created. Use the shared-group setup described above.
The local backend (or an unset shared_dir_storage) behaves as before.
# Deprecated form; write server.project_storage insteadserver: shared_dir_storage: backend: nfs nfs: mount_root: /mnt/scion-nfs subpath_root: projects shares: - id: shared server: 10.0.0.2 export: /scion-shared pv_name: scion-shared-pvcAgent Home Storage (server.home_storage)
Section titled “Agent Home Storage (server.home_storage)”Selects where the home directory of Kubernetes agents lives. With the default local value the home is inside the pod and is filled from the broker’s copy at every start. With nfs, each agent’s home is a directory on the NFS share of server.project_storage (which must have backend: nfs), kept across stops, restarts and pod replacements.
The nfs value is in development. It takes effect only when the hub’s hub.k8s_nfs_home experiment is on and allow_incomplete_phases is set. The export’s group must be the pod group (gid 1000). See Persistent Agent Home for how the home is created and used.
server.project_storage field |
Former server.home_storage field |
Type | Default | Description |
|---|---|---|---|---|
homes |
backend |
string | local |
local or nfs. nfs needs backend: nfs. |
homes_options.leaf |
leaf |
string | pod |
How an agent’s home directory is created on the export: pod (an init container in the agent’s pod) or broker (the broker, through its own mount of the export at nfs.mount_root). |
homes_options.stop_grace_seconds |
stop_grace_seconds |
int | 30 |
Termination grace period of pods with an NFS home. |
homes_options.termination_wait_seconds |
termination_wait_seconds |
int | 15 |
How long a start waits, beyond the grace period, for the agent’s previous pod to stop. Keep stop_grace_seconds plus this below 90 (the hub’s start window; the broker request timeout is 180); the server warns at startup and scion config validate warns when it is not. |
homes_options.skeleton_max_bytes |
skeleton_max_bytes |
int | 268435456 |
Largest image home skeleton copied into a new home. |
homes_options.allow_incomplete_phases |
allow_incomplete_phases |
bool | false |
Development only. Allows nfs homes while the feature is incomplete. |
server: project_storage: backend: nfs homes: nfs nfs: mount_root: /mnt/scion-nfs shares: - id: shared server: 10.0.0.2 export: /scion-projects pv_name: scion-projects-pvc homes_options: leaf: pod allow_incomplete_phases: true- One choice for the deployment: homes are placed by
server.project_storage.homesalone. The former per-runtime, per-profile and per-instancehome_storage_backendandhome_storage_leafkeys are removed. - Kubernetes only: agents on any other runtime always get a local home, whatever the settings say.
- Share: the home uses the first share and claim of the
server.project_storagenfsblock. - Hub agents only: an NFS home is named after the agent’s hub ID, at
<subpath_root>/<project id>/agents/<agent slug>/home-<agent id>on the export. A start without a hub agent ID fails. - Read from global settings: like the rest of
server.project_storage,homesandhomes_optionsare read from the globalsettings.yamlonly, and a change needs a restart. - Startup summary: the startup log has a warning for each invalid value.
Scheduler (server.scheduler)
Section titled “Scheduler (server.scheduler)”Controls the background task scheduler in the Hub. This regulates the tick interval and concurrency of recurring maintenance tasks (such as telemetry aggregation, session cleanups, and heartbeats) to match database capacity.
| Field | Type | Default | Description |
|---|---|---|---|
interval_seconds |
integer | 60 |
The root ticker interval in seconds. All recurring background tasks fire at multiples of this interval. Increasing this value reduces database connection pressure on smaller deployments. |
max_concurrency |
integer | 2 |
Limits the number of recurring maintenance tasks that can execute concurrently in a single tick. By default, this is capped at 2 to avoid database connection pool saturation. Set to 0 for unlimited concurrency (legacy behavior) or a higher value for larger deployments. |
Maintenance (server.maintenance)
Section titled “Maintenance (server.maintenance)”Controls how the Hub checks for and applies its own updates. The deployment tier sets the update mode: a binary Hub updates from release binaries (it checks the effective channel’s latest installable release in the Hub’s release catalog, read from the GitHub Releases API), while a source Hub rebuilds from its git checkout.
| Field | Type | Default | Description |
|---|---|---|---|
deployment_tier |
string | binary if no repository path is configured, otherwise source |
Update mode. binary updates from GitHub Releases; source updates from a git checkout. The single-node VM deploy script sets binary. |
release_channel |
string | Detected from the running version | Release channel to track: nightly, preview, or stable. When empty, the channel is derived from the version string (v0.5.0 → stable, v0.5.0-preview.1 or v0.5.0-rc1 → preview, nightly-* → nightly). Development builds have no channel and skip scheduled checks. A Hub configured for another channel than the one it runs is offered that channel’s latest release once it was published after the running one (for example, a nightly Hub set to preview). |
update_policy |
string | notify for binary, disabled for source |
notify checks on a schedule and shows an update-available banner in the admin UI, where an admin applies or dismisses it. auto checks on a schedule and installs the channel’s latest release automatically, through the same installer as the admin UI. disabled turns off scheduled checks; manual checks from the admin UI still work. |
check_interval_hours |
integer | 6 |
How often the scheduled release check runs. Minimum 1. Each run is jittered by up to ±30 minutes. |
github_repo |
string | GoogleCloudPlatform/scion |
GitHub repository whose releases the Hub lists and installs. |
install_root |
string | /opt/scion |
Versioned install root (releases/<tag>/scion, current, previous) used by release self-update. SCION_MAINTENANCE_INSTALL_ROOT overrides it. The single-node VM deploy script writes it. |
Scheduled checks run whenever update_policy is not disabled; a Hub in source mode skips them. Before update_policy defaulted to notify, a binary Hub defaulted to auto; set auto explicitly to keep unattended updates.
Release selection. An update that an operator starts from the Maintenance page or the API (an explicit target release, channel and pin) records that choice once the Hub has restarted on the new release and verified it. Auto-updates (update_policy: auto) never record a selection, so they never change which channel the Hub follows. update-status reports where the effective channel came from in channel.source:
selection: the recorded operator choice. It takes precedence overrelease_channel, so a later edit ofrelease_channelhas no effect until the operator picks a channel again on the Maintenance page.config:release_channelis set and no selection was recorded.detected: neither is set, so the channel is derived from the running version.
The pin (update-status pin) is latest when the operator picked the channel’s latest release, or the release tag otherwise; without a selection it is latest. Under auto, a Hub pinned to a release is not updated automatically; it only shows the update-available banner.
A verified Rebuild from Git (rebuild-server) records the update mode git (update-status mode.effective), with no channel or pin: scheduled release checks then skip the Hub, and release_channel applies again once an operator installs a release. Without a selection the mode follows deployment_tier. When a repository path is configured, the Maintenance page offers both modes; the selection changes only when a rebuild or a release update has been verified.
How a release is installed. At startup the Hub detects an installer strategy for its host, shown on the Maintenance page:
symlink: the running binary lives under<install_root>/releases/and<install_root>/currentis a symlink owned by the Hub’s user (the single-node VM layout). The Hub verifies the release, places it inreleases/<tag>/, takes a pre-update SQLite snapshot, switchescurrentatomically (the old release becomesprevious), then exits so that systemd restarts it on the new binary. Restart Hub also restarts this way. Restart Hub is refused (409) while an update or rebuild is running, and holds back new ones until the Hub is back.sudo-install: a repository path is configured and the Hub user has the starter hub’s exact sudoers rules. The verified binary is staged at<repo>/scion.rebuildand installed withsudo -n install, then the service is restarted withsudo -n systemctl restart. No other sudo rights are used.unsupported: anything else. The update and the automatic policy stop before anything is downloaded, and the Maintenance page shows the reason and how to fix it.
Rebuild from Git goes through the same installer. After the installer preflight it fetches and resets the repository checkout, builds the web assets and the binary with the commit and build time stamped in (version --format json reports them), checks that stamp, takes the same pre-update database snapshot as a release update (a failed snapshot installs nothing), and installs the build under the name git-<first 12 hex digits of the commit>: on the symlink layout into releases/git-<sha12>/, switched and retained like a release. The boot-failure guard covers a rebuild on top of a release (the Hub records a preview or stable release’s publish date for previous_safe), but not one whose previous build is itself a Git build (Git to Git, or a release installed on top of a Git build): a commit’s schema cannot be checked against the downgrade floor, so previous_safe is false and a crash loop leaves the Hub stopped for a manual restore (fail closed); with sudo-install through <repo>/scion.rebuild and the starter hub’s sudoers rules, as before. Rebuilding the commit that is already active on the symlink layout is refused before building. The run ends restarting and is completed once the Hub restarts on that build. The systemd unit comparison with the starter-hub template runs for sudo-install only.
Every installed release is downloaded only from the configured repository’s GitHub release, and checked against its SHA256SUMS and its own version output. When Runtime Broker plugins (scion-plugin-<name> regular executables in ~/.scion/plugins/broker, skipping symlinks) are installed on the host, update-binary (and auto updates) also downloads each plugin’s scion-plugin-<name>-linux-<arch>.tar.gz asset from the same release with the same URL pinning and SHA256SUMS verification, swaps the plugin binaries atomically alongside the scion binary, retains copies under <install_root>/releases/<tag>/plugins/ on the symlink layout, and restores the previous plugin binaries if the swap or restart fails or if scion-hub-rollback.service rolls the Hub back at startup (a plugin installed under a non-standard filename is warned about and left untouched; before restarting, the update removes its download directory and marks its .scion-update- plugin staging directory applied once the new builds are installed, and a marked staging directory is removed at the next Hub start and by the next update, while one kept by a failed restore is moved aside under .scion-plugins-kept- for the operator). Both update-binary and rebuild-server also append a ==> Settings check warning section to the operation output when settings.yaml has operator warnings (such as unset server.trusted_proxies), without failing the update. Before switching binaries, a SQLite Hub snapshots its database (VACUUM INTO, producing hub-<from-version>-<timestamp>.db, requiring at least 2× the database size in free disk space) and settings.yaml (settings-<timestamp>.yaml) into <install_root>/snapshots/ (or update-snapshots/ next to the database when install_root is unset) and keeps the five most recent snapshots. The Hub records the pending update in <install_root>/update-state.json before it restarts and marks verified: true on the next start only after the Hub API listener binds. On single-node VMs with the boot-failure rollback guard (scion-hub-rollback.service), a newly switched release that crashes repeatedly before reaching verified: true is rolled back to previous, written to /opt/scion/rollback.marker, and reported on last_update with status rolled_back (the guard swaps binaries only and leaves the Hub stopped if the new release already migrated the database past what previous supports). See Binary layout and updates for the layout and for converting an existing VM.
Confirmed downgrades. Installing an older release than the running one (update-binary when the target release is classified as a downgrade) requires explicit confirmation:
- Pass
params.confirm_downgradeequal to the resolved target release tag (v…ornightly-YYYYMMDD). Without it,POST /api/v1/admin/maintenance/operations/update-binary/runis refused before any run record is created with409 downgrade_requires_confirmation, returning the resolvedtarget_version,current_version,downgrade_floor, andriskslist inerror.details(which also notes any installed Runtime Broker plugin that has no asset orSHA256SUMSline in the target release and would therefore stay at its current version). - Targets older than the downgrade floor (
nightly-20261112) are classified asbelow-floorinGET /api/v1/admin/maintenance/releasesand refused before download withbelow_downgrade_floor. - Downgrades are supported only on SQLite (where the Hub takes a pre-update snapshot); non-SQLite stores refuse a downgrade before download with
downgrade_unsupported_driver.
On the preview channel, a weekly workflow (Fridays at 7:00 AM Pacific: 15:00 UTC in winter, 14:00 UTC in summer) publishes the next vX.Y.Z-preview.N build. It fast-forwards the newest release/vX.Y branch to the newest nightly-YYYYMMDD tag that the branch does not yet contain and whose Build & Test check succeeded, tags it, and builds the release, which updates LATEST.json and the preview branch. A week with no such nightly publishes no preview. If the release branch has commits that are not on main (for example, fixes cherry-picked during a freeze before stable), the weekly run fails instead of force-pushing. Maintainers then tag previews by hand until the line is promoted to vX.Y.Z on stable or the branch is realigned.
The related admin endpoints are GET /api/v1/admin/maintenance/update-status (running version, update mode, effective channel with its channel.source (selection, config or detected, see Release selection above) and pin, downgrade: {floor, snapshotsSupported}, installer, and the pending and last update), GET /api/v1/admin/maintenance/releases?refresh=0|1 (the release catalog per channel, each release classified against the running version including downgrade and below-floor; refresh is limited to one GitHub fetch a minute and the list is otherwise cached for 15 minutes), POST /api/v1/admin/maintenance/check-updates (manual check; ?mode=release checks the release catalog and ?mode=git lists the commits on origin/<repo_branch> that the running build lacks, comparing the checkout’s HEAD instead (reported as compared: "checkout") when the checkout does not contain the running commit; otherwise deployment_tier decides) and GET / DELETE on /api/v1/admin/maintenance/update-available (read or dismiss a pending update notification). All require the hub.maintenance.execute permission.
server: maintenance: deployment_tier: "binary" release_channel: "stable" update_policy: "notify" check_interval_hours: 12OIDC Identity Provider (server.oidc)
Section titled “OIDC Identity Provider (server.oidc)”Configuration for the Hub’s built-in OIDC Identity Provider feature.
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable the OIDC Identity Provider endpoints. |
issuer_url |
string | The public issuer URL of this Hub. If empty, the hub public URL is used. | |
token_lifetime |
duration | "15m" |
Validity duration for minted OIDC identity tokens (e.g. "15m", "1h"). |
user_token_audiences |
list | [] |
Peer hub issuer URLs for which this hub issues user access credentials to its own users (POST /api/v1/auth/federation-token, scion hub auth federation-token). Empty means none are issued. Experimental: also needs the federation.project_visitors experiment. Read at startup; a change needs a restart. |
user_token_lifetime |
duration | "15m" |
Validity of those user access credentials; a whole number of seconds, at most "1h". Read at startup. |
OIDC Federation (server.federation)
Section titled “OIDC Federation (server.federation)”Configuration for inbound OIDC-based federation authentication.
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable OIDC federation authentication. |
trusted_issuers |
list of objects | [] |
List of trusted OIDC issuers (see below). |
algorithms |
list of strings | ["RS256"] |
Supported cryptographic signing algorithms. |
cache.refresh_interval |
duration | "1h" |
How often to refresh cached issuer public keys (JWKS). |
cache.debounce_interval |
duration | "1s" |
Min interval between JWKS reload attempts to prevent DDOS. |
Trusted Issuer Settings (server.federation.trusted_issuers[])
Section titled “Trusted Issuer Settings (server.federation.trusted_issuers[])”| Field | Type | Default | Description |
|---|---|---|---|
issuer_url |
string | MANDATORY. The exact OIDC issuer URL (matching token iss claim). |
|
jwks_url |
string | The URL to fetch signing public keys. Discovered via OIDC discovery if empty. | |
expected_audience |
string | The expected audience aud claim in tokens. Required (non-empty) for a user-type Google issuer to enable external bearer tokens. |
|
allowed_projects |
list of strings | If set, restricts tokens to specific project UUIDs. | |
allowed_root_users |
list of strings | If set, restricts tokens to specific root user emails. | |
default_scopes |
list of strings | Default JWT scopes granted to federated agents. | |
issuer_type |
string | "hub" |
Type of issuer: "hub", "service_account", or "user". |
default_role |
string | "viewer" |
Deprecated. Accepted for compatibility and ignored; the Admin > Federation page no longer shows it. Project access for federated users comes from visitor grants (allow_visitor_grants). |
allowed_emails |
list of strings | Restrict user tokens to specific email claims (supports wildcards e.g. *@example.com). |
|
allowed_domains |
list of strings | user issuer: restrict user principals on both the OIDC federation token path and, for a Google issuer, the external bearer token path to these email domains (exact, case-insensitive, no wildcards or subdomain matching; the token’s email must be verified). The Hub sign-in policy still applies on the external bearer path. Never consulted for service accounts. Hub issuer with accept_user_tokens: restrict that hub’s user access credentials to these email domains, with the same matching. Not allowed on any other issuer. |
|
accept_user_tokens |
boolean | false |
issuer_type: hub only. Accept user access credentials (scion_token_kind: user) issued by this peer Hub. Such a credential must carry email with email_verified: true and no agent claims (project_id, agent_name, ancestry, root_user), and must pass allowed_emails and allowed_domains. Read on every request: a change applies to the next request. Requires the federation.project_visitors experiment; while it is off, these credentials are refused. Agent identity tokens from the issuer are not affected. |
allowed_gcp_projects |
list of strings | Google issuer only. Admit service-account principals whose GCP project ID (parsed from the service account email) is listed. Empty admits no service accounts. Distinct from allowed_projects, which matches Scion project IDs. |
|
allow_visitor_grants |
bool | false |
issuer_type user or hub only. Lets holders of project.manage_visitors (project owners and admins, hub admins) grant visitor access to users of this issuer, by email, while the federation.project_visitors experiment is on; granting also needs every permission of the project-visitor role on the project. The email must also pass allowed_emails and allowed_domains. A granted user’s first request from this issuer with that email (present, and verified where the token carries email_verified) links the grant to the token’s subject. When false, the Hub refuses visitor grants for the issuer, links no new visitor, and refuses the issuer’s linked visitors on their next request (403). Their grants are kept, so setting it back to true restores access with no grant change. In the web UI it is the switch Allow project owners to grant visitor access to users from this issuer on Admin > Federation. |
OIDC Login (server.oidc_login)
Section titled “OIDC Login (server.oidc_login)”Configuration for an external OIDC provider used for Web UI user login.
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable the external OIDC login provider. |
display_name |
string | The human-readable label shown on the login button (e.g. "Corporate SSO"). |
|
issuer_url |
string | The exact OIDC issuer URL. Used to perform discovery via {issuer_url}/.well-known/openid-configuration. |
|
client_id |
string | The OAuth2/OIDC Client ID. | |
client_secret |
string | The OAuth2/OIDC Client Secret (can be empty for public OIDC clients). | |
scopes |
list of strings | ["openid", "email", "profile"] |
Overrides the default scopes requested during login. |
Project Defaults (project_defaults)
Section titled “Project Defaults (project_defaults)”Configuration for project-level default behaviors across the Hub. Unlike most other server configurations, project_defaults is declared as a top-level section in settings.yaml (outside of the server: block).
| Field | Type | Default | Description |
|---|---|---|---|
default_scratchpad |
bool | true |
If enabled, automatically provisions a default scratchpad shared directory when a new project is created. |
Example:
# Declared at the top level of settings.yamlproject_defaults: default_scratchpad: trueEnvironment Variables
Section titled “Environment Variables”Most server settings can be overridden via environment variables using the SCION_SERVER_ prefix. Write each path segment in upper case and drop the underscores inside a multi-word field name: read_timeout becomes READTIMEOUT, not READ_TIMEOUT. A name that the Hub does not recognise is ignored and logged as a warning at startup, with a suggested spelling where one exists. An exported but empty SCION_SERVER_* or SCION_SEED_* variable is treated as unset, so it never overrides a configured value. Each key’s working variable is listed as x-env-var in the settings schema.
There are two exceptions to the pattern:
- Every setting under
server.brokeruses theRUNTIMEBROKERsegment, for exampleserver.broker.port->SCION_SERVER_RUNTIMEBROKER_PORTandserver.broker.broker_id->SCION_SERVER_RUNTIMEBROKER_BROKERID. server.hub.public_urlis set bySCION_SERVER_HUB_ENDPOINT.SCION_SERVER_HUB_PUBLICURLandSCION_SERVER_HUB_PUBLIC_URLset nothing.
Other spellings of a key are accepted as aliases and set the same key: the spelling that keeps a field’s underscores (SCION_SERVER_HUB_ADMIN_EMAILS), the broker names under the BROKER segment (SCION_SERVER_BROKER_PORT, SCION_SERVER_BROKER_BROKER_ID) and the flattened CORS names (SCION_SERVER_HUB_CORSENABLED). Prefer the names above. When a key’s name and one of its aliases are both set, the name above wins; when only aliases are set, the alias that sorts first wins. At startup the server logs a warning for each alias that is set, naming the variable to use instead, and says so when another spelling of the same key overrides it. Aliases will be removed after the next release.
When it starts, the server checks every SCION_SERVER_* and SCION_SEED_* variable it reads for a boolean, integer or number setting. It does not check a SCION_SEED_* variable of a setting that seeding does not seed, or an alias that another spelling of the same key overrides, since nothing reads those values. List values, such as the port list SCION_SERVER_HUB_CONDUIT_TCPALLOWEDPORTS, are not checked at this step. A boolean takes true/false, 1/0 or t/f, so SCION_SERVER_HUB_AUTOSUSPENDSTALLED=false turns the setting off. A value that does not parse, for example SCION_SERVER_HUB_AUTOSUSPENDSTALLED=maybe, stops the server with an error that names the variable and its setting (never the value). SCION_SERVER_ADMIN_MODE is checked with the spellings listed under Tracked Boolean Variables. An empty value is unset and is not checked.
server.log_format and server.env have no environment variable. Neither is read by the Hub: both are accepted so existing settings files still load. The log output format is chosen at startup; set SCION_LOG_GCP=true for Cloud Logging JSON. server.log_level (or SCION_SERVER_LOGLEVEL) is applied when the server starts, and again when a file-mode admin server-config save or reload re-reads the config; clearing it reverts to info. --debug and SCION_LOG_LEVEL take precedence over it (see Precedence). Upgrade note: a Hub whose settings still contain server.log_level: debug from an earlier change now starts at debug, because earlier releases ignored the setting at startup.
server.keyring_file (--keyring-file, SCION_SERVER_KEYRING_FILE) is an experimental signing-key pilot. It is the path of a version-1 JSON keyring file holding versioned signing and encryption keys. Mount the file through your secret delivery mechanism; only the path belongs in settings. The hub and web server process reads it once at startup, and --role relay never reads it. Startup is refused when:
- the
server.secret_rotationexperiment is off; --session-secret,SCION_SERVER_SESSION_SECRETorSESSION_SECRETis also set;- the file is invalid (unknown fields, duplicate key IDs, more than three entries per ring, inconsistent deadlines or key lengths);
- the file is writable or executable by others, or readable by others without
server.keyring_file_allow_world_readable; signing.emitislegacyand the active signing key is not listed insigning.legacy_read_ids(it would issue values without a key ID that no reader accepts);encryption.write_formatisv1and the active encryption key is not listed inencryption.legacy_v1_read_ids;- the local cipher cannot satisfy the database encryption policy (when a policy row is present in the database, the Hub must run with
server.keyring_file, writeenc:v2:under an allowed writer key ID, and hold every required reader key ID; legacy-mode startup without a keyring file is refused).
In this release web session cookies, signed skill file download URLs, artifact view URLs, list pagination cursors, attachment delivery URLs and port page links (the hub.port_view experiment) use the keyring’s signing ring. The active generation signs new cookies, and cookies signed by any generation in the file are accepted until that generation’s verify_until. Download URLs, view URLs, cursors, delivery URLs and page links are signed by the active generation. One that carries no key ID (the only form this release writes) is accepted when a generation listed in signing.legacy_read_ids signed it; one that carries a key ID is accepted only from that generation. Either way, acceptance ends at that generation’s verify_until. An unknown key ID is refused. Once the active generation reaches its sign_until, the server stops issuing them: skill file URLs are handed out unsigned, so brokers cannot download local-storage skill files, new artifact views and share-link views answer 503, a list page that needs a continuation cursor answers 500, attachment deliveries get no download URL, and opening an exposed port from the hub serves it through the port’s own link instead of a page link. /readyz also reports signing_issuance_closed. A cursor has no lifetime of its own: it stops working when its generation reaches verify_until, and the client must restart the listing from the first page. Conduit relay admission tokens, and relay-peer request signatures when server.hub.conduit.peer_secret is unset, also use the signing ring: the active generation issues them (including the key ID when signing.emit is "versioned", as ca3.<kid>.<sealed> on admission tokens and X-Conduit-Peer-Key-Id with X-Conduit-Peer-Signature-Version: 2 on relay-peer signatures, and omitting the key ID when signing.emit is "legacy"), a value carrying a key ID is accepted only from that generation (before its verify_until), and a value without a key ID is accepted only from a generation listed in signing.legacy_read_ids (before its verify_until), so list every generation any replica issues without a key ID there; issuance stops once the active generation reaches its sign_until. A dedicated peer_secret is unaffected by the keyring. The local secret backend, Hub-scoped secrets stored without an external secret backend (such as the GitHub App private key and webhook secret and chat integration tokens and signing secrets), and all Hub-stored infrastructure backups in the database (signing keys for agent and user tokens, skill downloads, artifact views, attachment delivery, list cursors, and port page links; the OIDC signing key and oidc_signing_keyset; and the Conduit grant key ring) share the Hub’s versioned value cipher (enc:v1: or enc:v2:<kid>:), keyed from the keyring’s encryption ring when server.keyring_file is set (or from the legacy key otherwise; without a keyring file, Hub-scoped integration secrets stay plaintext for rollback compatibility until rewrapped, and plaintext rows are refused once a database encryption policy exists). At startup, before any key synchronization or backup write, a startup encryption audit scans and authenticates all locally stored secret payloads (skipping reference-only rows), classifies audit.by_format and audit.by_kid by the key that actually authenticated each payload, and validates the decrypted structure of Hub-scoped infrastructure backups. Unreadable records are preserved and never overwritten: in keyring mode, any unreadable infrastructure backup keeps the Hub not ready (/readyz returns 503 with reason: "backup_unreadable"), while in legacy mode the Hub logs an UNHEALTHY error naming the affected record IDs and continues serving without overwriting them. Inspect a Hub replica’s active and accepted key IDs, encryption policy, startup audit counts, and readiness reasons with GET /api/v1/admin/keys/status (behind the server.secret_rotation experiment) or on the server host with scion server keys status, scion server keys check, and scion server keys verify (see GET /keys/status and scion server). The admin API never returns or accepts this setting.
The Hub’s agent and user JWTs (access, refresh, CLI and test-login tokens) also use the keyring’s signing ring. The active generation signs new JWTs (setting the JWT protected kid header to the active generation’s key ID when signing.emit is "versioned" or server.jwt.emit_version is "keyring", and omitting kid when emit is "legacy"). A JWT is accepted only by the generation its key ID names, or, for a JWT without a key ID, by a generation listed in legacy_read_ids (while server.jwt.accept_legacy is true), each until its verify_until. An unknown key ID never falls back to another key. Once the active generation reaches its sign_until, the server stops issuing JWTs. Setting server.jwt.emit_version: "keyring" refuses startup when server.keyring_file is not configured or no usable file keyring was loaded, when the signing ring has no active generation, or when an explicit agent or user JWT signing key is set. With a keyring file every signing purpose uses the signing ring’s generations (SharedSigningSecret is left unset so no consumer reads a legacy signing entry outside the ring), and the Hub does not sign or verify JWTs with any stored agent_signing_key or user_signing_key record, and neither rewrites, republishes nor deletes those records. The startup encryption audit described above still decrypts and checks them, so an unreadable one keeps the Hub not ready (backup_unreadable) even though no JWT uses it. If the file is missing, unreadable or incomplete, startup is refused rather than falling back to those records. While a legacy entry is present in the signing ring and listed in legacy_read_ids, its JWT keys equal the keys earlier releases derived from the same session secret, so tokens issued before the keyring file keep validating; once legacy has been retired from the keyring file, the Hub starts and signs without a legacy entry.
Without server.keyring_file (legacy mode), the Hub holds one key, which is the keyring’s legacy generation. It accepts the values it issues (which carry no key ID) and, so that a rollback from a keyring file whose active generation is legacy keeps working, the versioned values that carry the key ID legacy: enc:v2:legacy: stored secrets, ad2.legacy. attachment delivery URLs, c2.legacy. list cursors and ca3.legacy. Conduit relay admission tokens. A versioned value that carries any other key ID is refused, even when it was produced with the same key. Agent and user JWTs are accepted in legacy mode only when the protected kid header is absent or is legacy; a JWT with any other kid is refused, even when its signature verifies.
File permissions: mount the keyring file with mode 0400 (0440 when a group must read it), for example Kubernetes defaultMode: 0400. A world-readable keyring file (such as the Kubernetes Secret volume default 0644, or 0444) refuses startup unless the operator sets server.keyring_file_allow_world_readable (--keyring-file-allow-world-readable, SCION_SERVER_KEYRING_FILE_ALLOW_WORLD_READABLE; default false), which loads it with a warning. A file writable or executable by others is always refused, and a group-readable file loads with a warning. The override is operator-only: the admin API never returns or accepts it.
Examples:
server.hub.port->SCION_SERVER_HUB_PORTserver.hub.gcp_project_id->SCION_SERVER_HUB_GCPPROJECTIDserver.hub.gcp_iam_check_mode->SCION_SERVER_HUB_GCPIAMCHECKMODEserver.hub.gcp_iam_deny_unknown_policy->SCION_SERVER_HUB_GCPIAMDENYUNKNOWNPOLICYserver.hub.admin_emails->SCION_SERVER_HUB_ADMINEMAILSserver.hub.stalled_threshold->SCION_SERVER_HUB_STALLEDTHRESHOLDserver.hub.content_url->SCION_SERVER_HUB_CONTENTURLserver.hub.content_url_allow_same_site->SCION_SERVER_HUB_CONTENTURLALLOWSAMESITEserver.hub.monitoring_dashboard_url->SCION_SERVER_HUB_MONITORINGDASHBOARDURLserver.hub.observability.metrics_enabled->SCION_SERVER_HUB_OBSERVABILITY_METRICSENABLEDserver.hub.observability.tracing_enabled->SCION_SERVER_HUB_OBSERVABILITY_TRACINGENABLEDserver.auth.user_access_mode->SCION_SERVER_AUTH_USERACCESSMODEserver.broker.enabled->SCION_SERVER_RUNTIMEBROKER_ENABLEDserver.broker.container_hub_endpoint->SCION_SERVER_RUNTIMEBROKER_CONTAINERHUBENDPOINTserver.broker.state_dir->SCION_SERVER_RUNTIMEBROKER_STATEDIRserver.broker.broker_id->SCION_SERVER_RUNTIMEBROKER_BROKERIDserver.database.url->SCION_SERVER_DATABASE_URLserver.database.allow_shadowed_schema->SCION_SERVER_DATABASE_ALLOWSHADOWEDSCHEMAserver.auth.dev_mode->SCION_SERVER_AUTH_DEVMODEserver.secrets.backend->SCION_SERVER_SECRETS_BACKENDserver.secrets.gcp_project_id->SCION_SERVER_SECRETS_GCPPROJECTIDserver.secrets.gcp_credentials->SCION_SERVER_SECRETS_GCPCREDENTIALSserver.secrets.gcp_replication_locations->SCION_SERVER_SECRETS_GCPREPLICATIONLOCATIONSserver.scheduler.interval_seconds->SCION_SERVER_SCHEDULER_INTERVALSECONDSserver.scheduler.max_concurrency->SCION_SERVER_SCHEDULER_MAXCONCURRENCY
A value in settings.yaml, or in a SCION_* environment variable other than the SCION_SERVER_* and SCION_SEED_* variables the server checks at startup (see above), that does not parse for its key (for example a string where a number is expected) is ignored for that key only. The key falls back to its default, or stays unset, even when the bad value came from the environment and overrode a valid file value. A warning naming the key (never the value) is logged once. Every other key still applies, so one bad value no longer turns off unrelated Hub features. The server’s own configuration is not covered: a value in the server section of settings.yaml, or a SCION_SERVER_* variable, that does not parse can make the server ignore the whole server section, at startup or when it reloads settings.
Logging Environment Variables
Section titled “Logging Environment Variables”These environment variables control server-side logging behavior. They are not part of the settings.yaml structure.
| Variable | Description | Default |
|---|---|---|
SCION_LOG_GCP |
Enable GCP Cloud Logging JSON format on stdout | false |
SCION_LOG_LEVEL |
Log level: debug, info, warn or error, optionally followed by per-component levels such as info,hub.auth=debug. See Controlling the Log Level. |
info |
SCION_DEBUG |
Deprecated alias for SCION_LOG_LEVEL=debug. Any non-empty value enables it, and a warning is printed to stderr once. SCION_LOG_LEVEL wins if both are set. Ignored (no warning) by scion commands in agent CLI mode; see Debugging an agent. |
- |
SCION_CLOUD_LOGGING |
Send logs directly to Cloud Logging via client library | false |
SCION_CLOUD_LOGGING_LOG_ID |
Log name in Cloud Logging for application logs | scion-server |
SCION_GCP_PROJECT_ID |
GCP project ID for Cloud Logging (priority 1) | auto-detect |
GOOGLE_CLOUD_PROJECT |
GCP project ID for Cloud Logging (priority 2) | - |
SCION_SERVER_REQUEST_LOG_PATH |
Write HTTP request logs to a file at this path. Each line is a JSON object in HttpRequest format. When not set, request logs follow the default routing (stdout in background mode, suppressed in foreground mode, Cloud Logging when enabled). |
(disabled) |
See the Local Development Logging guide for details on log formats, request log fields, and Cloud Logging integration.
Shutdown and Signals
Section titled “Shutdown and Signals”Once its startup has installed signal handling, scion server start (the Hub, Web and co-located broker process) shuts down gracefully on either SIGINT (Ctrl+C) or SIGTERM (systemctl stop, Kubernetes, Cloud Run, container stop): from the first signal, /readyz returns 503 (status: not_ready, reason: shutting down), new requests other than health checks and Runtime Brokers’ own requests are refused with a retryable 503 (Retry-After), SSE streams end, the Conduit relay (or ingress proxy) drains, the co-located Runtime Broker and any flat Runtime Broker instances drain their in-flight launches while the in-process Hub still accepts their launch reports, and the Hub then closes its listener, drains in-flight HTTP requests, closes the audit writer, and flushes and closes its logs (see shutdown_grace and Shutdown under Log delivery guarantees for the full order and budgets). A second SIGINT or SIGTERM forces the shutdown: the server stops waiting for the drain, closes the audit writer, flushes its logs within a short bound and exits with status 1. An orchestrator that escalates to SIGKILL after its grace period also terminates the process immediately. To stop the server cleanly, send SIGINT or SIGTERM, or use scion server stop.
- Early startup. A
SIGINTorSIGTERMthat arrives during early startup, before signal handling is installed, normally ends the process without a drain, as before. If the server installs its signal handling while such a signal is still being handled, it may instead run the graceful shutdown. - Foreground runtime broker (
scion broker start --foreground). When the broker owns its port record (the normal case), it saves the record and then installs its ownSIGTERMhandler. Once that handler is installed, aSIGTERMremoves the port record; the process then either ends (if the server has not installed its signal handling yet) or runs the single graceful shutdown. The record removal and the start of the graceful shutdown run concurrently, with no order between them. ASIGTERMthat arrives after the record is saved but before the broker’s handler is installed ends the process by default and can leave the record behind. On a normal exit the broker also removes its record. - Other signals. Signals the server does not handle keep their default behavior. For other termination signals, for example
SIGHUPorSIGQUIT, that ends the process without a drain.
If the Hub fails to start (for example, when operational settings cannot be loaded), the server logs Hub server failed to start: <error>, flushes its logs, and exits with status 1.
Log delivery guarantees
Section titled “Log delivery guarantees”Server logging is best effort. Losses in the modes below are counted, never retried, and there is no exactly-once delivery: a record can be lost, and on some platforms it can appear twice (see Duplicates below). This section describes what the code guarantees; nothing here is a measured ingestion rate.
Local acceptance is not remote ingestion. A record counts as written when the local handler accepts it: the stdout write returned, or the Cloud Logging client put the entry in its in-memory buffer (8 MiB for application logs). The client sends buffered entries to Cloud Logging later, in the background. Nothing in the Hub confirms that an entry reached Cloud Logging, so no metric reports a delivery ratio for the Cloud path.
Paths.
- stdout (plain JSON, or Cloud Logging JSON with
SCION_LOG_GCP=true): written synchronously by the logging call. On Cloud Run and GKE the platform ships stdout to Cloud Logging; on other hosts that depends on the host’s log agent (for example the Ops Agent on Compute Engine). That shipping is outside the Hub. - Direct Cloud Logging (
SCION_CLOUD_LOGGING=true): the client library buffers and sends entries asynchronously. A circuit breaker probes the client by flushing it every 30 seconds; after 3 consecutive failed flushes it opens and the Hub stops feeding the Cloud path (stdout logging continues where a stdout handler is configured; see thecircuit_openrow). It probes again at the first 30-second check at least 60 seconds after opening (about 90 seconds with the defaults, and about every 90 seconds while probes keep failing) and closes when a probe succeeds. The request log (scion_request_log), message log (scion-messages) and artifact decision log (scion-artifacts) share the same client and circuit. - Decision-log audit records (
scion.audit, only when thehub.authorization_decision_audit_v2experiment is on): queued on a bounded asynchronous writer (2,048 records, 2 MiB) with one worker, a 2-second write budget per record and a 5-second drain at shutdown. The worker writes to the same handler chain as other logs, so the stdout and Cloud rules above apply after it.
Loss modes. Each loss mode below increments scion.logging.write.failures{writer, reason}. Losses outside these modes are not counted, for example: a process ended by a termination signal it does not handle (for example SIGINT or SIGTERM before scion server start installs its signal handling, or SIGHUP or SIGQUIT at any time; see Shutdown), a process killed by SIGKILL or the OOM killer, an unrecovered panic, a failed stdout write outside the audit writer, an OTel log export failure, and a request-log file write failure.
writer |
reason |
Meaning |
|---|---|---|
audit |
queue_full |
The writer’s queue was full (record count or bytes); the new record was dropped. |
audit |
oversize |
The record exceeded the per-record limits; it was rejected, not truncated. |
audit |
unsupported |
The record held a value kind the writer does not copy; it was rejected. |
audit |
closed |
The record arrived after the writer closed (shutdown). |
audit |
error |
The inner handler returned an error or panicked. |
audit |
timeout |
The write exceeded its 2-second budget. The record may still be written later; it is counted once, as a timeout. |
audit |
shutdown |
The record was still queued when the 5-second shutdown drain ended. |
cloud |
error |
The Cloud Logging client reported an error (a failed batch send, an invalid or oversized entry). One count per reported error, not per record. Under an error storm the client skips some reports, so this undercounts. |
cloud |
queue_full |
The client’s buffer was full and it dropped the entry. |
cloud |
circuit_open |
The record was dropped from the Cloud path because the circuit breaker was open or probing. Application and message logs still go to stdout, except on Cloud Run, where the Hub leaves out their stdout handler (see Duplicates). The request log has no stdout copy while Cloud Logging is on; it keeps a file copy only if SCION_SERVER_REQUEST_LOG_PATH is set. Otherwise the dropped record is lost from Cloud Logging and stdout (an OTel log copy, when OTel logging is enabled, is unaffected). |
cloud |
flush_error |
A periodic, probe or shutdown flush failed, timed out, or was skipped because an earlier flush was still running. A failed flush usually reports errors the client already counted as error, so the two can count the same incident. |
scion.logging.write.records{writer="audit"} counts records the audit writer’s handler accepted. There is no records series for writer="cloud", because a client buffer accept is not ingestion. Cloud Logging client errors are counted in scion.logging.write.failures{writer=cloud,reason=error|queue_full} and printed once to stderr as logging client: <error>. They are deliberately not sent through the structured logger (stdout JSON, OTel or Cloud Logging), to avoid feeding errors back into a failing Cloud Logging pipeline. Under an error storm the client drops some errors before the hook, so the count may be low.
State signals.
scion.logging.writer.stalled{writer}is 1 while an audit write has been in flight longer than its budget. Records keep queuing, then drop asqueue_full.scion.logging.writer.circuit_open{writer="cloud"}is 1 while the circuit breaker is open or probing.scion.logging.queue.depth{writer}andscion.logging.write.late_returns{writer}(writes that finished after being counted as timeouts; not failures)./healthzkeyaudit_log_writer:healthy,degraded: recent write failures(a failure in the last 5 minutes),degraded: writer stalledordegraded: writer closed./healthzkeycloud_logging, present only when direct Cloud Logging is configured:healthyordegraded: <reason> (<counts>). The reason iscircuit open(the circuit breaker is open or probing),consecutive flush failures(3 flushes in a row failed, the latest in about the last 5 minutes) orwrite failure ratio above 1%(errorplusqueue_fullexceed 1% of the records handed to the client in about the last 5 minutes, once at least 500 records were handed over). A single failed write is counted but does not degrade the key. The counts cover about the last 5 minutes (ten 30-second slots, so 4.5 to 5 minutes) and hold only numbers.errorcounts client error reports, not records: one failed batch is one report. Example:degraded: write failure ratio above 1% (window=5m0s writes=1200 error=9 queue_full=6 flush_error=0 circuit_open=0 ratio=1.25%). The ratio shown is rounded up to a whole 0.01%.
Both health keys are non-critical: they can make /healthz report degraded, never unhealthy, and they do not affect /readyz. The metrics are exported only when Hub metrics export is on (server.hub.observability.metrics_enabled, or server.hub.gcp_project_id while that switch is unset; metric group SCION_METRICS_LOGGING); the counters and health keys work without it. deploy/monitoring/alert-policies.yaml defines WARNING policies for write failures, a stalled writer and an open circuit. They take effect only when applied to Cloud Monitoring.
Shutdown. On a graceful shutdown (which signals trigger it, and when, is described in Shutdown and Signals) the Hub drains its servers, closes the audit writer (up to 5 seconds), flushes and shuts down metrics and traces, then flushes the message and request logs, stops the circuit breaker, and flushes and closes the Cloud Logging client. Because the metric exporter has already shut down, failures counted during these final log flushes are never exported; a failed final flush of the main Cloud log still prints error flushing Cloud Logging: ... to stderr. A process that ends without a graceful shutdown skips these steps, for example: a SIGINT or SIGTERM during early startup, another termination signal the server does not handle (for example SIGHUP or SIGQUIT), or SIGKILL. Queued audit records and buffered Cloud Logging entries are then lost and not counted. A Hub startup failure (Hub server failed to start) returns an error instead of exiting immediately, so the deferred log flush runs before the process exits with status 1. Known third-party hazard (pre-existing, not observed): in the Cloud Logging client (cloud.google.com/go/logging v1.13.2), a record logged after the client is closed whose send then fails reports the error on a closed channel, which panics; whether this can happen during Hub shutdown has not been determined.
Duplicates. When direct Cloud Logging is on and the K_SERVICE environment variable is set (Cloud Run), the Hub leaves out the stdout handler for application logs and the message log, so it sends each of those records on one path only (the direct client) and does not itself create a duplicate. Without K_SERVICE (GKE, Compute Engine, other hosts) both paths stay on. If that platform also ships stdout to Cloud Logging, each record can appear twice, once per path. This follows from the code; it has not been observed on a deployment. To avoid it there, turn off one of the two paths: direct Cloud Logging, or the platform’s stdout collection.
Boolean Environment Variable Parsing (parseBoolEnv)
Section titled “Boolean Environment Variable Parsing (parseBoolEnv)”The server reads the variables listed under Tracked Boolean Variables directly, with the parseBoolEnv parser. These rules apply only to them; a SCION_SERVER_* or SCION_SEED_* variable for a boolean setting takes only true/false, 1/0 or t/f, and a value that does not parse stops the server at startup (see Environment Variables).
- Supported Truthy Values:
true,1,t,yes,y,on(case-insensitive, whitespace-trimmed). - Supported Falsy Values:
false,0,f,no,n,off(case-insensitive, whitespace-trimmed). - Safety Warnings: Unset or empty values are
false, except forSCION_SERVER_ADMIN_MODE: when it is unset or empty, the server keeps the admin mode from its settings (a value of only whitespace turns admin mode off). An unrecognized non-empty value (e.g.,SCION_LOG_GCP=trur) triggers a warning at startup and is treated asfalse, except forSCION_SERVER_ADMIN_MODE, where it stops the server at startup.
Tracked Boolean Variables:
Section titled “Tracked Boolean Variables:”| Variable | Description | Default |
|---|---|---|
SCION_SERVER_ADMIN_MODE |
Forces the server into emergency maintenance mode (break-glass removal). An unrecognized non-empty value stops the server at startup instead of defaulting to false. |
false |
SCION_TRACING_ENABLED |
Enables Hub trace export when server.hub.gcp_project_id is set and server.hub.observability.tracing_enabled is unset. Ignored when that switch is set. |
false |
SCION_LOG_GCP |
Enables Google Cloud Logging JSON format on standard output. | false |
SCION_REQUIRE_STABLE_SIGNING_KEY |
Demands a persistent session/JWT signing key. When enabled, startup aborts (fail-closed) if no stable key/secret can be resolved in hosted mode, preventing JWT signature mismatches across replica restarts. | false |
Hub Endpoint Resolution
Section titled “Hub Endpoint Resolution”The operator-configured Hub endpoint is resolved from flags, the environment and settings.yaml, in that order. When no operator setting pins it (--base-url, SCION_SERVER_HUB_ENDPOINT or SCION_SERVER_BASE_URL), a server.hub.public_url saved in the database through the admin API replaces it at startup; see server.hub.public_url: environment first. A server.hub.public_url in settings.yaml does not pin the endpoint.
On a server that runs the Hub, the operator-configured endpoint is resolved in this order:
- The
--base-urlflag — the server’s public base URL (also used for OAuth redirects). SCION_SERVER_HUB_ENDPOINT— explicit Hub public URL.SCION_SERVER_BASE_URL— the server’s public base URL (also used for OAuth redirects).server.hub.public_urlinsettings.yaml.hub.endpoint(orSCION_HUB_ENDPOINT) — deprecated, see below.- IAP Audience Derivation (in Hosted HA mode with IAP authentication):
- For Cloud Run IAP audiences (
/projects/<number>/locations/<region>/services/<service>), Scion can auto-derive the Hub’s URL using the legacy Cloud Run URL format (https://<service>-<number>.<region>.run.app). Newer Cloud Run services use a different URL format (https://<service>-<hash>-<region>.a.run.app) where the hash cannot be derived from the project number — for those services, setSCION_SERVER_BASE_URLexplicitly instead of relying on auto-derivation. The derived URL is the IAP front end, which the Hub’s own host does not serve. On a co-located broker (for example, the single-node VM deployment), agents dispatched to Docker therefore receivehttp://scion-hub.internal:<hub listen port>instead. That hostname is mapped to the Docker host gateway, so the agents reach the Hub directly while staying on bridge networking. Agents dispatched to Podman receivehttp://host.containers.internal:<port>, which Podman resolves itself. On rootless Podman 5.0 to 5.2 with pasta networking, that name can be missing or unreachable (fixed in Podman 5.3). A host-gateway mapping has the same gap there, so upgrade Podman if agents cannot reach the Hub. This applies whatever the broker’s default runtime is, including Docker or Podman runtime profiles on a Kubernetes-default broker. Agents dispatched to Kubernetes runtime profiles, Cloud Run, or Apple container still receive the derived URL. - For GKE/GCLB backend-service IAP audiences (
/projects/<number>/global/backendServices/<id>), a URL cannot be derived from the ID. IfSCION_SERVER_BASE_URL(or other explicit URL settings) is not set, Scion will log a warning at startup and fall back tolocalhost, which is likely unreachable from dispatched agents.
- For Cloud Run IAP audiences (
- Auto-computed
http://localhost:{port}(last resort).
A server that runs only the Runtime Broker takes server.hub.public_url (or SCION_SERVER_HUB_ENDPOINT), then hub.endpoint; --base-url and SCION_SERVER_BASE_URL do not apply to it.
Upgrade note: earlier releases ranked server.hub.public_url from settings.yaml (and SCION_SERVER_HUB_ENDPOINT) above --base-url and SCION_SERVER_BASE_URL. A server that sets --base-url or SCION_SERVER_BASE_URL to a different URL than its settings.yaml value now uses the flag or environment value, and a server that sets --base-url to a different URL than its SCION_SERVER_HUB_ENDPOINT value now uses the flag value. The values derived from the Hub endpoint follow it, including the OIDC issuer default and the cloudrun_invoker audience default. Hosts set up with the starter-hub scripts (gce-start-hub.sh) always set SCION_SERVER_BASE_URL in the systemd unit, so on those hosts that value now applies even when settings.yaml sets a different server.hub.public_url. Remove the setting you do not want, or make the two agree.
The server reads its startup settings from the global settings directory (~/.scion) only: hub.endpoint, active_profile and profiles, the co-located broker’s default runtime, the image_registry check, the broker identity and auto_provide, the message broker and plugins, notification channels, cloud logging and maintenance settings. The Hub’s system runtime endpoints (/api/v1/system/runtime and the image pull) resolve the active profile and runtime from the global settings directory too. Starting the server from inside a project directory does not merge that project’s settings into these, and --global makes no difference to them. The settings reads made while the server runs (the profiles a Runtime Broker advertises and the Hub’s reincarnate image settings) use the global settings directory as well.
Taking the server’s endpoint from hub.endpoint (or SCION_HUB_ENDPOINT) is deprecated: the server logs a warning at startup, scion doctor reports it, and the fallback is removed in the next release. Set server.hub.public_url or server.broker.hub_endpoint instead. A workstation whose hub.endpoint is its own loopback URL is not affected.
For local development where the Hub runs on localhost but agents are in containers, set server.broker.container_hub_endpoint to a container-accessible address like http://host.containers.internal:8080.
server.broker.container_hub_endpoint is a per-broker setting, unlike server.hub.agent_endpoint, which applies to every broker. When it is set, that broker’s Docker and Podman agents receive it in place of the agent Hub endpoint, including a public https:// URL. Remote brokers that do not set it keep giving their agents the public URL. For example, a broker colocated with the Hub can set http://scion-hub.internal:8080 so its Docker agents reach the Hub over the Docker bridge, and a broker in the Hub’s private network can set an internal https:// address. Agents dispatched to Kubernetes, Cloud Run or Apple container keep the agent Hub endpoint.
Agents refuse to start with a plaintext http:// Hub URL unless its host is a loopback address or a host alias of the agent’s runtime: host.docker.internal or scion-hub.internal on Docker (each mapped to the host gateway; a co-located broker maps scion-hub.internal), and host.containers.internal on Podman. An IP address or any other hostname, including an internal one, needs https://. On Podman, use http://host.containers.internal:<port> rather than scion-hub.internal.
When the setting is unset, the co-located broker computes a value itself (the public domain when this host serves it, or a host-gateway address). Apart from the IAP rewrite described above, that computed value only replaces a localhost endpoint.
The endpoint resolved above (or server.hub.agent_endpoint, if set — see below) is what the co-located broker then forwards to the container, applying its own bridging rules (host-gateway mapping for a hostname on co-located Docker, no rewrite for an already-reachable IP, and the cloudrun-sandbox runtime’s own link-local logic).
The hub a Runtime Broker reports to
Section titled “The hub a Runtime Broker reports to”A Runtime Broker sends its heartbeat and control channel to server.broker.hub_endpoint (SCION_SERVER_RUNTIMEBROKER_HUBENDPOINT). On a combo server whose Runtime Broker runs next to the Hub, the Runtime Broker uses the Hub’s localhost address when that setting is unset. A server that runs only a Runtime Broker falls back to the CLI’s hub.endpoint (deprecated, see below). scion runtime-broker register and join save server.broker.hub_endpoint when it is unset, except on a host that runs a local workstation Hub, and never write hub.endpoint.
Deprecated fallback to the CLI’s hub endpoint
Section titled “Deprecated fallback to the CLI’s hub endpoint”hub.endpoint and SCION_HUB_ENDPOINT are CLI settings. The CLI now keeps its Hubs in a per-user hub list (see Working with Several Hubs), and the server’s own endpoints belong in server.hub.public_url and server.broker.hub_endpoint. The CLI takes its Hub from the root --hub flag, then SCION_HUB_ENDPOINT (then SCION_HUB_URL), then the Hub the current project directory is linked to, then the current Hub (scion use); it reads no hub.endpoint settings key. When the server’s endpoint actually comes from the legacy fallback, the server logs this warning once at startup:
deprecated: the server's hub endpoint came from hub.endpoint/SCION_HUB_ENDPOINT; set server.hub.public_url (SCION_SERVER_HUB_ENDPOINT) or server.broker.hub_endpoint (SCION_SERVER_RUNTIMEBROKER_HUBENDPOINT).There is no warning when removing the fallback would not change the result, for example on a workstation whose hub.endpoint is its own loopback address. On a host configured as a server, scion doctor reports the same text as its server-hub-endpoint check; it cannot see the server’s flags, so it warns only when every way the server could run would be affected. Selecting a Hub with scion use never changes the endpoint a server advertises.
Splitting the agent endpoint from the public URL
Section titled “Splitting the agent endpoint from the public URL”server.hub.agent_endpoint overrides the endpoint above for agents only — invite links, chat-bridge links, the OIDC issuer default, and the cloudrun_invoker audience default keep reading public_url. Use it when agents must reach the Hub on an address that would be wrong for a human clicking a link, such as an internal VPC IP in a topology where a proxy fronts the Hub for users. When unset, agents receive the Hub’s regular endpoint — public_url, or the endpoint the Hub resolves above when public_url is itself unset.
The value must be scheme://host[:port] only: http or https, an IP literal or a hostname of letters, digits, _, -, and . (Docker Compose-style service names such as scion_hub are accepted), and an optional port — no path, query, fragment, or credentials (a trailing / is stripped from an otherwise-bare URL). The Hub rejects anything else at startup with an error naming server.hub.agent_endpoint.
Scope: this value is injected into agents on every runtime broker attached to this Hub, including remote brokers. Only set it if every broker’s agents can reach the address and it is this Hub on each of those networks — otherwise agents dispatched from a remote broker will send their Hub credentials to whatever answers at that address on their own network.
Security: with an http:// value, agent bearer tokens, and the secrets and tokens agents fetch from the Hub, travel unencrypted on the network between agents and the Hub. Use https:// unless that network is trusted and isolated (for example, a private VPC subnet with no untrusted tenants).
Plugin API URL and public URL
Section titled “Plugin API URL and public URL”Broker plugins receive two Hub URLs with their credentials:
public_url: the Hub endpoint resolved above (server.hub.public_url, or the operator setting that pins it:--base-url,SCION_SERVER_HUB_ENDPOINTorSCION_SERVER_BASE_URL). It is meant for links a person opens in a browser, such as registration links, but no plugin reads it yet (see the warning below).hub_url: the URL the plugin calls the Hub API on. It isserver.hub.internal_urlwhen set, and otherwise the same value aspublic_url.
Behind an auth proxy (IAP or a custom OAuth proxy), a VPN or a TLS-terminating load balancer, set public_url to the address users reach and internal_url to an address the plugins reach directly, such as http://127.0.0.1:<port> for plugins running on the Hub host. Self-managed bridges also receive this hub_url on reconfigure; the bundled chat-app and a2a-bridge ignore it.
Security: plugin API calls are HMAC-signed, but with an http:// value off loopback they travel unencrypted.
Web Base URL
Section titled “Web Base URL”The web server’s base URL is the base of the OAuth redirect URI (<base URL>/auth/callback/<provider>), decides whether the session cookie is marked Secure, and is sent to the browser client. It is resolved once at startup, from operator inputs only:
- The
--base-urlflag. SCION_SERVER_BASE_URL.SCION_SERVER_HUB_ENDPOINT.server.hub.public_urlin the globalsettings.yaml.- In hosted mode with a Cloud Run IAP audience (
server.auth.proxy.iap.audiencein the globalsettings.yaml, or itsSCION_SERVER_AUTH_PROXY_IAP_AUDIENCEenvironment override), the Cloud Run URL derived from it (see IAP Audience Derivation). It is alwayshttps. http://localhost:{port}(the default).
Note that SCION_SERVER_BASE_URL comes before SCION_SERVER_HUB_ENDPOINT here, the reverse of the Hub endpoint order above: the two inputs the web server always read keep their order, and the Hub endpoint values only fill in when neither is set. To give agents a different Hub URL than users, set server.hub.agent_endpoint (see Splitting the agent endpoint from the public URL), not SCION_SERVER_HUB_ENDPOINT. On a server that runs the web frontend without the Hub, only --base-url and SCION_SERVER_BASE_URL apply: there server.hub.public_url and SCION_SERVER_HUB_ENDPOINT name the Hub the Runtime Broker reports to.
A trailing / is removed and the scheme is lower-cased. The base URL is never taken from the settings database or from a server.hub.public_url saved through the admin settings API, and not from hub.endpoint (SCION_HUB_ENDPOINT). The IAP-derived URL is built from the audience in the global settings file or its SCION_SERVER_AUTH_PROXY_IAP_AUDIENCE environment override: operator configuration, never the database. Changing it needs access to the server’s flags, environment or global settings file, and a restart, so a write through the admin API cannot redirect OAuth logins or downgrade the session cookie. The server logs the URL and its source at startup, for example Web base URL: https://scion.example.com (from server.hub.public_url in /home/scion/.scion/settings.yaml).
The OAuth providers check the redirect URI against the one registered for the OAuth client. Google requires an exact match, and GitHub requires the same host and port as the registered callback URL (a path below it is accepted). If the base URL changes, logins fail until the registered redirect URI is updated to match. Neither provider accepts a different host.
Listener check: on every listener, loopback included, the base URL must parse, use http or https, and have a host; otherwise the server does not start, and the error names the URL and where it came from. In addition, the server refuses to start when the web server would listen on an address that is not loopback and its base URL is not https (including the default http://localhost:{port}), because OAuth callbacks and the session cookie would then cross the network unencrypted. The check runs early in startup, before the database and the Hub are set up, right after the dev-auth bind check. Loopback means the listen host is literally localhost, an address in 127.0.0.0/8, or ::1. Names are never resolved, so a hostname that resolves to 127.0.0.1 (for example through /etc/hosts) does not count as loopback. The error names the base URL, where it came from, and the listen address. To fix it, set an https base URL (--base-url, SCION_SERVER_BASE_URL, SCION_SERVER_HUB_ENDPOINT, or server.hub.public_url) or listen on a loopback host (--host 127.0.0.1). A workstation server listens on 127.0.0.1 by default and is not affected.
Runtime Broker transport and trusted proxies
Section titled “Runtime Broker transport and trusted proxies”The Runtime Broker join and secret rotation responses carry the broker’s HMAC secret, Runtime Broker registration and join-token re-mint (POST /api/v1/brokers, used by scion runtime-broker register and scion hub brokers join-token create) return a join token, and the control channel and the broker’s conduit session carry the secrets of the agents it runs. The Hub answers these requests only when they reach it over TLS or from the Hub’s own host:
- Over TLS: if TLS ends at a proxy in front of the Hub (a load balancer, an ingress, Caddy, nginx), the Hub cannot see the TLS connection. It trusts the proxy’s forwarded protocol only when the request comes straight from an address listed in
server.trusted_proxies: a singleX-Forwarded-Proto: https, or a single RFC 7239Forwardedelement withproto=https(when both headers are present they must agree). Whenserver.trusted_proxiesis not set (the default), only a proxy on the Hub’s own host (127.0.0.1and::1) is trusted; set it to[]to trust no proxy. Cloud Run’s front end cannot be listed: its source addresses and its handling ofX-Forwarded-Protoare not documented, so a Cloud Run-hosted Hub serves only its co-located Runtime Broker. - From the Hub’s host: a request from a loopback address with no forwarding headers, such as the co-located Runtime Broker or a CLI on the same machine.
Any other request gets 403 with the error code transport_not_confidential. A refused registration mints or replaces no join token. A refused join also invalidates the join token it presented, because the token has crossed the network unencrypted; mint a new one. A Hub whose TLS ends at a proxy on another host must list the proxy’s addresses in server.trusted_proxies, or remote Runtime Brokers cannot register, join, rotate their secret or connect, and users cannot mint join tokens through it. The 403 body carries details.reason (untrusted_local_proxy, local_proxy_not_trusted, proxy_proto_not_https or not_tls) and details.setting (server.trusted_proxies). A refusal of a request from a proxy on the Hub’s own host is logged at ERROR and reported as the non-critical health check broker_transport_proxy for 15 minutes (surfaced in the health summary and scion doctor as a warning attention item naming server.trusted_proxies), and the Runtime Broker logs the reason and marks its control_channel or conduit check unhealthy.
server.trusted_proxies (SCION_SERVER_TRUSTEDPROXIES, comma-separated) is a list of IP addresses (taken as single hosts) or CIDR ranges: the addresses the proxy connects to the Hub from. Whether the key is set decides the default:
- Not set (no key in the settings file, or a key with no value, and no environment variable): the Hub and the conduit relay trust a proxy on their own host only,
127.0.0.1/32and::1/128, and log one warning at startup. This keeps a same-host proxy such as Caddy or nginx working.scion server preflight, and the==> Settings checksection of the Admin -> Maintenanceupdate-binaryandrebuild-serveroperations, warn when the Hub’s public URL ishttpsand the key is not set, since a proxy on another host is not covered by the default. See the upgrade checklist. - Set to a list: exactly those addresses are trusted. Listing
127.0.0.1and::1confirms the default and silences the warning. - Set to
[](orSCION_SERVER_TRUSTEDPROXIESdefined as empty): to opt out of the default, writetrusted_proxies: []; a baretrusted_proxies:ortrusted_proxies: nullcounts as not set. No proxy is trusted, and only TLS or a loopback request with no forwarding headers is accepted. The Hub keepstrusted_proxies: []when it rewritessettings.yaml(for example when the runtime or the image registry is changed in the web UI), and migrating a legacyserver.yamlkeepstrustedProxies: []astrusted_proxies: [].
The default covers a proxy that connects from loopback. A proxy container on the same host usually connects from a bridge network address (for example 172.17.0.1), not loopback, so it is not covered: list that address. Until then its refusals are logged at WARN with reason not_tls, without the ERROR line and the health check of a loopback proxy, and the Runtime Broker log names the reason and server.trusted_proxies.
The proxy must overwrite X-Forwarded-Proto (and any Forwarded header), not append to it: a header with more than one value is not trusted. Only the immediate peer address is compared; X-Forwarded-For is not used. An entry that is not an IP address or CIDR range, or that is wider than an IPv4 /8 or an IPv6 /16 (so 0.0.0.0/0 and ::/0 are refused), stops the server at startup. The list is used for this check only: it does not enable proxy header login or change how client addresses are rate limited. It is restart-bound and never returned or accepted by the admin API.
server: trusted_proxies: - 10.20.0.0/16 # the ingress controller's pod rangeThe Runtime Broker applies the same rule from its side: scion runtime-broker register and join, and the Runtime Broker process, refuse a plain http:// Hub URL whose host is not loopback (localhost, 127.0.0.0/8 or ::1; names are never resolved). Use the Hub’s https:// URL, or a loopback address such as an SSH port forward (http://127.0.0.1:<port>). The Hub likewise never sends requests to a Runtime Broker’s HTTP endpoint that is plain http:// on a non-loopback address.
Notification channels
Section titled “Notification channels”Notification channels deliver agent messages to external systems. Configure them
under server.hub.notification_channels as a list of channel objects. Each object
has a type, a params map, and optional filters.
server: hub: notification_channels: - type: <channel-type> params: # channel-specific key/value pairs filter_urgent_only: false # if true, only deliver urgent messages filter_types: # if set, only deliver these message types - input-needed - state-changeSlack channel
Section titled “Slack channel”Delivers notifications via a Slack incoming webhook using Slack’s text payload
format.
Type: slack
Parameters:
| Param | Required | Description |
|---|---|---|
webhook_url |
yes | Slack incoming webhook URL (must use https://). |
channel |
no | Override the webhook’s default channel. |
mention_on_urgent |
no | Mention string added when msg.Urgent == true (e.g. @here, @channel). |
Example:
notification_channels: - type: slack params: webhook_url: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXX mention_on_urgent: "@here"Webhook channel
Section titled “Webhook channel”Delivers notifications as a raw HTTP POST to an arbitrary URL. Use this when you need the full structured payload without truncation or when integrating with a custom receiver.
Type: webhook
Parameters:
| Param | Required | Description |
|---|---|---|
webhook_url |
yes | Destination URL (must use https://). |
Example:
notification_channels: - type: webhook params: webhook_url: https://example.com/scion-notificationsEmail channel
Section titled “Email channel”Delivers notifications by SMTP email. Each From, To (per recipient) and Subject header value is kept on a single line: any run of \r and \n characters is collapsed to a single space and trimmed at both ends.
Type: email
Parameters:
| Param | Required | Description |
|---|---|---|
host |
yes | SMTP server host (e.g. smtp.example.com). |
port |
no | SMTP server port (defaults to "587"). |
from |
yes | Sender email address. |
to |
yes | Recipient email address(es), comma-separated. |
username |
no | SMTP authentication username (authentication is skipped when empty). |
password |
no | SMTP authentication password. |
Example:
notification_channels: - type: email params: host: smtp.example.com from: scion@example.com to: oncall@example.comTeams channel
Section titled “Teams channel”Delivers notifications to a Microsoft Teams conversation via the Bot Connector REST API.
Type: teams
Parameters:
| Param | Required | Description |
|---|---|---|
app_id |
yes | Azure App Registration client ID. |
app_secret |
yes | Azure App Registration client secret. |
tenant_id |
yes | Azure AD tenant ID (must be a valid UUID). |
conversation_id |
yes | Teams conversation ID to deliver notifications to. |
service_url |
yes | Bot Framework service URL. Must use https:// and its host must be one of the recognised Bot Framework service hosts: smba.trafficmanager.net, smba.infra.gcc.teams.microsoft.com, smba.infra.gov.teams.microsoft.us, or smba.infra.dod.teams.microsoft.us. |
Parameter values are used exactly as written; settings files do not expand environment variables.
Example:
notification_channels: - type: teams params: app_id: 00000000-0000-0000-0000-000000000000 app_secret: <client-secret> tenant_id: 11111111-1111-1111-1111-111111111111 conversation_id: 19:meeting_example@thread.v2 service_url: https://smba.trafficmanager.net/amerDiscord channel
Section titled “Discord channel”Delivers notifications via a Discord incoming webhook using Discord’s native
webhook format (rich embeds, colour-coded severity, allowed-mentions-controlled
role/user pings). Unlike the Slack channel, the Discord channel targets the
Discord-native endpoint — the /slack-compatibility suffix is explicitly
rejected because it dilutes what each channel type means and silently hides
the user’s real intent.
Type: discord
Parameters:
| Param | Required | Description |
|---|---|---|
webhook_url |
yes | Discord incoming webhook URL. Must use https:// and one of the allowed Discord hosts: discord.com, discordapp.com, ptb.discord.com, canary.discord.com. Path must begin with /api/webhooks/ and must not end with /slack. |
mention_on_urgent |
no | Mention string applied when msg.Urgent == true. Use Discord mention syntax: <@&ROLE_ID> for a role, <@USER_ID> for a user. @here and @everyone are intentionally not supported — the channel sets allowed_mentions.parse: [] so Discord will not resolve them even if present. |
username |
no | Override the webhook’s default username for delivered messages. |
avatar_url |
no | Override the webhook’s default avatar for delivered messages. |
Embed colours by message type:
| Type | Colour | Hex |
|---|---|---|
state-change |
blue | #3498db |
input-needed |
yellow | #f1c40f |
instruction |
grey | #95a5a6 |
| (urgent — any type) | red | #e74c3c (overrides the type colour) |
Truncation: Discord caps embed descriptions at 2048 characters. Messages
longer than that are truncated with a …(truncated) marker — use the webhook
channel type if you need the full structured payload without truncation.
Example:
notification_channels: - type: discord params: webhook_url: https://discord.com/api/webhooks/123456789012345678/abcDEFghiJKLmnoPQR_stu mention_on_urgent: "<@&987654321098765432>" username: Scion Hub filter_urgent_only: false filter_types: - input-needed - state-changeTwo-Tier Settings Architecture (HA Deployments)
Section titled “Two-Tier Settings Architecture (HA Deployments)”In HA deployments where multiple Hub replicas share a Postgres database, settings are split into two tiers to prevent node drift while keeping bootstrap settings file-managed.
Layer 0 — Bootstrap (file + env only)
Section titled “Layer 0 — Bootstrap (file + env only)”Settings required before the database connection exists, or that are restart-bound. Managed exclusively via settings.yaml and SCION_SERVER_* environment variables. Cannot be written via the admin API — PUT /api/v1/admin/server-config returns 422 if any Layer-0 key is present.
| Group | Keys (server. prefix unless noted) |
|---|---|
| Database | database.* |
| Listeners | hub.port, hub.host, hub.read_timeout, hub.write_timeout, broker.* |
| Auth stack | auth.mode, auth.dev_mode, auth.dev_token, auth.dev_token_file, auth.proxy.*, auth.transport.*, oauth.*, oidc_login.* |
| Secrets/storage | secrets.*, storage.*, project_storage.*, and the deprecated workspace_storage.*, shared_dir_storage.*, home_storage.* |
| Identity/mode | mode, env, hub.hub_id, hub.gcp_project_id |
| Logging | log_level, log_format (log_format is accepted but ignored) |
| Server keyring | keyring_file, keyring_file_allow_world_readable (experimental; never returned or accepted by the admin API) |
| Runtime Broker transport | trusted_proxies (never returned or accepted by the admin API) |
| CORS | hub.cors.*, broker.cors |
| Messaging/plugins | message_broker.*, plugins.* |
| Async agent create | hub.launch_timeout, hub.launch_keepalive_seconds |
| Diagnostics | hub.perf_trace |
| Membership standing | hub.membership_sweep_report_only |
| Heartbeat reconcile | hub.missing_agent_grace |
| Conduit relay | hub.conduit.* |
| Separate relay | relay.* |
| Port proxy | hub.port_proxy.* |
| Long-lived requests | hub.sse.*, hub.platform_request_timeout |
| Chat search | hub.chat.* |
Layer 1 — Operational (hub_settings table)
Section titled “Layer 1 — Operational (hub_settings table)”Settings that can be changed at runtime and are shared across all replicas. Stored as section-per-row in the hub_settings table of the Hub database on every driver, SQLite included. settings.yaml only seeds them and is the fallback for a section with no database row (see the Admin Settings Model). The maintenance section is runtime/API-only: it has no settings.yaml representation and starts from admin_mode until a row exists.
| Section | Contents |
|---|---|
access |
admin_emails, user_access_mode, authorized_domains, default_user_role |
lifecycle |
auto_suspend_stalled, stalled_threshold, soft_delete_retention, soft_delete_retain_files, start_claim_lease_ttl, start_max_duration, start_unconfirmed_hold, start_create_unconfirmed_hold |
maintenance |
admin_mode, maintenance_message (durable + cluster-wide) |
telemetry |
Full telemetry.* subtree (enabled, cloud, hub, local, filter, resource) |
agent_defaults |
default_template, default_harness_config, default_max_turns, default_max_model_calls, default_max_duration, default_resources, default_model, default_thinking_level, default_max_agent_role, default_agent_role, default_runtime_broker, default_timezone, default_gcp_identity_mode, default_gcp_identity_service_account_id |
federation |
enabled, trusted_issuers[], algorithms, refresh_interval, debounce_interval |
endpoints |
hub.public_url, hub.hub_name, hub.monitoring_dashboard_url, hub.content_url, hub.content_url_allow_same_site, image_registry |
hub_observability |
hub.observability.metrics_enabled, hub.observability.tracing_enabled (restart required) |
github_app |
app_id, api_base_url, webhooks_enabled, installation_url, private_key_path |
notifications |
notification_channels[] |
project_defaults |
default_scratchpad |
auto_expose_ports |
enabled |
quotas |
enforce_broker_quotas |
agent_secrets |
user_scope_only |
agent_delegation_policy |
remove map per boundary kind (project, hub) listing permissions removed from what may be delegated to agents; it can only narrow (hub.agent_delegation experiment; runtime/DB-only; refused to user access tokens) |
messaging |
conversation_envelope_switch, cross_project_messaging_enabled, offload_threshold_runes, attachment_delivery_max_bytes, attachment_inbox_retention_days, failed_message_retention_days (managed via /api/v1/admin/messaging) |
runtimes |
The whole runtimes map |
profiles |
The whole profiles map |
harness_configs |
The whole harness_configs map |
(reserved) global_defaults |
Reserved for future hub-resource design — not implemented |
agent_defaults.default_timezone is the Hub default TZ for agent containers: an IANA zone name, used only when the agent has no pin and no TZ environment variable applies. Empty means no default (the image default, UTC). An invalid name or Local is rejected with 422. In settings.yaml, and in the PUT /api/v1/admin/server-config request body, it is the top-level default_timezone field. It does not change how times are stored or displayed. See Times and Timezones.
A saved change to agent_delegation_policy can briefly not be in effect on the Hub that saved it: when that Hub cannot re-read the settings right after the write, it applies the change on its next settings refresh (the settings event or the 60-second poll), as the other replicas do.
Precedence
Section titled “Precedence”On every driver, the effective value for any Layer-1 key is resolved in this order (highest priority first):
hub_settingsDB row — cluster-shared, set via admin API. A row fully owns its section.- Bootstrap merge — used only when the section has no row:
SCION_SERVER_*, thensettings.yaml, thenSCION_SEED_*, then compiled defaults.settings.yamlis the global~/.scion/settings.yaml; see--configand the bootstrap merge for how a--configlocation takes part.
A SCION_SERVER_* variable on a Layer-1 key therefore does not override a row an admin has saved. It only changes the seed (re-synced into seeded rows at restart) and the fallback, and it is deprecated for Layer-1 keys in favour of SCION_SEED_*.
Seeding and Migration
Section titled “Seeding and Migration”- Every start: the replica that takes the seed advisory lock syncs
hub_settings(Layer-1 sections only) from its bootstrap merge:SCION_SEED_*,settings.yaml, thenSCION_SERVER_*. A missing section is created, a section no admin has edited (seeded) is re-synced when its content differs, and an edited (managed) section is not touched. A replica that finds the lock held skips the sync. - DB wins: once a section is seeded/written to DB, the DB row fully owns that section. Omitted fields within the section fall to compiled defaults, not to the file.
- Rollback safety: older builds ignore the
hub_settingstable entirely and read files — rolling back reverts to pre-change behavior. This covers operational settings only: other stored data is rewritten one way, so a binary rollback after a newer build has started is unsupported. See Hub Upgrade and Binary Rollback.
Environment Override Warnings
Section titled “Environment Override Warnings”Because env overrides on Layer-1 keys reintroduce per-node drift, the system warns administrators:
-
GET /api/v1/admin/server-configincludes anenv_overridesarray listing every key, Layer-0 or Layer-1, that aSCION_SERVER_*variable sets on the serving node. Keys that no settings write path accepts (for example the hub class, the test identity settings, the relay-peer secret and the server keyring file) are left out, so the response does not reveal whether they are set. -
The same response includes
env_values, a map with exactly theenv_overrideskeys, each{ "value": "********", "masked": true }for a masked key, or{ "value": ..., "in_effect": true|false, "saved_differs": true }otherwise (maskedandsaved_differsare left out when false):valueis the environment value as the Hub parsed it (string, number, boolean or list).maskedistruewhenvalueis the placeholder********: the key is a secret (masked as in the rest of the response), it sits underprofiles,runtimesorharness_configs, or its value cannot be shown safely.in_effectistruewhen the Hub on the serving node runs with the environment value:- In
settings.yamlmode the environment layer wins over the file, so it istrue, except for the keys below that start-up sets afterwards. - On database-backed Hubs a Layer-1 key is in effect only when the environment value equals the value the Hub applies (a saved database value, or a seeded row, wins). Other Layer-0 keys are read at startup with the environment on top and are in effect, except for the keys below.
scion server startsets some keys after loading the settings, whatever the environment says: on a workstation Hub the hub host, the broker switch, dev auth, the storage provider and the secrets backend; and in any mode the keys of--port,--host,--db,--enable-runtime-broker,--runtime-broker-port,--dev-auth,--storage-bucketand--storage-dir. For these keysin_effectcompares the environment value with the value start-up resolved. The--conduit-*flags are not compared (a known limitation): for their keysin_effectfollows the settings precedence.
- In
saved_differsistruewhen a stored value exists (insettings.yaml, or in an admin-saved database row for a Layer-1 key) and differs from the environment value. A default, a seeded row, or a stored value equal to the environment value is not reported.- A masked entry carries only
value(********) andmasked: neitherin_effectnorsaved_differsis computed or returned for it. Both compare the environment value with a stored or running value, so for a secret they would let a caller who can write settings confirm a guessed value.
env_valuesis additive;env_overridesis unchanged. -
The same response includes
running_values, read-only, for the keys thatscion server startsets after loading the settings (listed above; the--conduit-*keys are not included). An entry{ "value": ..., "source": "startup" }gives the value the serving node runs with, from a workstation default, a start flag or the coded default, when it differs from the value the response shows. It is left out when the environment sets the key to that value (env_valuesthen reports it as in effect). A secret key never has an entry, masked or not: whether an entry exists compares the running value with the stored one, so for a secret it would confirm a guessed value. These keys are all restart-required, so the start-up value is the value in effect until the next start. Values that come from a--configfile are not covered. -
A startup
WARNlog lists any overridden Layer-1 keys. -
The admin UI renders a visible warning banner when env overrides are detected, and a badge on each affected field, or on the section for the
runtimesandprofilesmaps: “Overridden by environment on this node” with the value in effect, “Environment value superseded by the saved setting” when a saved value wins, or “Environment value not in effect on this node” when a start-up default or flag wins.
Admin API Behavior Notes
Section titled “Admin API Behavior Notes”PUT partitioning: The request body is partitioned by the section registry. Layer-1 fields (including runtimes, profiles, and harness_configs) are written to DB sections in the hub_settings table as JSONB documents, while Layer-0 fields trigger a 422 rejection on hosted Hubs (on workstation Hubs they are edited in place in settings.yaml; see Layer-0 edits by deployment mode). A key the Hub would not persist (an unknown key at any depth, a flat dotted key such as "server.hub.auto_suspend_stalled" at the top level, or a field with no storage) is also rejected: the response is 422 with error: unpersisted_keys_rejected and the offending paths in keys, and nothing is saved. Fields whose value equals what GET returns are treated as echoes and accepted, so sending the GET body back still succeeds. In both DB mode (every non-map section) and settings.yaml mode (server, telemetry, and federation sections), a partial PUT merges onto the current settings and keeps every omitted key, so the admin UI can send only the fields you changed. runtimes is merged per entry in both modes. The map you send decides which entries exist: an entry you leave out, or send as null, is deleted. Within an entry you send, a key you leave out keeps its stored value, a key sent as null (or as an empty value: "", false, {}, []) is cleared, and any other key you send replaces the stored value. A stored key that is not a runtime setting, for example runtimes.<name>.resources written into settings.yaml by hand, is kept as stored and named in the response warnings; nothing reads it. A stored key that matches a runtime setting only when case is ignored (for example Namespace) is kept as stored and named in a warning that suggests the canonical spelling. A body that sends such a key is refused as an unknown key. profiles and harness_configs still replace the stored map whole. See Resource quantities for the checks on default_resources and profile resources. default_thinking_level must be between 1 and 100 (0 or any other out-of-range integer is refused with 422 validation_error; null clears it), and {"federation": null} is refused with 422 validation_error (on a database-backed Hub, use DELETE /api/v1/admin/server-config/sections/federation to restore the bootstrap federation settings).
Revision CAS: The request body may include expected_revisions — a map of section name to expected revision number. On mismatch, the response is 409 Conflict (revision_conflict); error.details.conflicted lists each conflicting section with its expected and current revision (see Settings save errors). In DB mode, a section the request sends uses the revision the Hub read while building it as an implicit CAS when expected_revisions omits it, so a concurrent change to that row between read and write returns 409 even without expected_revisions. This covers every merged section, the per-entry runtimes merge, and the whole-map profiles section, which replaces the stored map. Only harness_configs uses last-writer-wins semantics when omitted from expected_revisions. Sections are written in alphabetical order for deterministic partial-apply behavior, and live snapshot applies are ordered by a store-wide revision so an older snapshot never overwrites a newer one. An expected revision of 0 is create-only: it succeeds only while the section has no stored row. The admin UI (Server Configuration and Federation pages, database-backed and workstation Hubs) sends only the sections you changed, each whole, with the revision each had when the page loaded (0 for a section with no stored row), so a section another admin saved in the meantime is refused with 409 instead of being overwritten.
Presence-aware clearing: The PUT handler distinguishes omitted fields, which keep the current stored value, from fields sent as an empty value ("", []) or JSON null, which clear the stored key. In DB mode, a cleared key is unset; settings.yaml and environment bootstrap values are not re-applied, except for the keys listed under Hot reload vs restart. This allows clearing admin_emails, user_access_mode, authorized_domains, default_user_role, notification_channels, public_url, content_url, monitoring_dashboard_url, and other operational settings without sending every field. In PUT /api/v1/admin/integrations/{name}/config, settings values are string | null, and a key sent as null is deleted so the plugin default applies again.
Masked secrets: GET /api/v1/admin/server-config masks secrets (OAuth client secrets, GitHub App keys, notification channel parameters, and other credentials). A PUT may send a masked placeholder back only inside a block that exactly matches the stored block once masked; the Hub then keeps the stored secret. The block is the structure the secret sits in (for example one OAuth provider, the GitHub App, or one notification channel). To change any field of such a block, send every secret in that block in clear. Any other placeholder is rejected with 400, so it is never stored over a real value. The admin web UI leaves unedited masked blocks out of its saves.
Maintenance durability: PUT /api/v1/admin/maintenance writes to the maintenance section in DB, making admin/maintenance mode durable across restarts and propagated to all replicas. SCION_SERVER_ADMINMODE env var still force-enables per node for break-glass access. In file/SQLite mode, maintenance changes are ephemeral (in-memory only, lost on restart). Use SCION_SERVER_ADMINMODE=true env var for persistent control.
Schema endpoint: GET /api/v1/admin/server-config/schema returns JSON-schema fragments and koanf key paths per section for UI form generation and CLI validation.
Kubernetes placement and identity checks
Section titled “Kubernetes placement and identity checks”A save checks the Kubernetes placement and identity fields of runtime entries and profiles with the rules scion config validate applies: node_selector keys and values, tolerations (an empty key needs operator Exists; Exists takes no value; the effect is empty, NoSchedule, PreferNoSchedule or NoExecute), runtime_class_name, priority_class_name (runtime entries), kubernetes_service_account_mappings (lowercase GCP service account email to Kubernetes ServiceAccount name) and kubernetes_block_service_account, and that runtime entries targeting the same cluster and namespace (the same context and namespace) agree on placement.
- A problem the save causes is refused (
422in DB mode,400insettings.yamlmode), naming the field: an invalid value in a field the save sets or changes, an invalid value on a new entry, or a disagreement in a sibling group where the save adds an entry or changes a member’s placement, context or namespace. - A problem already stored that the save does not touch is accepted with a warning in
warnings, so a stored problem never blocks an unrelated edit. Fixing a stored disagreement in the save is accepted.
Resource quantities
Section titled “Resource quantities”default_resources, profiles.<name>.resources and profiles.<name>.harness_overrides.<harness>.resources take Kubernetes quantities: cpu in cores or millicores (2, 500m), memory and disk in bytes with a unit (512Mi, 16Gi, 20Gi; Ki, Mi, Gi and Ti are powers of 1024, k, M, G and T powers of 1000). A memory or disk value without a unit is a byte count, so memory: "16" is 16 bytes, not 16 GiB. A pod with such a limit starts but fails: its in-memory volumes are sized from the memory limit, so writes fail with ENOSPC.
A save (PUT /api/v1/admin/server-config, both modes) checks every value it sends:
| Value | Result |
|---|---|
| Not a Kubernetes quantity, zero or negative | Refused (422) |
memory or disk below 1Mi, with or without a unit (catches "16") |
Refused |
A cpu or memory request above its limit |
Refused |
memory or disk of 1Mi or more without a unit ("2147483648") |
Saved, with a warning that suggests a unit (2Gi) |
memory below 64Mi |
Saved, with a warning |
A refusal names the field, for example profiles.gke.resources.limits.memory: "16" has no unit, so it means 16 bytes, below the minimum of 1Mi; use a quantity with a unit, such as 16Gi or 512Mi; details.errors lists every refused value. Warnings come back in warnings on the 200 response. The admin UI applies the same rules as you type: an error blocks Save & Reload, a warning is shown under the field. Settings already stored are not refused at startup; scion config validate reports these values as warnings. A save re-checks every profile it sends, and the page sends every profile on each save, so a stored invalid value has to be fixed before the next save of the profiles goes through.
GCP IAM Check Mode
Section titled “GCP IAM Check Mode”The gcp_iam_check_mode setting controls whether the Hub verifies that a caller holds the iam.serviceAccounts.actAs IAM permission on a GCP service account before allowing it to be assigned to an agent. This uses the GCP Policy Troubleshooter v3 API.
Values
Section titled “Values”| Value | Behaviour |
|---|---|
"off" (default) |
No GCP IAM check. Project owners, admins and members can assign any verified project-scoped service account in their project to an agent on Scion role authority alone, with no GCP IAM iam.serviceAccounts.actAs check on the caller. Hub-scoped service accounts remain unassignable in "off". See the caution above. |
"enforce" |
The Hub calls Policy Troubleshooter to verify the caller has actAs. Denials are enforced. |
Configuration
Section titled “Configuration”server: hub: gcp_iam_check_mode: "off" # or "enforce"Or via environment variable:
export SCION_SERVER_HUB_GCPIAMCHECKMODE=enforceAn unset value takes the default shown above. A value that is set but not recognised (for example a typo) takes the stricter value instead: "enforce" for gcp_iam_check_mode and "fail-closed" for gcp_iam_deny_unknown_policy, with a warning in the Hub log.
Changing the Settings from the Admin UI
Section titled “Changing the Settings from the Admin UI”On a Hub with database-backed settings (every driver), gcp_iam_check_mode and gcp_iam_deny_unknown_policy are the gcp_iam operational settings section. A hub admin edits them on the GCP Identity tab of Admin > Server Config (or with PUT /api/v1/admin/server-config), and the change applies on every replica without a restart. The value in settings.yaml or the environment is the deploy-time value the Hub starts from; a saved value overrides it in either direction.
- Only a hub admin signed in with an interactive session can change these settings or reset the section. User access tokens, federated users, agents and brokers are refused with
403. - A saved value must be one of the values listed above; an empty or unrecognised value is rejected with
422and nothing is saved. - A change that moves either setting to its less strict value is refused with
409while hub-scoped service account assignment is configured: the hub default GCP identity mode is Assign, or any hub-scoped service account is registered. Resetting the section to its deploy-time value follows the same rule. - Every change is recorded in the mutation audit log (
hub_setting_update) with the caller, time, old and new value, and the surface (server-configorsection-reset) before it is written. If the record cannot be written, the change is not made. - Each other replica applies the change on reload only if the stored values are valid and the change passes the same rule. A change that moves either setting to its less strict value must also be named by the latest audit record for each changed key, and that write must not be recorded as not made. The replica then records
hub_setting_apply(surfacereload, with the original caller, or the hub itself when no audited write names the change). Otherwise the replica keeps the values it applied, logs an error and recordshub_setting_apply_refusedonce. This covers a stored value that cannot be used and a less strict value that no audited write names, including one left by a row that changes or disappears. If the settings store cannot be read, the Hub keeps the values it last applied. - The page shows the values the Hub applies, and marks a key that an environment variable sets on this node.
Enablement Checklist
Section titled “Enablement Checklist”Before setting gcp_iam_check_mode: enforce:
-
Enable the Policy Troubleshooter API on the Hub’s GCP project:
Terminal window gcloud services enable policytroubleshooter.googleapis.com -
Grant the Hub SA
roles/iam.securityReviewer:- On the Hub’s own GCP project (minimum; covers Hub-minted SAs).
- On each org or project containing BYOSA service accounts, if applicable.
-
(Recommended) Grant
roles/iam.denyReviewerandroles/browserat the org level for full deny-policy and resource hierarchy evaluation. -
(Optional) Configure domain-wide delegation with Workspace
groups.readfor group-binding resolution. Without this, group-grantedserviceAccountUserbindings produce an indeterminate result (denied by default). -
Test with a known-good SA assignment before enabling in production.
Group-Binding Limitation
Section titled “Group-Binding Limitation”When roles/iam.serviceAccountUser is granted to a Google Workspace group, the Hub SA must have domain-wide delegation with the groups.read privilege to resolve the membership. Without it, Policy Troubleshooter returns MEMBERSHIP_UNKNOWN_INFO, which under fail-closed rules is treated as a denial.
This denies legitimately authorized users whose actAs grant arrives via a group binding, even when the PT API is fully available and functioning correctly.
| Mitigation | Cost | Resolves groups? |
|---|---|---|
Grant Hub SA domain-wide delegation + groups.read |
High (org-admin consent per Workspace) | Yes |
Grant actAs directly to users, not via groups |
Low (per-SA IAM binding) | Avoided |
Leave gcp_iam_check_mode: "off" |
Zero | N/A (check disabled) |
BYOSA Cross-Org Access
Section titled “BYOSA Cross-Org Access”For BYOSA service accounts (accounts in a customer’s org, not the Hub’s), the Hub SA needs roles/iam.securityReviewer in the customer’s organisation or at minimum on the project containing the SA. Some customers may refuse this grant. Their options are:
- Leave
gcp_iam_check_mode: "off"(the default). - Grant
securityRevieweron the specific project (not org-wide). - Accept that BYOSA assignments will fail closed until the grant is made.
Required IAM Permissions Summary
Section titled “Required IAM Permissions Summary”| Role / Permission | Scope | Purpose | Required? |
|---|---|---|---|
roles/iam.securityReviewer |
Org or project containing the target SA | Read allow policies, role bindings, and role definitions | Yes |
roles/iam.denyReviewer |
Org or folder | Read IAM Deny policies | Recommended |
roles/browser |
Org | Read project/folder hierarchy for policy inheritance | Recommended |
Workspace Admin groups.read (via domain-wide delegation) |
Google Workspace domain | Resolve group memberships in IAM bindings | Only if group-granted actAs must be resolved |
| PT API enabled on Hub’s GCP project | Hub’s GCP project | policytroubleshooter.googleapis.com |
Yes |