Admin Settings Model
This document describes how Scion manages operational settings in the Hub settings database, including the configuration precedence model, the seeded/managed lifecycle, and the admin settings UI. Every Hub keeps its operational settings in its database, whichever driver it uses: embedded SQLite (workstation and single-node hosted) or Postgres (HA hosted).
Configuration Precedence
Section titled “Configuration Precedence”Scion resolves configuration values using a layered merge:
coded defaults → SCION_SEED_* → settings.yaml → SCION_SERVER_*Each layer overrides the one before it. The final merged result is the bootstrap configuration — the starting point for a hub instance.
settings.yaml is the global ~/.scion/settings.yaml. With scion server start --config <path>, the merge picks its settings file with the same rule the server uses to load its configuration at startup:
- If the global file has a
serverkey, it is used and the--configsettings.yamlis not read. - If the global file has no
serverkey and the--configsettings.yamlhas one, theserverblock and the top-level hub sections (telemetry,quotas,agent_secrets,project_defaults,default_harness_config,default_timezoneanddefault_gcp_identity_*) come only from the--configfile. Other keys it does not set, such asimage_registry,runtimesandprofiles, still come from the global file. - If neither file has a
serverkey, the--configsettings.yamlis used only when there is no globalsettings.yaml, and the legacyserver.yamlin the--configlocation, or the named file, is layered as at startup.
Layer-0 vs Layer-1 Keys
Section titled “Layer-0 vs Layer-1 Keys”Settings are classified into two layers:
| Layer | Examples | Behavior |
|---|---|---|
| Layer-0 (bootstrap) | server.mode, server.database.*, server.storage.*, server.secrets.*, server.hub.port, server.auth.dev_mode |
Always resolved from bootstrap configuration (settings.yaml and environment). Editable through the admin UI only on workstation Hubs; see Layer-0 edits by deployment mode. |
| Layer-1 (operational) | server.hub.admin_emails, server.auth.user_access_mode, server.auth.default_user_role, server.auth.authorized_domains, server.hub.public_url, server.hub.content_url, server.hub.content_url_allow_same_site, server.hub.hub_name, image_registry, telemetry.*, the agent defaults (top-level default_* keys such as default_template and default_timezone), server.github_app.*, server.notification_channels, server.federation.*, runtimes, profiles, harness_configs, auto_expose_ports.enabled, quotas.enforce_broker_quotas, agent_secrets.user_scope_only, project_defaults.default_scratchpad, server.hub.observability.metrics_enabled, server.hub.observability.tracing_enabled |
Stored in the Hub database on every driver and editable via the admin UI. Bootstrap values serve as initial defaults. |
For the full key lists by section, see Layer 0 and Layer 1 in the server configuration reference.
Resolution
Section titled “Resolution”On every Hub, SQLite or Postgres, Layer-1 settings follow this resolution:
Database value > Bootstrap merge (for Layer-1 keys)Bootstrap only (for Layer-0 keys)The database always wins for Layer-1 keys. Bootstrap values are used as initial seeds and as fallback when no database row exists. Editing a Layer-1 key in settings.yaml therefore only changes the seed: once an admin has saved that section through the API, the database value wins, on SQLite exactly as on Postgres.
This applies to SCION_SERVER_* too. On a Layer-1 key it is only the top of the bootstrap merge: it changes the seed (which a seeded section re-syncs from at restart) and the fallback, and it never overrides a value an admin has saved. The admin UI marks such keys (see Visual Indicators), and the variable is deprecated for Layer-1 keys (see SCION_SERVER_* Deprecation for Layer-1).
server.hub.public_url: environment first
Section titled “server.hub.public_url: environment first”server.hub.public_url is one of two Layer-1 keys where an operator setting on the node ranks above the database (the other is server.auth.authorized_domains). The Hub resolves it once at startup, in this order:
- Operator pin. If the
--base-urlflag,SCION_SERVER_HUB_ENDPOINTorSCION_SERVER_BASE_URLis set, the Hub uses its operator-configured endpoint: the first of them that is set, in that order, as described in Hub Endpoint Resolution. A value saved in the database is not used. - Database. Otherwise a value an admin saved through the admin API or UI is used, even when
settings.yamlalso sets one. A stale bootstrap file therefore does not override a value managed in the UI. Rows the Hub seeds from bootstrap material at startup do not count as saved values, and a save that leaves the Public URL unchanged (or sends back the value shown) does not store it. A public URL stored by a save in an earlier release stays in effect until an admin clears it. - File. Otherwise the
settings.yamlvalue applies, then the existing fallbacks.
The endpoint resolved this way is the Hub endpoint everywhere it is used: the URL dispatched agents receive (unless server.hub.agent_endpoint is set), invite links, chat-bridge settings, the hub.endpoint written into hub-managed project settings, the broker join response, the OIDC issuer default and the cloudrun_invoker transport audience default. The web server’s base URL (OAuth redirect base, session cookie Secure flag, and browser config) is resolved separately at startup from operator inputs only and never reads the settings database; see Web Base URL.
- Restart required. A save is stored at once and listed in the save report’s
pending_restart. The Hub picks it up at the next restart. - Validation. A value saved in the database must be an absolute URL of the form
scheme://host[:port]:https, orhttponly forlocalhostor a loopback address; a host; no user credentials, query, fragment or path. A trailing/is removed. The admin API rejects anything else with422 validation_errornamingserver.hub.public_url. At startup an invalid database value is logged as an error and not used; the operator-configured endpoint applies instead. Values from the environment, flags andsettings.yamlkeep their existing handling. - Startup log. The Hub logs one line with the operator-configured endpoint and its source, and one line with the endpoint in effect and its source (
env,flag,database,file, or a fallback such aslocalhost). When a database value replaces asettings.yamlvalue, it logs that too. - Audit. A change to the stored value, by a save or a section reset, is recorded as a
hub_setting_updateaudit record with the actor and the old and new values, before it is written. If the record cannot be written, the change is not made. If the write then fails, ahub_setting_update_not_appliedrecord follows. - Admin UI. When an operator setting pins the endpoint, the Public URL field is read-only and names the setting that pins it. Otherwise the field is editable and shows the saved value or, with none saved, the
settings.yamlor environment value; it never shows a seed value, which startup does not use. An In effect hint next to it shows the endpoint in effect and its source (the admin API reports it inpublic_url_status).
The environment variable for this key is SCION_SERVER_HUB_ENDPOINT, and its seed variable is SCION_SEED_SERVER_HUB_ENDPOINT. Startup does not use the seed variable to resolve the endpoint, so the admin UI does not show its value as the field value. The spellings SCION_SERVER_HUB_PUBLICURL and SCION_SEED_SERVER_HUB_PUBLICURL are no longer read; the Hub warns about them at startup and names the replacement. Because the environment pins this key, SCION_SERVER_HUB_ENDPOINT is not part of the SCION_SERVER_* deprecation for Layer-1 keys.
server.hub.content_url: the content host
Section titled “server.hub.content_url: the content host”server.hub.content_url is the Hub’s content host: a second public address, apart from server.hub.public_url, on which shared artifact links and HTML artifact views are served. It is empty by default, which serves everything on public_url. See Content host for what the Hub enforces there and how to set up the proxy.
- Who can set it. Hub administrators, through the
serverobject ofPUT /api/v1/admin/server-config({"server": {"hub": {"content_url": "https://example-content.net"}}}), in database and file mode, likepublic_url. User access tokens cannot write it: a token body that carries it is refused as a whole, and nothing in it is saved. A token cannot reset theendpointssection either, since that section holds keys tokens cannot write.""clears the saved value; asettings.yamlor environment value then applies. - Validation. The value must be
https://host[:port](httponly forlocalhostor a loopback address), with no path, query, fragment or user info; IPv6 literals are refused. It is stored normalized (lower case, default port dropped). A value on the same host name aspublic_urlorserver.hub.agent_endpointis refused with422 validation_errornaming the key, whatever its scheme or port (for local testing, use another loopback name, such as127.0.0.1for a Hub onlocalhost). A value onpublic_url’s registrable domain is refused unlessserver.hub.content_url_allow_same_siteis set; with the override it is saved and logged with a warning, because it is not a separate security origin for cookies. The supported setup is a content host on a different registrable domain.server.hub.content_url_allow_same_siteis set the same way ({"server": {"hub": {"content_url_allow_same_site": true}}}), by administrators only, and every change of it is audited likecontent_url. - Applies without a restart. A save applies at once on the serving node and reaches other replicas through the settings event. A stored value that is invalid, that conflicts with
public_url(for example afterpublic_urlchanges), or that is onpublic_url’s registrable domain whileserver.hub.content_url_allow_same_siteis not set, is ignored with a logged warning or error, and shared artifacts stay onpublic_url. When you change or clear it, the previous host is no longer confined: it is served like any other host name that reaches the Hub. Remove its DNS record or proxy route at the same time. The Hub’s cookies stay host-only, so no credential reaches it, but requests on it stop being confined. - Status. When the Hub’s settings are stored in the database (every Hub with operational settings, SQLite included),
content_url_statusin theGET /api/v1/admin/server-configresponse is read-only and tells you what the Hub does with the value as applied:stateisunset,in_effect(with the normalized URL ineffective),ignored_invalid,refused_same_host(on the Hub’s own host name),ignored_same_site(onpublic_url’s registrable domain, withoverride_keyset to"server.hub.content_url_allow_same_site"), orignored_agent_endpoint_host. APUTbody that sendscontent_url_statusback has it dropped. On such a Hub the Admin > Server Config > Hub page shows the stored value with this state and its reason. A Hub that keeps its settings only insettings.yaml(file mode) returns nocontent_url_status, and its page shows no content URL row. A configured value that is not in effect, or is in effect only throughserver.hub.content_url_allow_same_site(a warning), is logged with its reason. A Hub with operational settings logs it at startup; a file-mode Hub logs it the first time it resolves the content URL, for example for the first request on a host name other than the Hub’s. - Audit. Every change of the value in effect, by a save or a section reset, is recorded as a
hub_setting_updateaudit record onserver.hub.content_urlwith the actor and the old and new values, before it is written. If the record cannot be written, the change is not made. If the write then fails, ahub_setting_update_not_appliedrecord follows. A save that does not change the value records nothing. - Environment.
SCION_SERVER_HUB_CONTENTURLsets it from the environment, andSCION_SERVER_HUB_CONTENTURLALLOWSAMESITEsets the same-site override.
server.auth.authorized_domains: environment first
Section titled “server.auth.authorized_domains: environment first”The sign-in domain list is applied without a restart, like the other access keys. It resolves in a different order, on purpose: a node whose deployment restricts sign-in keeps that restriction whatever is saved.
- Environment pin. If
SCION_SERVER_AUTH_AUTHORIZEDDOMAINSis set on the node, the Hub uses its list. A value saved in the database is stored but not applied on that node. The save report lists the key underskipped, and the field’s environment badge shows the environment value as in effect. - Database. Otherwise a list an admin saved is used.
- File and seed. Otherwise the Hub uses the
settings.yamllist, then theSCION_SEED_SERVER_AUTH_AUTHORIZEDDOMAINSseed.
Clearing the saved list (or saving the access section without one) returns the Hub to the list it started with, from settings.yaml and the environment. It does not allow every domain. Each node falls back to its own startup list, so on a Hub with several replicas the list in effect after a clear can differ between nodes whose settings.yaml or environment differ. To lift a restriction set in settings.yaml or the environment, change the deployment configuration and restart. GET shows that startup list while no list is saved, and a save that sends it back stores nothing.
A save that sends authorized_domains or admin_emails is refused with 422 validation_error, naming server.auth.authorized_domains, if the access settings it would leave in effect refuse sign-in for the account making the save. Resetting the access section (DELETE /api/v1/admin/server-config/sections/access, or scion hub settings unset --section access --reset) runs the same check against the startup values the reset would restore. That is the case when the account’s email is not in admin_emails (as saved, or the running list when the save leaves it empty) and the domain list in effect after the save is not empty and does not match the email’s domain. The domain list in effect is the saved list, or the startup list when none is saved. On a database-backed Hub the saved list is checked even on a node that pins the list, because it applies on every node without the pin, and on this node once the pin is removed. On such a node, a save that sends admin_emails is also checked against the pinned list. Nothing is written when the save is refused. In settings.yaml mode (a single node), a save that sends only authorized_domains while SCION_SERVER_AUTH_AUTHORIZEDDOMAINS is set is not checked, because the environment list stays in effect. The check covers the saving account only; it does not stop a save that excludes other administrators.
SCION_SERVER_AUTH_AUTHORIZEDDOMAINS is not part of the SCION_SERVER_* deprecation for Layer-1 keys. The other access keys (admin_emails, user_access_mode, default_user_role) keep the database-first rule: a saved value wins over their SCION_SERVER_* variables.
Hot reload vs restart
Section titled “Hot reload vs restart”- Layer-1: a save applies without a restart. The serving node applies it immediately, and other replicas pick it up through the settings event (within about 2 seconds), with a 60-second poll as a backstop. Live applies are ordered by a store-wide revision so an older snapshot never overwrites a newer one. A few Layer-1 keys take effect only after a restart. The schema endpoint (
GET /api/v1/admin/server-config/schema) lists them per section inrestart_required: todayserver.hub.public_urlandserver.github_app.app_id. - Layer-0: changes take effect only when the Hub restarts. On a workstation Hub, a save that changes a Layer-0 key in
settings.yamllists it in the response’sreload.requires_restart.log_levelis the exception and applies immediately. - Restoring
hub_settings: the Hub orders settings by a revision counter stored inhub_settings(the reserved_settings_revisionrow). If the table is restored from a backup, or the database is replaced, while a Hub is running, the stored counter can be lower than the one the Hub has seen. Each replica detects this on its next settings read (the settings event or the 60-second poll), logs a warning (the stored settings revision went backward) and resynchronises its settings to the database. This assumes that every replica reads from the one primary database it writes to; the Hub does not route reads to read replicas. The database URL must therefore target the read-write primary: with several hosts in the URL, settarget_session_attrs=read-write, and do not put the Hub behind a proxy that sends some connections to a standby. Restarting the Hub after a restore is still the most predictable option, in particular for the keys that take effect only after a restart.
Clearing one of these Layer-1 keys returns the running Hub to the value it started with (from settings.yaml, the seed or the environment): soft_delete_retention, soft_delete_retain_files, auto_expose_ports.enabled, telemetry.enabled and server.auth.authorized_domains (its startup list comes from settings.yaml and the environment only). Clearing image_registry returns it to the settings-file value (see below). Two sections revert as a whole:
- GitHub App: clearing
app_idreturns all the non-secret GitHub App fields to their startup values. Whileapp_idis set, clearing another field applies the empty value, exceptprivate_key_path, which keeps the startup path. - Telemetry: a telemetry section with no keys returns to the startup telemetry config. Clearing one telemetry key while others remain applies the config without that key, not its startup value.
Some admin fields are saved only when you change them:
- Federation page: a save sends only the fields that differ from the values the page loaded, and the Hub keeps every key a request leaves out (in database and
settings.yamlmodes alike). A default the page shows for a keyGETleft out (enabled,algorithms,refresh_interval,debounce_interval) is therefore not stored unless you edit it. Adding, editing or removing an issuer saves straight away, sendingtrusted_issuerstogether with any other field you have changed. - Server Config page: the
auto_expose_ports.enabled,telemetry.enabled,telemetry.cloud.enabled,telemetry.cloud.cloud_logging,telemetry.hub.enabled,quotas.enforce_broker_quotasandagent_secrets.user_scope_onlyswitches show Not set while no value is stored and are sent only when you change them; Use default clears a stored value (the page sendsnull).default_runtime_brokeris sent only when you change it, so a broker name or slug stays as you wrote it although the dropdown shows the broker. Some other Server Config fields are still sent on every save, including the default the page shows whenGETleft the key out:server.auth.user_access_modeandserver.auth.default_user_role, and insettings.yamlmode alsoserver.broker.auto_provideandserver.native_chat.enabled. On a workstation Hub, Layer-0 andsettings.yaml-only fields are sent only when changed (see Layer-0 edits by deployment mode). telemetry.cloud.cloud_loggingon a database-backed Hub: the Hub reads this switch only at startup, fromSCION_CLOUD_LOGGINGorsettings.yaml, before the settings database is opened, so a stored value would never take effect. A database-backedPUT /api/v1/admin/server-config(workstation Hubs included) therefore refuses a value sent for the key itself (true,falseornull, at any nesting or dotted spelling) that differs from whatGETshows, with422and{"error": "startup_only_key_rejected", "keys": ["telemetry.cloud.cloud_logging"]}, and writes nothing. Sending back the valueGETshows is accepted and stores nothing. The check looks only at the key: a body that replaces or nulls a parent object ({"telemetry": {"cloud": null}}) is not checked for it and follows the normal telemetry rules, which can drop a stored, unused value. The Server Config page shows the switch read-only there and never sends it. Insettings.yamlmode the switch stays editable and applies at the next start.- Integrations page: an unset text field shows the plugin default as its placeholder, and an unset select or switch shows the default as its value; neither is saved unless you change it. Clearing a stored text value, or Use default on a select or switch, sends
nullinPUT /api/v1/admin/integrations/{name}/config, which deletes the key from the plugin config so the plugin default applies again. Turning a stored switch off sends"false".
For soft_delete_retention and soft_delete_retain_files, GET returns that running value while no value is stored. A save that sends the same value back leaves the key unset: it is not stored and not reported. A retention is compared as a duration, so 72h0m0s matches 72h.
image_registry follows one order everywhere; see Image registry order. In the admin settings:
- The top-level field shows the saved value, or else this Hub replica’s settings-file value (
settings.yaml,SCION_SEED_IMAGEREGISTRYandSCION_SERVER_IMAGEREGISTRY). It never shows an environment value such asSCION_IMAGE_REGISTRY. A save stores the field only when it differs from the value shown, so saving the page unchanged stores nothing. To pin the settings-file value, change it once. - Each profile’s field shows only a value saved for that profile. A hint shows what the profile inherits and from where. Typing a value saves it.
- Each profile’s
image_tag(profiles.<name>.image_tag, behind thehub.image_tag_settingexperiment; see Image tag) likewise shows and keeps only a value saved in the Hub for that profile, never a settings-file profile value, so savingprofilesdoes not copy a settings-fileimage_taginto the database. Whilehub.image_tag_settingis off, a save that sets or changes a profile’simage_tagis refused with412 experiment_disabled(details.experiment: "hub.image_tag_setting"). image_registry_provenancein theGETresponse is read-only and tells you where the value in effect comes from: the Hub-wide source and value, every Runtime Broker that reportsSCION_IMAGE_REGISTRY(which beats every saved or file value for that Runtime Broker’s agents), and each profile’s value. The page shows it as one badge on the field. Every value in the report has its credentials part (everything up to the last@that is not part of a digest, after any scheme) replaced with***, as in***@registry.example.comforuser:password@registry.example.com.
placement_preflight in the GET response is a read-only Kubernetes placement report, with counts only (no entry names or values). settings_counts counts the placement findings in the stored runtimes and profiles, by category, and settings_repair_sections names the sections to fix. agent_counts counts the agents of the co-located legacy Runtime Broker whose stored configuration would be refused at their next start or restart. agents_incomplete, agents_not_covered (agents of other Runtime Brokers, or not yet placed) and agents_not_enumerated are incomplete coverage, not clean results. The category names are the ones the scion doctor placement lines use; see Legacy host placement check. The agent half of the report is bounded in time: agents not read in time count as incomplete.
A save that sends runtimes or profiles also checks the Kubernetes placement rules across runtime entries and profiles: placement on a runtime entry that is not Kubernetes, or on a profile whose runtime entry is not; a profile with placement whose runtime entry does not exist (in the runtimes or among the built-in ones); and a profile whose scion.dev/pool node selector differs from its runtime entry’s. As for the placement fields, a problem the save introduces is refused with 422 validation_error, naming the settings path, and one already stored is returned as a save warning. A body that spells runtimes or profiles in another case is refused with 422 validation_error; one that repeats a member name is refused with 400 invalid_request. The two sections are written in one transaction that also checks the revision of a section the save leaves out: if another save changed it in the meantime, the save answers 409, lists that section in conflicted with "read": true, and writes nothing. Resetting runtimes or profiles (DELETE /api/v1/admin/server-config/sections/{name}) runs the same checks on the pair the reset leaves: a problem the reset introduces is refused with 422 validation_error in the same form as a save’s (the settings path in details.field, every finding in details.errors), and a concurrent change of the other section answers 409.
Save report
Section titled “Save report”A database-backed save (PUT /api/v1/admin/server-config) returns a reload object:
| Field | Meaning |
|---|---|
applied |
Database sections written, plus, on a workstation Hub, the settings.yaml changes applied live. |
applied_keys |
Changed Layer-1 keys (koanf paths, for example server.hub.soft_delete_retention) that the running Hub now uses. |
pending_restart |
Changed Layer-1 keys flagged restart_required: saved, but used only after a restart. |
skipped |
Changed Layer-1 keys that were saved but are not in use, as a list of {"key", "reason"} objects. They are not in applied_keys. |
file_applied |
Workstation Hubs: the settings.yaml changes applied live (also listed in applied). |
requires_restart |
Workstation Hubs: the settings.yaml changes that need a restart. |
A key is listed in applied_keys or pending_restart only when its value changed. An echoed server.hub.hub_name equal to the hub name in effect (the bootstrap value when no endpoints row is stored yet) is not a change and is not listed. GitHub App fields other than app_id are listed under neither heading while the saved github_app section has no app_id, because they are not used then; a change to app_id itself is listed in pending_restart. When the running Hub cannot use a saved change, the key is listed in skipped instead of applied_keys: server.notification_channels when a notification channel fails validation (the reason gives the channel’s position; the channels that pass validation are in use), and the changed server.federation.* keys when the federation configuration is not installed: it fails validation, its authenticator cannot be built, or the test admin tier is on. Rarely, the Hub that serves the save cannot include a section it just wrote in its own update (it could not re-read the database after the write, or newer writes kept overtaking the re-read); that section’s changed keys are then listed in skipped with the reason Saved; this Hub applies it on its next settings refresh. The change is saved, and the Hub picks it up within the usual propagation time. A reason never repeats a configured value. The Server Config page lists these keys under Saved but not applied.
A 409 revision_conflict response covers the sections written before the conflict:
| Field | Meaning |
|---|---|
applied |
Revisions of the sections this request wrote. |
unchanged_revisions |
Current revisions of sections the request sent unchanged; they were not written. |
conflicted |
The section that conflicted, with its expected and current revision. |
applied_keys, pending_restart, skipped |
As in reload, for the sections written before the conflict. |
When a save hits 409 revision_conflict after earlier sections were written, the Server Config conflict banner also lists the keys saved before the conflict: applied_keys under Applied now, pending_restart under Saved; applies after restart, and skipped under Saved but not applied.
Layer-0 edits by deployment mode
Section titled “Layer-0 edits by deployment mode”Whether Layer-0 settings can be edited through PUT /api/v1/admin/server-config depends on the deployment mode, not on the database driver. GET reports it as layer0_editable.
- Hosted Hubs (started with
--hosted, orserver.mode: hostedinsettings.yaml/SCION_SERVER_MODE=hosted): Layer-1 keys are written to the database. A Layer-0 field the body carries is rejected with422 layer0_rejectedwhen its value differs from whatGETreturns, an explicitfalse,""or0included. Fields with no database home (such asschema_version,active_profile,workspace_pathorauto_inject_gcloud_adc) follow the same rule with422 unclassified_keys_rejectedor422 unpersisted_keys_rejected. An unchanged value is ignored, so sending back the bodyGETreturned is a200that writes nothing. Deployment tooling ownssettings.yaml. - Workstation Hubs (the default mode): Layer-1 keys are written to the database. Layer-0 keys and keys with no database home are written to
settings.yaml.log_levelis applied immediately and the co-located broker reloads its runtime. Every other key that changed in the file is listed in the response’sreload.requires_restart, andfile_keyslists every key that changed in the file.
Layer-1 keys written to the database are merged section by section onto the stored row: a key the body leaves out keeps its stored value, an explicit null clears it, and a save that races another write to the same section returns a 409 Conflict error instead of overwriting it.
Federation settings are sent as the top-level federation object of the PUT body, in database and settings.yaml modes alike. On a database-backed Hub, federation sent under server.federation is refused with 422 unless it repeats what GET returns. {"federation": null} is refused in every mode with 422 validation_error and details.keys: ["federation"]; nothing is saved. To return the federation settings to their bootstrap values, use DELETE /api/v1/admin/server-config/sections/federation (see Reset to Bootstrap). To clear single fields, send them explicitly (false, [] or ""). GET reports the federation settings in effect in both places it shows them, the top-level federation object and server.federation. On a database-backed Hub these are the stored values (or the bootstrap values while no row is stored), never a settings.yaml value the stored row replaced. A stored federation section that sets enabled or trusted_issuers is shown even when it is disabled and has no trusted issuers, with its other fields such as algorithms or the intervals.
server.mode must be empty (workstation), workstation, hosted or production (the legacy name for hosted). Values are case-sensitive. The server refuses to start with any other value (a typo used to mean workstation), and a workstation save that sets another value is rejected with 400. To fix a bad value, edit server.mode in settings.yaml or SCION_SERVER_MODE; --hosted selects hosted mode but does not make a bad value valid.
On a workstation Hub the file edit follows the request body field by field:
- Each field the body carries is written exactly as sent, an explicit empty value (
"",false,0,[]) included. Onlynullremoves a key or block. - The Hub then compares the settings it would load from the edited file with the settings it loads today (the server config the Hub builds, the typed settings, and the
active_profile/workspace_pathmerge over the built-in defaults). Fields whose effective value does not change are not written, not listed infile_keysand not reported as needing a restart. If nothing effective changes, the file is not written, so saving an unchanged form writes nothing. For example,""for a key the file does not have changes nothing and is skipped.[]forserver.hub.cors.allowed_originsreplaces the default list with an empty one and is written. - A field whose value in the edited file would not be what was sent is rejected with
422 unsaved_keys_rejectednaming it, and nothing is saved. - Fields the body leaves out are kept.
- Creating a
corsblock (server.hub.corsorserver.broker.cors) without sendingenabledalso writesenabledwith its current value. Inside the block a missingenabledmeans off. A missing block means on for the Hub and off for the broker (broker CORS is off by default), so adding an origin never silently changes whether CORS is on. server.broker.broker_idandbroker_tokenare written by the Hub itself. A save may send them back unchanged (the token as********), but changing or clearing them is rejected with422 hub_owned_keys_rejected. Anullon a block that contains them ({"server":null}or{"server":{"broker":null}}) removes everything else in that block and keeps them.- If
settings.yamlhas noserverkey and a deprecatedserver.yamlis in effect (in~/.scion, in the server’s--configpath, or in its working directory), the Hub still reads its server configuration fromserver.yaml. A save that would create theserverblock insettings.yaml(and so drop everyserver.yamlsetting at the next start) is refused with409 legacy_server_yaml. Move the contents ofserver.yamlunder a top-levelserver:key insettings.yaml, removeserver.yaml, then save again. - The admin UI sends only the fields you changed since the page loaded, so defaults the form shows for keys the file does not have are never written.
- The file is edited in place: comments, key order and keys the Hub does not know survive. A
settings.yamlthat uses YAML anchors or aliases on an edited path cannot be edited in place, and the save is rejected with422(edit that file by hand).
Some workstation fields are overridden at every start by the workstation defaults and scion server start flags, so a value in settings.yaml has no effect: server.broker.enabled (--enable-runtime-broker), server.hub.host (--host), server.auth.dev_mode (--dev-auth), server.storage.provider (--storage-bucket) and server.secrets.backend. The admin UI shows them read-only with a “Set by workstation startup defaults / server flags” badge. In workstation mode the runtime broker listens on 127.0.0.1 unless server.broker.host is set to another address; --host does not move it, and 0.0.0.0 counts as unset.
A workstation save that touches both destinations is checked as a whole first. Any validation failure, for either part, writes nothing. The Hub then takes its settings-file lock and prepares the new settings.yaml in memory, writes the database sections, and writes the file last, atomically, before releasing the lock. These writers in the same server process take the same lock for their whole read-modify-write, so neither of two concurrent changes is lost: Runtime Broker registration writing its ID and token, the identity and workstation-settings endpoints, the plugin-registration writes made by the integrations API, and the hub-sync cleanup of project settings. The loopback runtime and image-registry endpoints (PUT /api/v1/system/runtime and PUT /api/v1/system/registry) write the database profiles and endpoints sections, because those are Layer-1 keys. The lock also works across processes, so a scion CLI command that writes settings while the server runs waits for it too. It is a file lock on .settings.lock in the settings directory. A writer that cannot get it within 10 seconds fails with an error that says to try again, and writes nothing. If a database write fails (for example 409 revision_conflict), settings.yaml is left unchanged. Database sections written before the failure stay written, as for any save that touches several sections, and are listed in applied. Only if the final file write fails are the database changes saved without the file changes; the 500 response lists them.
Of the system routes, only PUT /api/v1/system/identity and PATCH /api/v1/system/workstation-settings require a Hub admin (the hub.config.update permission, held by the hub-admin and super-admin roles; the workstation’s dev user holds it as super-admin); other signed-in callers get 403.
On every Hub, a key the API cannot store anywhere (an unknown or misspelled key, or a field with no settings home) is rejected with 422 unpersisted_keys_rejected, unless its value is unchanged from what GET returns. Keys that are set only in the settings file or the environment and are never writable through the API (such as server.hub.gcp_project_id, which is also omitted from GET /api/v1/admin/server-config and env_overrides and has no UI field; see Hub server settings) are rejected with 422 validation_error in every mode, with the refused keys in keys, even when sent as "" or null. A save never answers 200 for a key it dropped. An empty body is a 400.
Seeded/Managed Lifecycle
Section titled “Seeded/Managed Lifecycle”Each settings section in the database has an origin that tracks whether it was populated by the system or edited by an admin.
Seeded Sections
Section titled “Seeded Sections”When a hub first boots with a postgres database, it populates the settings database from the bootstrap configuration. These sections are marked as seeded — they track the deployment configuration.
Seeded sections:
- Re-sync from bootstrap on every hub restart (if the deployment config changes, the database updates)
- Show a caption in the admin UI: “Tracking deployment configuration — will re-sync on restart until edited”
- Skip writes when the bootstrap value hasn’t changed (no unnecessary revision bumps)
Managed Sections
Section titled “Managed Sections”When an admin edits a setting in the UI and saves, the section becomes managed. Managed sections:
- Are owned by the database — bootstrap changes no longer overwrite them
- Show standard metadata in the admin UI (source, revision, last updated by/at)
- Display superseded notices when the deployment config differs from the database value
Reset to Bootstrap
Section titled “Reset to Bootstrap”A managed section can be reset to tracking mode via the “Reset to bootstrap” button in the admin UI. This:
- Deletes the database row for that section
- The section falls back to bootstrap values
- On next boot, the section is re-seeded from current deployment config
The endpoint is DELETE /api/v1/admin/server-config/sections/{name}; from the CLI, scion hub settings unset --section NAME --reset. The experiments section is refused there: use scion hub experiments reset --all, which is compare-and-set.
Environment Variable Namespaces
Section titled “Environment Variable Namespaces”SCION_SEED_* (New)
Section titled “SCION_SEED_* (New)”Use SCION_SEED_* to provide initial/default values for operational settings. Seeds are overridable — admins can change them in the UI, and the database value takes precedence.
Seeds re-sync on restart for sections that haven’t been admin-edited (seeded sections). Once a section is managed, the seed value is superseded.
Mapping rules:
server.*keys:SCION_SEED_SERVER_+ uppercase snake_case suffix- Non-
server.*keys:SCION_SEED_+ uppercase snake_case suffix
| Setting | Seed Environment Variable |
|---|---|
server.hub.admin_emails |
SCION_SEED_SERVER_HUB_ADMINEMAILS |
server.hub.public_url |
SCION_SEED_SERVER_HUB_ENDPOINT |
server.auth.user_access_mode |
SCION_SEED_SERVER_AUTH_USERACCESSMODE |
server.auth.authorized_domains |
SCION_SEED_SERVER_AUTH_AUTHORIZEDDOMAINS |
server.auth.default_user_role |
SCION_SEED_SERVER_AUTH_DEFAULTUSERROLE |
server.hub.auto_suspend_stalled |
SCION_SEED_SERVER_HUB_AUTOSUSPENDSTALLED |
server.hub.auto_suspend_idle |
SCION_SEED_SERVER_HUB_AUTOSUSPENDIDLE |
server.hub.soft_delete_retain_files |
SCION_SEED_SERVER_HUB_SOFTDELETERETAINFILES |
image_registry |
SCION_SEED_IMAGEREGISTRY |
telemetry.enabled |
SCION_SEED_TELEMETRY_ENABLED |
server.github_app.webhooks_enabled |
SCION_SEED_SERVER_GITHUBAPP_WEBHOOKSENABLED |
SCION_SERVER_* (Existing)
Section titled “SCION_SERVER_* (Existing)”SCION_SERVER_* remains the authoritative namespace for Layer-0 (bootstrap) settings:
| Setting | Environment Variable |
|---|---|
server.hub.port |
SCION_SERVER_HUB_PORT |
server.hub.host |
SCION_SERVER_HUB_HOST |
server.database.driver |
SCION_SERVER_DATABASE_DRIVER |
server.database.url |
SCION_SERVER_DATABASE_URL |
server.auth.dev_mode |
SCION_SERVER_AUTH_DEVMODE |
server.secrets.backend |
SCION_SERVER_SECRETS_BACKEND |
server.mode |
SCION_SERVER_MODE |
server.log_level |
SCION_SERVER_LOGLEVEL (applied at startup and on a workstation server-config save, see below) |
server.log_level, or SCION_SERVER_LOGLEVEL, is applied when the server starts and again when a workstation admin server-config save that changes server.log_level re-reads the config. Clearing it reverts to info. --debug and SCION_LOG_LEVEL take precedence over it.
SCION_SERVER_* Deprecation for Layer-1
Section titled “SCION_SERVER_* Deprecation for Layer-1”The deprecation notice also appears in the admin UI when deprecated variables are detected.
Maintenance Mode
Section titled “Maintenance Mode”Maintenance mode set through the admin API (PUT /api/v1/admin/maintenance) is stored in the Hub database on every driver, so it survives restarts. Until a maintenance row exists, the Hub reports and builds on the state it started with (SCION_SERVER_ADMIN_MODE or settings.yaml admin_mode).
Break-glass by deployment mode
Section titled “Break-glass by deployment mode”- Hosted Hubs: maintenance is cluster-consistent. A database row is propagated to all replicas and cannot be overridden by per-node environment variables or
settings.yaml. - Workstation Hubs: starting the Hub with
SCION_SERVER_ADMIN_MODE=trueoradmin_mode: trueinsettings.yamlkeeps it in maintenance for that run, even when the database row says otherwise.GET /api/v1/admin/maintenancethen reports"break_glass": trueand the admin UI shows a notice. Restart without the setting to hand control back to the row.
While a break-glass is active, a save from the maintenance page updates the stored row (for example its message) but cannot take this run out of maintenance. When a row already exists, a save that does not set enabled keeps the row’s own on/off value, so the Hub leaves maintenance after a restart without the break-glass. When no row exists yet, such a save stores the current state, which is “on” during a break-glass. Turn maintenance off explicitly, or the Hub stays in maintenance after that restart.
admin_mode: true in settings.yaml can only be cleared by editing the file: the admin API stores maintenance in the database, not in settings.yaml.
The Maintenance page shows the message blocked users see (message in GET /api/v1/admin/maintenance; the Hub’s default, System offline for maintenance, while none is saved) and lets you edit it whether or not maintenance mode is on. Save message sends only message, so the mode is unchanged. Use the default message sends "message": "", which clears the saved message.
Profiling
Section titled “Profiling”The profiling section holds operational switches for in-app profiling. It is stored only in the Hub database: it has no settings.yaml key and no environment variable, and it is not seeded. A change applies to every replica through the usual settings propagation.
| Key | Type | Default | Effect |
|---|---|---|---|
readiness_marks |
boolean | false |
The web client writes readiness marks (User Timing marks for agent data arrival, first visible rows and graph ready). See readiness marks. |
- Changing it:
PUT /api/v1/admin/profilingwith{"readiness_marks": true}(orfalse, ornullto reset). It needs a hub administrator on an interactive session; user access tokens are refused. There is no admin UI card.GET /api/v1/admin/profilingreturns the current value and the section revision. - Reading it:
GET /api/v1/profilingreturns{"readinessMarks": true|false}to any signed-in caller. - In the web client: while it is on, the page shell’s initial data carries
"readinessMarks": truefor a signed-in user, the same value that user reads fromGET /api/v1/profiling. Signed-out pages, and every page while it is off, are served exactly as without the setting. The client reads the value once per page load, so a change applies on the next full load. - Off, the client makes no
performance.markcalls and the server adds nothing to the shell.
Chat Retention
Section titled “Chat Retention”The chat_retention section bounds how long the Hub keeps chat messages (pkg/config/opsettings/chat_retention.go). Like profiling, it is stored only in the Hub database: it has no settings.yaml key and no environment variable, and it is not seeded. An absent row resolves to the compiled defaults (enabled: false, messages_retention_days: null, deleted_conversation_grace_days: 30, default sweep tuning), and a stored document that fails to parse or validate fails closed (enabled: false, no window). It is gated by the hub.chat_retention experiment (off by default). The background "chat-retention" singleton purges messages only while hub.chat_retention is on and enabled is true (enabled: false doubles as a legal hold). Each tick first purges the messages of conversations and topics soft-deleted more than deleted_conversation_grace_days ago, whatever the window. When messages_retention_days is set, it then purges messages older than the window. With enabled: true and no window, only the deleted-conversation purge runs.
| Key | Type | Default | Constraints | Effect |
|---|---|---|---|---|
enabled |
boolean | false |
— | Turns the retention sweep on while hub.chat_retention is on. While false, nothing is purged. |
messages_retention_days |
integer or null |
null (keep forever) |
7–36500 when set |
Hub-wide age window in days (ChatRetentionMinDays = 7, ChatRetentionMaxDays = 36500). When null (absent in the stored document), messages are kept forever, except those of deleted conversations and topics. |
deleted_conversation_grace_days |
integer | 30 |
1–36500 |
Days after a conversation or topic is soft-deleted until its messages are purged, even when the window is longer or null. |
sweep.chunk_size |
integer | 200 |
25–1000 |
Largest number of messages deleted in one transaction. |
sweep.pause_ms |
integer | 500 |
0–10000 |
Pause after each chunk, in milliseconds. |
sweep.tick_budget_seconds |
integer | 30 |
1–40 |
Wall time one tick may run, in seconds. |
sweep.tick_interval_minutes |
integer | 10 |
1–1440 |
Time between ticks, in minutes. |
- Permissions and credentials:
GET /api/v1/admin/chat-retentionandPUT /api/v1/admin/chat-retentionare both Hub-admin routes (RouteID: "admin.chatRetention",Classification: RouteHubAdmin, permissionhub.config.updateonhub) restricted to interactive sessions (SessionOnly: ReasonIrreversibleCascade); user access tokens and test-admin identities are refused. While thehub.chat_retentionexperiment is off, bothGETandPUTreturn404 not_found(Route not found); unsupported HTTP methods return405whether or not the experiment is on. - Response (
GETandPUT): Returnsenabled,messages_retention_days(nullwhen no window is set),deleted_conversation_grace_days,sweep(chunk_size,pause_ms,tick_budget_seconds,tick_interval_minutes),last_applied_cutoff(the UTC cutoff the sweep last reached, ornulluntil it reaches one), andrevision(the section revision forexpected_revision). PUTsemantics and error codes: Omitted fields are unchanged; explicitnullresets a field to its compiled default (enabled: false,messages_retention_days: null,deleted_conversation_grace_days: 30, or default sweep tuning;"sweep": nullresets all four sweep fields). Changingenabled,messages_retention_daysordeleted_conversation_grace_daysrequiresexpected_revision(400 invalid_requestif missing); a staleexpected_revisionor concurrent modification returns409 revision_conflict. Unknown JSON fields, malformed JSON, or out-of-range values return400 invalid_request(or400 validation_failedon schema validation). Without database-backed operational settings,PUTreturns501 not_implemented. Every change is written with compare-and-set on the section revision and recorded in the mutation audit (hub_setting_updateonchat_retention, executoradmin-chat-retention) before the write; if the audit record cannot be written, the save is refused with500 internal_errorand settings are not changed (hub_setting_update_not_appliedis recorded if the settings write itself fails).- Run history (
GETonly): The response also carriesruns(the 10 most recentchat_retention_runsrows, newest first:id,trigger(sweep,manualorpreview),dry_run,started_at,finished_at,status(running,ok,partialorfailed),actor(the user ID of the administrator who started a manual run or preview),error,messages,projects,totals(the counts summed over every project),messages_retention_daysanddeleted_conversation_grace_days),last_preview(the most recent preview in the same shape, ornull), andpurged_since_preview(messages deleted by the sweep and manual runs that finished after the last preview started;0without a preview). Runs hold totals only, never a message ID, content or per-project detail. A sweep run row stays open and accumulates for 24 hours; rows are kept for 365 days. - Admin UI: While the
hub.chat_retentionweb flag is on, Admin → Server Config renders the Chat retention panel (<scion-admin-chat-retention>). Below the settings it shows the run history, the last preview (“Last preview: N messages in M projects,
Preview and manual run
Section titled “Preview and manual run”Two routine maintenance operations, run from Admin → Maintenance, the Run preview button, or POST /api/v1/admin/maintenance/operations/<key>/run. Like every maintenance operation they need hub.maintenance.execute (Hub administrators). While hub.chat_retention is off they are not listed and every route under them answers 404 not_found.
chat-retention-previewwalks the same selections as the sweep, with the same chunk size and pause, and deletes nothing: first the messages of conversations and topics deleted more thandeleted_conversation_grace_daysago, then, when a window is set, the messages older than the window (a message counted in the first phase is not counted again). Pending and deferred messages are skipped, as the sweep skips them. It works whether or notenabledis on. The optional parametermessages_retention_dayspreviews that window instead of the saved one; it must be a JSON string of whole days from7to36500({"params":{"messages_retention_days":"365"}}), and any other value is refused with400 invalid_param. The result is achat_retention_runsrow withtrigger: preview,dry_run: true, the counts per project and phase, the policy and the actor; the run log lists the counts per project.chat-retention-runruns the sweep once with the saved settings, without the per-tick time budget (the pause between chunks stays, so a large backlog takes hours; a run stops after 24 hours and records what it did). It waits for a running sweep tick to finish first and then holds the same lock, so ticks skip while it runs. Its main walk starts at the beginning, as a preview’s does (the sweep’s cursor is reset, as the daily reset does), so it also deletes messages a tick skipped while they were pending or deferred and that were dispatched since. It records atrigger: manualrow with the actor. It takes no parameters and is refused with409 chat_retention_disabledwhile the sweep is not enabled.
On an unchanged database, a preview’s counts equal what a following manual run deletes. A preview that reaches its time limit is recorded as partial with the counts so far; one whose selection fails, including a selection that times out, is recorded as failed with the error. A preview or manual run that a Hub restart interrupts is marked failed (“interrupted”) when the Hub starts again, as interrupted maintenance runs are. With several Hub replicas, a manual run is marked interrupted only when no replica holds the chat retention lock (or when it started longer ago than any run can last); a preview still running on another replica may show as failed until it finishes and records its own result.
Failed agent messages
Section titled “Failed agent messages”messaging.failed_message_retention_days (default 7, minimum 1; GET/PUT /api/v1/admin/messaging) is how many days a message to an agent whose delivery failed is kept before the Hub deletes it. This purge is not gated by hub.chat_retention or enabled. It runs on its own schedule with the chat retention cascade and pacing (chat_retention.sweep), deletes only failed messages addressed to agents (never user: recipients), and deletes the bytes and then the metadata row of each attachment it leaves with no message; when a byte delete fails, the row stays for the attachment garbage collection. While chat retention is enabled, its counts are added to the open sweep run under failed.
Jump to a purged message
Section titled “Jump to a purged message”Jumping to a message that was purged (GET /api/v1/chat/conversations/:key/messages?around=<id>) returns 404 with code message_unavailable, the same response as for a message that never existed, and the web shows “That message is no longer available.” A reply whose quoted message was purged shows “[no longer available]”.
Runbook: enabling retention on a large hub
Section titled “Runbook: enabling retention on a large hub”- Take and verify a database backup. Deletes cannot be undone, and the Hub keeps no export of purged messages.
- Run a preview (
chat-retention-preview, or Run preview in the panel) of the window you intend to reach, and check the counts per project in its run log. - Enable the sweep with a long window first, for example 365 days, and let the backlog drain. The sweep deletes at most about 12,000 messages per tick at the default tuning (about 72,000 an hour), so a multi-million-message backlog takes days; the panel’s “Backlog draining” estimate shows the progress. A
chat-retention-rundrains faster, without the tick budget, but still pauses between chunks. - Shorten the window in steps once the backlog is drained, previewing each new window first.
SQLite does not shrink its database file after deletes: the freed pages are reused by new data, but the file keeps its size. Reclaiming the space needs an offline VACUUM of the database while the Hub is stopped (with free disk space of about the database’s size), after a backup.
HA Bootstrap Guidance
Section titled “HA Bootstrap Guidance”For high-availability deployments with multiple hub replicas:
-
Initial setup: Use
SCION_SEED_*environment variables to configure operational settings across all replicas. All replicas share the same bootstrap configuration. -
After first admin edit: Once an admin edits a setting in the UI, the section becomes managed. The database value propagates to all replicas within ~2 seconds via the event system. No per-node env configuration is needed.
-
Updating seeds: To change the default for a seeded (unedited) section, update the
SCION_SEED_*variable and restart the replicas. The new value syncs to the database on boot. -
Managed sections: For sections that have been admin-edited, changing the
SCION_SEED_*variable has no effect — the database value wins. A “superseded” notice in the admin UI shows which deployment values differ from the database. -
Resetting: To return a managed section to tracking mode, use “Reset to bootstrap” in the admin UI. The section reverts to the seed value and will track future seed changes.
From the CLI
Section titled “From the CLI”scion hub settings reads and changes the same settings as the admin page, through the same endpoints (see the CLI reference):
scion hub settings get # sections, sources and revisionsscion hub settings get --section agent_defaultsscion hub settings set default_max_turns 200scion hub settings set runtimes.k8s.namespace agentsscion hub settings unset server.hub.monitoring_dashboard_urlscion hub settings schema # sections and keysEach set or unset sends the revision of every section it changes in expected_revisions, as read just before the save, so a concurrent change is a 409 listing each section’s expected and current revision rather than a silent overwrite. The save report is printed: keys in effect now, keys that apply after a restart, keys saved but not in use, and warnings. Project defaults and experiment overrides have their own commands, scion hub project-defaults and scion hub experiments.
Admin UI
Section titled “Admin UI”The admin settings page (/admin/server-config) is layer-aware, permission-gated, and has been restructured for improved usability:
- Permission-Gated Tabs: Settings page tabs are dynamically gated by the caller’s actual resource-level permissions. For example, a role with template-only permissions (like
template.*) sees only the Templates tab, with other administrative tabs hidden. Nav and route guards use granular, per-item permission checks driven by the/api/v1/admin/statuspermissions array. - Hosted Hubs: Layer-0 fields are read-only with a “Managed via deployment configuration” badge. Layer-1 fields are editable.
- Workstation Hubs: Fields pinned by
SCION_SERVER_*environment variables are read-only with a “Set via environment variable” badge, and fields that server start flags override with a “Set by workstation startup defaults / server flags” badge. Other fields are editable: Layer-1 fields save to the database, Layer-0 fields tosettings.yaml. - Default Agent Role, Default Maximum Agent Role and Default Harness Auth are shown read-only on every Hub. The Hub cannot save or apply them from its settings database yet (ptone/scion#3067).
Layout Structure
Section titled “Layout Structure”To streamline management of complex environments, the General settings tab is restructured into three high-density, focused cards:
- General Card: Houses central Hub identity, registration endpoints, and core server configurations. At the top of the General tab, the Scion Server Version section displays a read-only
Scion version:summary line with the running version and effective channel from/api/v1/admin/maintenance/update-status(falling back to the server-config build version without a channel whenupdate-statusis unavailable), alongsideGit Commit,Build Time, and a Manage updates link to/admin/maintenance; updates and rebuilds are started on the Maintenance page only. - Agent Defaults Card (with dedicated sub-tabs):
- Configures default resource constraints (
max_turns,max_duration, limits). These fields can be cleared to blank in the UI, which persists them asnullin the Hub settings database instead of preserving their previous values, allowing administrators to remove default constraints entirely. A change to the limits (default_max_turns,default_max_model_calls,default_max_duration) or todefault_resourcesreaches existing agents at their next start or restart, unless the agent, its template or (for resources) its broker profile sets the value. Agents created before this behaviour was added keep the limits they have stored. See Where hubagent_defaultsrank. - Introduces Default Model (
default_model), Default Thinking Level (default_thinking_level), and Default Agent Role (default_agent_role) fields directly into the agent default pipeline (with the default agent role updated frombaselinetofullfor usability). - Introduces Default Runtime Broker (
default_runtime_broker), a hub-level default that participates in the broker resolution cascade. When a project has no default broker and no broker is explicitly requested, the hub-level default is used if the broker is online and dispatchable. - Default Timezone (
agent_defaults.default_timezone): an IANA zone picker for theTZsent to agent containers that have no pinned timezone and noTZin the Hub environment-variable store. Empty means no default, so those containers use the image default (UTC). Invalid names andLocalare rejected with422. It sets agent containers only, not how the web UI displays times. See Times and Timezones for the full chain. - Adds a hub-level Default GCP Identity Mode (
agent_defaults.default_gcp_identity_mode) with an optional service account picker (default_gcp_identity_service_account_id). It is the fallback for new agents when neither the create request (not applicable to a scheduled dispatch, which has none) nor the project’s default GCP identity sets a mode — including agents dispatched by a schedule, which follow the same fallback ladder as interactive/API creates. Passthrough applies only on the Hub’s own embedded (co-located) broker, as recorded by the Hub server at startup; elsewhere, including any Kubernetes profile, the hub default is left unset and the agent gets the runtime default (Block, or Passthrough on Kubernetes). A broker label cannot opt a broker in. Assign accepts only a verified hub-scoped service account and requiresgcp_iam_check_mode: enforce. Other values are rejected with422. A save that sends either key checks the pair the Hub will run with. Insettings.yamlmode, a key that aSCION_SERVER_*variable sets is taken from the environment for that check, because the environment value is the one in effect; the save still writes only the values sent. When the pair stored insettings.yamlwould be refused on its own (for exampleassignwith no account while the environment sets the mode toblock), the save is accepted and itswarningssay so, because that stored pair would apply once the variable is removed. On the Server Config page the service account picker follows that mode too. The setting is stored in the Hub settings database on every driver (SQLite or Postgres). It applies to new agents without a restart and persists across restarts. On the Kubernetes runtime an explicit Block (from any rung, including this one) runs the agent pod as a zero-privilege ServiceAccount; see block on Kubernetes. An agent with no mode configured anywhere gets Passthrough on Kubernetes. A hub default of Passthrough is denied for Kubernetes profiles, so those agents get the same runtime default. See Hub-Default GCP Identity. - Houses the Telemetry Toggle, which has been moved to this card to keep telemetry configuration closely aligned with operational defaults.
- Configures default resource constraints (
- Project Default Settings Card: Configures platform-level default annotations and behaviors for newly created projects.
- Quotas Card: Houses the Enforce broker agent quotas switch (
quotas.enforce_broker_quotas), on by default. When turned off, agent creates on a broker are no longer rejected oncemax_agents_per_brokeris reached — usage is still counted (reservations, release, reconcile and backfill are unaffected), so the Admin → Quotas usage page keeps showing the true count (e.g. “31 / 16”) while it is off, and turning the switch back on makes the next over-cap create return429immediately with no reconcile wait. The switch only affectsmax_agents_per_broker; other quota limits (such asmax_agents_per_project) are always enforced. This is a normal Layer-1 section on every driver (PUT /api/v1/admin/server-config, no restart, propagates to replicas within 60s). The card links to Admin → Quotas, where the hub-widemax_agents_per_brokerdefault itself is edited.max_agents_per_projectcounts the agents in a project that hold capacity: agents that are starting or running, and agents whose stop is queued for an offline broker. Stopped, suspended, errored and halted agents do not count, so starting, resuming, restarting, waking or reincarnating one re-checks the limit and is refused with429 quota_exceeded(naming the project, its count and its limit) when the project is full. The total is shared by everyone in the project, but the cap it is compared against is the one resolved for the agent’s creator: an entitlement binding for that user or one of their groups, else the hub-wide default. Two creators with different caps can therefore get different answers at the same project total. This is intended, because an admin raise applies to one principal. An agent created by another agent is judged on the creating agent’s cap: a user’s raise does not carry over, but the agent’s group bindings apply. The Admin → Quotas row for this limit shows the count summed across all projects. - Agent Secrets Card: Houses the Restrict agent-written secrets to profile scope switch (
agent_secrets.user_scope_only), off by default. When on, the hub rejects every agent-originated secret write that resolves to project scope — including an empty scope, which defaults to project — with403 secret_scope_restricted, whatever harness, flag, orforceis used. This covers the web terminal’s Capture Auth button (which disables the Project option and preselects Profile while the setting is on) and ad-hocsciontool secret setcalls alike. User-originated project writes (web UI,scion hub secret set --project, chat and Discord apps) and agent user-scope writes are unaffected. Existing project secrets are not removed or migrated — turning the setting on stops new agent writes only; use the project secrets UI to clean up existing ones. This is a normal Layer-1 section on every driver (PUT /api/v1/admin/server-config, no restart, propagates to replicas within 60s).
Runtimes & Profiles Tab
Section titled “Runtimes & Profiles Tab”To support complete infrastructure configuration directly from the Web Dashboard, the admin page features a dedicated Runtimes & Profiles tab. This tab provides full CRUD (Create, Read, Update, Delete) editors for core execution settings:
- Runtimes Editor: Type-aware input fields tailored to specific runtime types:
- Docker & Podman: Fields for daemon sockets, host networks, and storage paths.
- Kubernetes (GKE): Fields for cluster endpoints, namespace scoping, service account bindings, and PVC volumes, including Shared dir storage class (
shared_dir_storage_class, placeholder “e.g. standard-rwx”, hint “Kubernetes only. Empty uses the cluster default.”) and Shared dir size (shared_dir_size, placeholder “e.g. 10Gi”, hint “Kubernetes only. A Kubernetes quantity, such as 10Gi or 1Ti.”). A Kubernetes placement and identity section editsnode_selector(label key and value rows),tolerations(key, operator Equal or Exists, value, effect),runtime_class_name,priority_class_name,kubernetes_service_account_mappings(GCP service account email to Kubernetes ServiceAccount rows) andkubernetes_block_service_account. The page checks them as you type with the hub’s rules: a value you set that is invalid blocks Save & Reload; an invalid value already stored that you leave as it is shows a warning and does not block Save. - Cloud Run: Fields for service endpoints, memory limits, and CPU allocation.
- Profiles Editor: Configures resource constraints (limits), target container image registries, and specific template overrides. Supports a native Runtime Profile Selector when launching or configuring agents. On a profile whose runtime is not
docker,podman,container, orcloudrun, the profile editor also shows Shared dir storage class (shared_dir_storage_class) and Shared dir size (shared_dir_size), with inline Kubernetes quantity validation onshared_dir_size(“shared_dir_size must be a positive Kubernetes quantity, such as 10Gi or 1Ti.”) that blocks Save & Reload when invalid and clears the stored key when emptied. - Saving runtimes: a save sends each runtime entry as edited, with
nullfor each field you cleared, while the Hub keeps any stored field the page does not show. Changing a runtime’s type to or from Cloud Run clears the other type’s fields; when any of them hold a value, the page names them and asks you to confirm first. See theruntimesmerge rule and Kubernetes placement and identity checks. - Resource values: CPU, memory and disk fields (Agent Defaults → Resources and each profile) are checked as you type. A memory or disk value needs a unit:
16is 16 bytes, so write16Gi. See Resource quantities. - Harness Configs Editor: Features a rich JSON textarea for writing and editing raw harness properties directly, making it easy to tune model parameters or customize env environments.
In highly available (HA) Postgres deployments, all configurations edited through this tab are stored in the database’s hub_settings table as whole-map JSONB documents. This ensures edits propagate immediately to all replicas, support optimistic locking (CAS), and are never silently dropped or ignored on server PUT actions.
Furthermore, these database-backed settings are wired directly into the runtime broker consumption path. During agent dispatch, both the Scion Hub and the Runtime Broker resolve runtime execution environments and resource profiles dynamically from these persisted database settings. This eliminates the need to distribute or maintain on-disk configuration files (such as local settings.yaml files) on individual broker nodes, ensuring a centralized, real-time control plane.
Navigation Updates
Section titled “Navigation Updates”- Message Broker Integration: The Message Broker configuration has been consolidated and moved from the General tab to the Hub Server tab, grouping external integrations and network-bound transports in one logical place.
- Agent Start: The Hub Server tab’s lifecycle card has one field each for the agent start timing (
server.hub.start_claim_lease_ttl,start_max_duration,start_unconfirmed_hold,start_create_unconfirmed_hold). An empty field uses the default (90s, 12m, 17m and 16m); the hints give each allowed range. They save with the lifecycle section. The Hub refuses an unparseable or out-of-range value with422 validation_errornaming the key, in database andsettings.yamlmodes alike; a save that sends none of the four keys is not checked against a stored value. - Hub name (
server.hub.hub_name) has no field on the page. Set it insettings.yaml(orSCION_SERVER_HUB_HUBNAME), or throughPUT /api/v1/admin/server-configon a database-backed Hub. A saved change applies live: it relabels the Hub’s own log records and new Secret Manager secrets (scion-hub-name), and the agent log views search Cloud Logging under the new name. Logs written under the old name stop appearing in the agent log views, and agents keep the old name (SCION_HUB_NAME) until the Hub restarts (ptone/scion#7098), so restart the Hub after a rename. All nodes of a Hub should share one name. - Hub Observability Export: The Hub Server tab has one switch each for the Hub’s own metrics and trace export (
server.hub.observability.*). Both apply after a restart. An unset switch shows the state in effect and keeps the earlier rule; Use default clears a switch that is set. The section also shows each exporter’s state at startup and, read-only, the export project with its source (env,fileormetadata). Only a Hub admin in an interactive session can change the switches; a user access token is refused. See Observability export.
Visual Indicators
Section titled “Visual Indicators”| Indicator | Meaning |
|---|---|
| 🔒 Managed via deployment configuration | Layer-0 field on a hosted Hub — not editable |
| 🔒 Set via environment variable | Field pinned by SCION_SERVER_* on a workstation Hub |
| ⚠ Overridden by environment on this node | A SCION_SERVER_* variable sets this field’s key on the node that served the page (listed in env_overrides), and the Hub runs with that value. The field shows the environment value as In effect on this node. When a different value is stored (saved_differs), the stored value is labelled as not in effect: Saved value for a database setting, settings.yaml value for a file setting. For a masked (secret) key, whether the environment value is in effect is not reported, so the field shows only this badge. |
| ⚠ Environment value superseded by the saved setting | A SCION_SERVER_* variable sets this key on this node, but a saved database value wins over it: on a Layer-1 key the variable only changes the bootstrap seed and fallback. The field shows the environment value on a secondary not in effect line. |
| ⚠ Environment value not in effect on this node | A SCION_SERVER_* variable sets this key, but scion server start sets it afterwards, from the workstation defaults or a command-line flag. The field shows the value start-up set as In effect on this node (from running_values) when it differs, then its configured value (from settings.yaml, or the default), labelled Configured value (settings.yaml or default), not the value in effect, and the environment value on a secondary not in effect line. |
⚠ Section badge with a key list (runtimes, profiles, harness_configs) |
The same badges for a map section: one badge for the section, and a list of the environment-set keys under it. The values of these keys are always masked, and for a masked key whether it is in effect is not reported (a secret field keeps the plain Overridden by environment on this node badge). |
| In effect on this node: … (start-up default or flag) | A key that scion server start sets after loading the settings (a workstation default, a start flag or the coded default) runs with another value than the one shown (running_values). Shown on read-only and editable fields; on a read-only field the shown value is labelled Configured value (settings.yaml or default), not the value in effect. Never shown for a secret key. |
| ⚠ Some settings are overridden by environment variables on this node | Banner listing every key in env_overrides |
| 🔒 Set by workstation startup defaults / server flags | Workstation field that scion server start overrides at every start |
| Tracking deployment configuration | Seeded section — re-syncs on restart |
| ⓘ Superseded by database value | Deployment config differs from the admin-set value |
| Deprecation banner | SCION_SERVER_* used for Layer-1 settings |
The environment values come from env_values in GET /api/v1/admin/server-config (see Environment Override Warnings). Secret values are masked there as in the rest of the response, and the page never shows them.
The Federation page and the Project Default Settings scratchpad switch on the Server Config page use the same badges, from the same GET. On the Federation page, a global field (enabled, algorithms, refresh_interval, debounce_interval) whose environment value is in effect (in settings.yaml mode) is read-only, shows the environment value, and is never sent in a save. When a saved value wins instead (federation is a Layer-1 section, so on a database-backed Hub), the field stays editable and shows the environment value on a not in effect line. The scratchpad switch stays editable: it exists only on database-backed Hubs, where the saved value wins.
server.auth.authorized_domains is a special case on database-backed Hubs: SCION_SERVER_AUTH_AUTHORIZEDDOMAINS pins the list on the node (see server.auth.authorized_domains: environment first). So when it is set, the environment value is shown as in effect, and the key is not reported as superseded.
The superseded notice names the layer each deployment value comes from: SCION_SERVER_*, settings.yaml (or the --config location), SCION_SEED_*, or built-in default for a value that no setting or variable sets. A layer is named when it sets the key, even to the default value, so a SCION_SEED_* value equal to the default is shown as SCION_SEED_*. The admin API reports these as source in superseded_keys: server_env, yaml, seed_env, or default.
Saving changed sections
Section titled “Saving changed sections”On a database-backed or workstation Hub, Save & Reload on the Server Configuration page sends only the settings sections you changed, each in full, together with the revision each section had when the page loaded. Sections you did not touch are not sent, so a change another admin saved to them is kept. If another admin saved one of the sections you changed after the page loaded, the Hub refuses the save with 409 revision_conflict and nothing in that section is overwritten; the page shows the “Settings changed since you loaded this page” banner, and Reload fetches the current values and revisions. A save with no changes sends nothing and reports No changes to save. The Federation page sends its section’s revision the same way. On a settings-file Hub the page sends the full configuration, as before, without revisions.
Error Handling
Section titled “Error Handling”The admin UI provides structured feedback on save. Every error uses the common envelope {"error": {"code", "message", "details"}} (see Settings save errors):
| Response | UI Treatment |
|---|---|
| 200 | Success; shows ignored-keys notice if applicable |
409 revision_conflict |
“Settings changed since you loaded this page” banner with Reload button, listing any keys saved before the conflict (applied_keys, pending_restart, and skipped) |
409 legacy_server_yaml |
Shows the message: the server config still comes from a deprecated server.yaml; move it under server: in settings.yaml, remove server.yaml, then save again. Nothing was saved |
412 experiment_disabled |
profiles.<name>.image_tag was set or changed while the hub.image_tag_setting experiment is off (details.experiment: "hub.image_tag_setting"); nothing was saved |
422 validation_error |
Inline validation errors at each field: details.errors lists every rejected value with its koanf path in field (and section for a schema error). For example a value that fails the section schema, a duration-valued key, including one nested in a map, that does not parse as a duration (for example "5 minutes"; "" or null clears the key), default_thinking_level outside 1-100 (0 included; null clears it), or {"federation": null} (details.keys). Nothing was saved |
422 layer0_rejected |
Safety-net notice on hosted Hubs (should not occur with layer-aware UI) |
422 startup_only_key_rejected |
On a database-backed Hub, telemetry.cloud.cloud_logging was sent with a value different from what GET returns (keys: ["telemetry.cloud.cloud_logging"]); nothing was saved |
422 unclassified_keys_rejected / unpersisted_keys_rejected / hub_owned_keys_rejected / unsaved_keys_rejected |
Shows the message and the offending keys; nothing was saved |