Skip to content

Experiments

An experiment is an unfinished or risky feature that ships in the binary, is off or on per hub, and is controlled by admins from Admin → Server Config → Experiments. Each experiment is declared once in code with a name, a description, a default, and the layers it gates (the web UI, the hub server, or both). From that tab, admins can enable or disable any registered experiment for every user of the hub; the change takes effect on the next page load.

The Experiments tab lists every experiment this hub version knows about: its title, description, stage badge, layer badges (UI, Server, or both), its default, and a switch for the current effective value. Admins can also list, enable, disable, or reset experiment overrides from the CLI with scion hub experiments (list, enable NAME [NAME ...], disable NAME [NAME ...], and reset [NAME ...] or reset --all), which requires hub.experiments.update (or a hub-boundary access token with the hub_experiments:update scope). Managing experiments in the Web UI needs two permissions: hub.config.read to open the Server Config page at all, and hub.experiments.update to view and change the list on the Experiments tab itself. An admin who holds only hub.config.read sees the tab but gets a message asking for hub.experiments.update; granting roles should account for both.

  • Toggling an experiment stores an explicit override for that experiment and applies it immediately — there is no separate save step. The row then shows “· overridden”, even if the new value matches the default. “Reset to default” (shown only when an override is set) removes the override, so the experiment follows the registry default again.
  • Default means the value used when no admin has set an override for that experiment.
  • Next-page-load semantics. Changes apply to all users of this hub. Users see them the next time they load or refresh the page — there is no live push to already-open tabs.
  • Overrides for experiments unknown to this hub version are kept. The hub stores overrides by name, and different hub versions may know different experiments. If this hub doesn’t recognize a stored name, it leaves that override untouched (so a rolling deploy or a rollback never silently discards a kill switch) and lists it on the tab as a one-line note naming each one. Remove a stale entry on its own, without resetting every experiment, with PUT /api/v1/admin/experiments and body {"overrides":{"<name>":null},"expected_revision":<revision>}, using the revision from GET /api/v1/admin/experiments; a stale revision returns 409, so re-read and retry.
  • The admin value applies on every page load where the experiments request succeeds. If that request fails for a given page load (a transient error, a mixed-version replica during a rollout, an expired session), the browser falls back to its localStorage override, then to the compiled default, for that page load only; the next successful load restores the admin value.
  • If the stored experiment settings become unreadable: every server-layer experiment turns off (regardless of its default or any override), every web-only experiment returns to its registry default (so a disabled default-on UI experiment can reappear until the row is reset), an error is logged on the hub, and the tab shows a banner with a “Reset all to defaults” button that clears every stored override, including overrides for experiments not known to this hub version.

An agent can see which experiments are on with scion whoami --full, which reads GET /api/v1/experiments live (see scion whoami). That includes hub.artifacts (and web-only experiments such as web.artifact_review_editor, the graphical review editor for Markdown artifacts, and web.artifact_manage_access, the unified Manage access dialog on the artifact page; see Artifacts). The environment variable SCION_HUB_EXPERIMENTS covers only the experiments dispatched to agents (currently hub.conduit, hub.conduit_deliver and hub.exec_schedules), fixed when the agent starts; it is not the list of enabled experiments.

  • hub.reincarnate_harness_switch (Reincarnate harness switch, server layer, alpha, off by default). Enables scion reincarnate <agent> --harness <harness-config> on the Hub. The request deletes and rebuilds the agent home while keeping the agent identity and workspace; naming the current harness-config still resets the home. Run --dry-run first to review the HOME RESET warning. There is no rollback or confirmation prompt. Requires a Runtime Broker that advertises home-reset support for the agent’s runtime. See Switching harness-config for runtime support, credentials, dropped settings and failure recovery.
  • hub.async_agent_start (Asynchronous agent start, alpha, on by default). Lets the Hub answer a start, resume, or restart of an existing agent (POST /api/v1/agents/{id}/start and /restart with "async": true, which scion start and scion resume send and then wait on) as soon as a Runtime Broker that advertises asyncStart accepts the launch, while the broker finishes starting the container in the background; requires hub.async_agent_launch (also on by default). While any live Hub instance in the hub_instances registry lacks async_start_v1, starts stay synchronous across the fleet. Nodes on a version from before the hub_instances registry do not appear in that table, so when upgrading a multi-node Hub from such a version, disable hub.async_agent_start first and remove the override after every node has upgraded. Before downgrading a Hub node to an older binary, disable hub.async_agent_start and wait launch_timeout plus 15 seconds for in-flight starts to settle. See Server Configuration — Asynchronous agent start.
  • hub.flat_runtime_brokers (Flat Runtime Brokers, alpha, off by default). Lets a Runtime Broker register and serve a single runtime target with a stable identity (runtime_target in settings.yaml), pins new agents to that target, and shows the Brokers area in the web UI. Every legacy Runtime Broker reports a read-only, publishable conversion plan (conversionPlanPresentation, the publishable form of what scion runtime-broker convert --plan prints) on its heartbeats whatever the experiment state; the Hub stores it only while the experiment is on, and GET /api/v1/runtime-brokers/{id}/conversion-plan returns the latest stored report (broker.read; 404 while the experiment is off). A legacy Runtime Broker’s Overview tab in the Brokers area shows that plan read-only in Plan a move to flat Runtime Brokers (proposed instances, profile map, findings, capacity split, and bound agents and projects), and scion runtime-broker convert (--plan, --apply, --status, --rollback) plans, executes, checks, or rolls back the conversion on the host. See Multi-Broker Architecture — Flat Runtime Brokers and scion runtime-broker convert.
  • hub.conduit_deliver (Message delivery over Conduit, alpha, off by default). Delivers messages to Cloud Run agents over their Conduit session instead of through runtime exec, so no IAP exec connector is needed. It requires hub.conduit. An agent started before it was turned on keeps using exec until it restarts, and Docker and Kubernetes agents keep using exec. Only message text and interrupts go over Conduit: a message with attachments, text over 512 KiB, or harness delivery settings outside the route’s bounds uses exec where the runtime has it. Before an agent’s Conduit session is up, delivery retries, then uses exec, or fails before anything is pasted. Before you turn it on for a shared or production hub, rerun the Conduit load test (criterion 19) with this traffic.
  • hub.exec_schedules (Exec schedules, alpha, off by default). Exec-only scheduled events and recurring schedules: run a command (argv) in an agent’s container at a scheduled time or on a cron schedule, over the agent’s Conduit session, with no message and no LLM turn. It requires hub.conduit and NTP-synchronised agent clocks (sciontool refuses starts past not_after plus a 10-second clock tolerance). Agents launched while it is on advertise the exec-start and exec-status control routes; an agent started before it was turned on does not until it restarts (skipped with agent executor unavailable). Create one-shot exec events with scion schedule create --type exec --agent <name> (--in|--at) [--timeout 60s] -- <argv...> (POST /api/v1/projects/{projectId}/scheduled-events with eventType: "exec") and recurring exec schedules with scion schedule create-recurring --name <name> --cron <expr> [--time-zone <IANA>] --type exec --agent <name> [--timeout 60s] -- <argv...> (POST /api/v1/projects/{projectId}/schedules with eventType: "exec", agentId or agentName, command, and optional timeoutSeconds from 1 to 3600). In the Web UI, the project schedule and scheduled-event forms also offer exec while the experiment is on. A recurring fire that arrives while a previous run of that schedule is still running is skipped (previous run in progress) via the Hub’s per-schedule database overlap lease (leaseUntil = startedAt + timeout + 120s) and sciontool’s live-process guard; completed fires update runCount, errorCount (failed fires only), and lastRunAt/lastRunStatus/lastRunError, and scion schedule get and scion schedule history display execSummary (outcome, exitCode, durationMs, outputBytes, startedAt, finishedAt). Creating, resuming (POST .../resume), or editing future dispatch requires a user or agent that may attach to the agent (agent.attach), checked again on every fire; on PATCH /api/v1/projects/{projectId}/schedules/{id}, eventType cannot be changed to or from exec, dispatch-changing edits require live agent.attach on the stored target when it exists, and deleting the target pauses recurring exec schedules with statusReason: "target_deleted" (re-targeting a schedule whose target was deleted requires a replacement command or payload). Command output is not stored, and command is revealed in payload only to callers whose current credential holds live agent.attach on the target—otherwise redacted to commandRedacted: true, argc, and commandSha256 (matching agentcontrol.ExecArgvDigest: the lowercase hex SHA-256 of the canonical UTF-8 JSON array encoding of command with no HTML escaping and the trailing newline trimmed; also recorded as command_sha256 on exec audit events). Turning hub.exec_schedules off refuses new exec creations, PATCH edits, and resumes (while pause and delete still work), skips future fires, and continues exec-status outcome recovery for already-admitted runs; turning hub.conduit off also refuses reconnecting agent sessions, so outcome recovery continues only over sessions that stay connected, and runs whose session drops (or across a Hub restart with hub.conduit off) end abandoned at lease expiry. See Scheduling & Scheduled Events — Exec Schedules.
  • hub.port_view (Port page view, alpha, off by default). Opening an agent’s exposed port from the Hub serves the page through a short-lived page link, so the page’s relative scripts (including modules), styles, images, same-app requests and WebSockets load. Pages opened this way can only use relative URLs: URLs that start with / do not work. Browser storage, cookies and web workers are not available to the page. The link lasts server.hub.port_proxy.view_ttl (default 2 hours); opening the port from the Hub again gives a new one. It works over either port proxy transport; a page’s WebSocket works only over Conduit (the hub.conduit experiment). The page link grants access, so do not share it. Keep it off on a Hub behind an authenticating front proxy (for example IAP) until it has been verified with one.
  • hub.chat_retention (Chat retention, alpha, off by default). Enables the chat_retention admin settings section (GET/PUT /api/v1/admin/chat-retention and the Chat retention panel in Admin → Server Config; 404 while the experiment is off) and the "chat-retention" background sweep. By default chat_retention.enabled is false and chat_retention.messages_retention_days is null (keep messages forever). When hub.chat_retention is on and enabled is true, the sweep purges the messages of conversations and topics deleted more than deleted_conversation_grace_days (default 30) ago, and, when messages_retention_days is set (7–36500 days), messages older than that window, in paced chunks (sweep.chunk_size, pause_ms, tick_budget_seconds, tick_interval_minutes). Take a verified database backup before enabling this experiment in production. See Admin Settings — Chat Retention.
  • hub.agent_edit_held (Edit running agents (apply on next start), alpha, off by default). Lets PATCH /api/v1/agents/{id} (and the project-scoped PATCH and the agent Configure dialog) keep config edits that cannot be applied immediately in pendingConfig (or discard them with DELETE /api/v1/agents/{id}/pending-config) instead of returning 409. When pending edits are folded (at the next start, restart, resume, or wake, or on a config PATCH in a phase with no container), each pending key’s recorded saver is re-checked against that principal’s current agent.update grant and active account: keys whose saver may no longer edit the agent, whose account is inactive or missing, or that were saved with a delegated credential or by a principal kind not re-checked at fold time are discarded (not applied), audited as agent_pending_config_discarded (saver_unauthorized, saver_account_inactive, saver_unresolvable, or saver_delegated), and named in warnings, while held edits by non-admin users wait while maintenance mode is on. If a saver re-check cannot be decided because of a lookup or authorization fault, nothing is dropped or written and the start or PATCH answers 503 unavailable. See API Reference — Agents.
  • hub.lifecycle_events (Agent lifecycle events, alpha, off by default). Writes one transactional agent.lifecycle event per agent lifecycle transition (created, started, stopped, completed, deleted, error) to an internal outbox table and delivers it asynchronously to the scion-analytics Cloud Logging log (or stdout on a loopback SQLite dev-auth workstation server). Requires a pinned server.hub.hub_id (SCION_SERVER_HUB_HUBID) and Cloud Logging enabled (server.hub.logging.cloud_logging: true with cloud_project_id); enable the experiment on every Hub replica together. Includes an additive database migration that adds lifecycle cursor attribution columns (last_lifecycle_actor_*, last_lifecycle_principal_tier, last_lifecycle_run_id) to agents with no backfill. See Server Configuration — Agent lifecycle events.
  • hub.flat_cloudrun_target (Flat Cloud Run runtime target, alpha, off by default). Lets a flat Runtime Broker register and be pinned to a Cloud Run runtime target (target type cloudrun). On a co-located host (server.broker.instances with enableHub), a flat cloudrun instance activates at startup using only its own runtime_target.credentials_file (must be a regular file, not a symlink or directory, and not group- or world-writable), resolves the GCP project number via Cloud Resource Manager for cloudrun:<projectNumber>/<region> scope identity, verifies server.project_storage (backend: nfs with homes: nfs), and constructs a cloudrun-instances runtime isolated from ambient ADC and global runtimes:/profiles:; on a remote host (enableHub off), a flat cloudrun instance is not activated (cloudrun targets require a colocated host in this release). It requires hub.flat_runtime_brokers, and the Hub checks it in two places: a new flat registration whose target type is cloudrun, and agent create on a flat Runtime Broker whose target type is cloudrun. With it off, both answer 412 experiment_disabled with details.experiment set to hub.flat_cloudrun_target; with hub.flat_runtime_brokers also off, the refusal names hub.flat_runtime_brokers. The target type must be exactly cloudrun: a spelling that differs only by case or surrounding whitespace (for example CloudRun) is not the cloudrun type. It is refused at registration and at agent create with 422 runtime_target_type_unsupported whether this experiment is on or off; only hub.flat_runtime_brokers being off is reported before it. details.runtimeTargetType echoes the refused value cut to at most 64 bytes. An existing cloudrun Runtime Broker still re-registers and its existing agents still start, so turning the experiment off stops new agents without stranding the Runtime Broker. Docker and Kubernetes targets do not need it. It is a server-only experiment and is not sent to agents. GA gate: it stays alpha and off by default until agent secrets are delivered by a means other than plain Cloud Run instance environment variables.
  • hub.artifact_sponsor_review_grant (Automatic review access for the delegating user, alpha, off by default). When an agent creates a new artifact in a project, gives the agent’s recorded delegating user a review grant on it, if that user is active and holds a role that reads in the artifact’s home project. The grant counts only while they keep that role, until someone changes its level, and can be removed like any grant. Existing artifacts get nothing, and turning the experiment off stops new grants without removing those already made. Requires hub.artifacts. See Artifacts — Grants.
  • hub.artifact_archive (Artifact zip download, alpha, off by default). Adds Download all (.zip) to an artifact’s Files tab and the route behind it, GET /api/v1/artifacts/{id}/versions/{seq}/archive (404 while the experiment is off). It needs hub.artifacts. The archive streams through the Hub on every storage backend, bounded by the publish limits (max_files, max_bundle_bytes) and by 4 concurrent archives (2 per user or agent). See Artifacts — Zip download.
  • hub.artifacts_open_review_check (Warn before publishing over an open artifact review, alpha, off by default). Tracks open web review editor sessions (POST /api/v1/artifacts/{id}/review-sessions, POST .../{sessionId}/heartbeat, and DELETE .../{sessionId}; 404 while the experiment is off) and refuses a new non-review publish version on POST /api/v1/artifacts and POST /api/v1/artifacts/{id}/versions with 409 review_open when another user has an open review on the current version, unless "force": true is set in the publish manifest (which publishes the version, returns the open reviews in warnings, and logs version_forced in the artifact decision log). Requires hub.artifacts; keep it off until the Hub drops reviewers who lost access to the artifact. See Artifacts — Open reviews.
  • hub.project_self_leave (Leave a project, alpha, off by default). Lets a user leave a project they were added to directly: POST /api/v1/projects/{id}/leave (optionally {"dryRun": true}, a preview that changes nothing) removes the caller’s own direct project roles in one step, and revokes their access tokens limited to that project when they keep no other access. The last usable direct owner and a member only through a group are refused with 409. Project _capabilities report canLeave and leaveBlockedReason (last_owner or group_derived). When no project access remains, the Hub suspends and holds the agents the caller started in the project, together with every agent those agents created (see Agent Suspension When a Member Leaves a Project). The CLI command is scion project leave. In the web UI, Leave project is in the project page’s header menu, the project settings Danger Zone, and the user’s own row in the Members editor; it opens a dialog that previews the dry run and can delete the user’s agents first (see Leaving a Project). See API Reference — Projects. While it is off, the route answers 404 and the canLeave and leaveBlockedReason capabilities are not reported, so the web UI does not offer the action.
  • hub.image_tag_setting (Image tag setting, alpha, off by default). Lets the image_tag setting choose the tag of the bundled scion-* agent images whose tag is latest or absent: version selects the Hub binary’s release version (latest for any non-release build, nightly included), and any other value is a literal tag. Images with another tag, a digest, or a registry host are left alone, except exactly <image_registry>/scion-<name> (latest or untagged), the form the image_registry rewrite produces, which is retagged too. While it is off, image_tag is ignored and image references are unchanged: a Hub settings save that sets image_tag (database or settings-file mode) is refused with 412 experiment_disabled, and a value from the environment or a settings file other than latest is reported with a startup warning (latest, the opt-out, is logged at Info). While it is on, the Hub applies the tag to the images it sends and checks, and sends it with each dispatch (create, start, resume and restart), so the Runtime Broker applies it to the images it resolves from templates and harness configs. Maintenance builds and the system image build keep latest. The Hub process’s SCION_IMAGE_TAG is the environment layer; a Runtime Broker’s own environment is not consulted. Agents need images built for the tag (see “Build images for a release” in image-build/README.md). See Image tag.
  • hub.image_tag_version_default (Image tag version default, alpha, on by default). When image_tag is unset, a release build of the Hub (vX.Y.Z, vX.Y.Z-preview.N or vX.Y.Z-rc.N) uses its own version as the tag of the bundled scion-* agent images whose tag is latest or absent, with the same retagging rules as hub.image_tag_setting, so agents run the images published for that release; development, nightly and other builds keep latest. An explicit --image scion-<name>:latest, or a bare scion-<name>, is retagged too. Any set image_tag takes precedence (image_tag: latest opts out), even while hub.image_tag_setting is off. The version comes from the build’s version stamp only, so a build of a tagged commit with local changes counts as that release. Turn it off with scion hub experiments disable hub.image_tag_version_default or under Admin → Server Config → Experiments; while it is off, an unset image_tag leaves image references unchanged. See Image tag.
  • hub.runtime_orphan_reconcile (Runtime orphan report, alpha, off by default). Reports runtime objects (containers, pods, Cloud Run instances) that no Hub agent owns. On each Runtime Broker heartbeat, the Hub classifies the inventory entries it cannot match to one of that broker’s agents, but only on targets the broker listed completely: a project-filtered broker (multi-hub mode) or a target whose listing failed records nothing. The reasons are no_agent (the project is known and no agent has the entry’s name or agent ID, for example after scion delete --force), superseded (an agent with the same name but another ID exists), deleted (the agent is soft-deleted), other_broker, unknown_project (the project is not in this Hub’s database) and unlabeled (no project or name). Agents with a delete in flight or a queued start are skipped. Runtime Brokers report each entry’s agent ID; entries from older brokers are matched by name only. Each new orphan logs one warning; the scion.hub.runtime_orphans gauge (scion_hub_runtime_orphans in Prometheus form) counts them by broker, kind and reason; GET /api/v1/admin/runtime-orphans lists them with first and last seen times (filters broker, project and kind; requires hub.health.read on a browser session, 404 while the experiment is off). An entry missing from three complete inventories in a row is removed. Report only: nothing is stopped or deleted. While it is off, nothing is classified or recorded.
  • lifecycle.idle_suspend (Idle suspend and auto-wake, alpha, off by default). An agent the Hub suspended on its own (its suspend reason is idle or stalled, shown by scion list and the API’s suspendReason) is resumed automatically when a direct message, a web chat message, a mention, a notification or a scheduled message arrives for it, with no --wake needed. The sender only needs permission to message the agent, not permission to resume it. Agents a user suspended (reason user), agents suspended before suspend reasons were recorded, and held agents are never woken this way. Direct messages, web chat primaries and scheduled messages wait for the resume and then deliver; mentions, web chat secondaries and notifications are accepted at once and delivered after a background resume. While it is off, the suspend reason is still recorded and shown, and nothing is woken without an explicit wake.
  • hub.cloudrun_native_identity (Native agent identity on Cloud Run Instances, alpha, off by default). Dispatched by the Hub with each agent create, start, and restart. On each dispatch while the experiment is on, the Hub checks the Runtime Broker’s stored per-profile service account report (mappingsReportVersion 3, mappingsComplete, confirmed within 15 minutes, with cloudRunNativeIdentity: true on runtimes.<name>.cloudrun profiles, serviceAccountMappings entries of source: "cloudrun_allowlist", and cloudRunBlockAccount) before contacting the broker, and refuses an unmapped or unsupported identity with 422 identity_not_mapped (checkedBy: "hub_report" and details.reason cloudrun_account_not_attachable, cloudrun_block_account_missing, cloudrun_passthrough_unavailable, cloudrun_native_identity_unsupported, cloudrun_broker_upgrade_required, cloudrun_profile_unknown, or cloudrun_report_not_current). When the precheck passes on a native cloudrun profile, the Hub records native_cloud_run: true on appliedConfig before calling the broker (cleared only after a confirmed stop or delete), sends the native marker (gcp_identity.native_cloud_run on create, SCION_METADATA_NATIVE_CLOUDRUN=1 in resolved env on start and restart), strips gcp:token:* scopes from the agent’s Hub tokens, and refuses POST /api/v1/agent/gcp-token and POST /api/v1/agent/gcp-id-token with 403 forbidden ("the hub does not issue GCP tokens for agents on this Cloud Run Instances profile; use the instance's metadata server"), including while native_cloud_run remains recorded if the experiment is later turned off. On the Runtime Broker, mode assign runs the instance as the Hub-assigned service account (requiring the Hub’s native marker and an entry in global runtimes.<name>.cloudrun.assignable_service_accounts, refusing when SCION_INIT_FIREWALL is enabled or invalid, and setting SCION_METADATA_MODE=passthrough inside the container so the instance’s metadata server serves that account directly); mode block, an unset mode, or a hub-default passthrough runs the instance as global runtimes.<name>.cloudrun.block_service_account with SCION_METADATA_MODE=block inside the container (400 if block_service_account is unset); explicit or project-default passthrough on cloudrun and assign or block on cloudrun-instances or a flat Cloud Run instance are refused with 400. Existing instances of the agent’s run carry the scion-identity hash label and are deleted and recreated on start or restart rather than started in place. See Permissions — Hub-Default GCP Identity.
  • hub.cloudrun_ephemeral_agents (Ephemeral Cloud Run agents, server layer, alpha, off by default). Gates persistence: "ephemeral" on agent creation (config.persistence on POST /api/v1/agents, fixed at creation and recorded in the scion.dev/persistence annotation; an omitted or empty value defaults to "persistent", and any other value answers 400 validation_error). An ephemeral agent gets a fresh memory-backed Agent home and workspace on every start — nothing written inside the container survives stop. While the experiment is off, creating an agent with persistence: "ephemeral" answers 412 experiment_disabled (details.experiment: "hub.cloudrun_ephemeral_agents"), and starting an existing ephemeral agent is refused on the Runtime Broker (HubAgentDefaults.Experiments carries hub.cloudrun_ephemeral_agents on each dispatch). Creating or starting an ephemeral agent on a Runtime Broker that does not advertise the ephemeralPersistence capability (only Cloud Run Instances and flat cloudrun Runtime Brokers advertise it) answers 422 persistence_unsupported (details.persistence: "ephemeral", details.runtimeBrokerId), and creating one on a flat cloudrun Runtime Broker also requires hub.flat_cloudrun_target (412 experiment_disabled). Changing config.persistence via PATCH is refused (TX fixed). See Agent Configuration — Environment keys and API Reference — Agents.
  • hub.welcome_project (Welcome project, alpha, off by default). Adds an opt-in Welcome project step (step: "welcome") to the setup wizard before First workspace, backed by POST /api/v1/system/welcome/preview, POST /api/v1/system/welcome/install, and POST /api/v1/system/welcome/skip (404 while the experiment is off). Preview resolves the allow-listed GitHub repositories (scion-frontiers/agent-team, addyosmani/agent-skills, and microsoft/playwright-cli) to commit SHAs, downloads their archives from github.com / codeload.github.com, verifies each tarball’s PAX global header commit ID against the resolved commit SHA (archive commit id does not match the requested commit), and vendors each template’s referenced skills into skills/<name>/ with local paths, returning preview_id, expires_at, source, manifest (agent_team_repo, agent_team_sha, skills, templates, files_sha256, skipped_entries), and file_count (422 welcome_project_failed on failure). Install ({"preview_id": "<id>"}, 400 welcome_preview_invalid if unknown, used, or expired) re-hashes the staged files against manifest.files_sha256, creates a Hub-managed welcome project (or welcome-1 when welcome is taken, with annotation scion.welcome/manifest), imports the staged templates at project scope, starts the coordinator agent when a Runtime Broker and harness credentials are ready (or records pending_credentials or pending_broker), and records the final state (started, pending_credentials, pending_broker, skipped, or failed; 409 welcome_project_done on a repeat) in the welcome_project hub_settings ledger. See Onboarding Wizard — Welcome project.
  • hub.system_one (System One message guardrails, server layer, alpha, off by default). Enables Hub-scoped System One provider instance administration and probing (GET/POST /api/v1/admin/system-one/instances, GET/DELETE /api/v1/admin/system-one/instances/{name}, and POST /api/v1/admin/system-one/instances/{name}/probe, backed by scion admin system-one instance create|list|delete|probe), with runtime and probe evaluations bounded by server.hub.system_one.max_eval_time (SCION_SERVER_HUB_SYSTEMONE_MAXEVALTIME, default 3s, maximum 30s). Outbound provider calls enforce HTTPS and refuse loopback, RFC 1918, link-local, cloud metadata, and unspecified IP addresses at connect time (allowPrivateNetwork: true is refused). While it is off, the System One admin routes answer 404 and stored provider instances are kept.
  • web.decisions (Decisions, alpha, off by default). Lets an agent ask a user to decide between a few options (scion message <recipient> [message] --choices <json> or --choices-file <path|->, with --update to replace an earlier version of a decision key), and surfaces those Decisions across Web Chat, project and profile Decisions pages, the notification bell, and scion decisions. A choice prompt (message type choice-prompt) carries up to 10 decisions; each has a key that is unique per agent (one the agent picks in id, such as D234, or a number 1..N the Hub assigns), a markdown question (up to 2,000 runes) and 2 to 10 choices (each a string label or {"id", "label", "description"}). For a single-decision prompt (answerMode: "single"), clicking a choice records the answer (POST /api/v1/decisions/{rowId}/answer, body {"choice_id": "<id>", "idempotency_key"?: "<key>"}) and sends a choice-response to the agent. For a prompt with 2 to 10 decisions (answerMode: "set"), selecting choices stages them locally until every open decision has a selection, and Submit answers submits the set in one transaction (POST /api/v1/choice-prompts/{messageId}/answers, body {"answers": [{"decision_row_id": "<uuid>", "choice_id": "<id>"}, ...], "idempotency_key"?: "<key>"}) and dispatches one combined choice-response (carrying promptMessageId, batchId, and answers) to the agent (calling POST /api/v1/decisions/{rowId}/answer on a multi-decision prompt is refused with 409 batch_required, and a set submit whose actionable decisions changed is refused with 409 answer_set_changed). Reposting a key replaces the earlier version only when the agent asks for it (--update / choices_update: true; without it, 409 choice_key_exists). The Hub checks the experiment on sends carrying choices (refused with 400 experiment_disabled while it is off, never silently dropped), on all decision, answer and roll-up routes (GET /api/v1/decisions/{rowId}, POST /api/v1/decisions/{rowId}/answer, POST /api/v1/choice-prompts/{messageId}/answers, GET /api/v1/agent/decisions — with state=pending|decided|all, limit, cursor, and q of 2 to 256 characters matching the question, key or effective answer label under hub.chat.search.timeout (503 SEARCH_TIMEOUT on expiry) — GET /api/v1/agent/decisions/{key}, POST /api/v1/agent/decisions/interpreted, GET /api/v1/projects/{id}/decisions, and GET /api/v1/users/me/decisions, all of which answer 404 while it is off), on the coalesced decisions_waiting notification bell item, and in chat history, which omits the decision fields while it is off; the web layer toggles the cards, pages, nav entries, and Decisions profile settings card. In Web Chat a prompt shows as a card: the recommended choice is outlined; a multi-decision card selects nothing until the user clicks (each user can have recommended choices preselected with the preselectRecommendedChoices profile setting), counts the open decisions still without a selection (N remaining), and keeps Submit answers disabled until that reaches 0, with a tooltip naming the missing keys; a replaced decision links to its new version (Jump to v2); a viewer who cannot answer (the asking agent does not accept messages from them, or the conversation is read-only for them) sees the card disabled with the reason; and a choice-response shows as a compact line that a set response expands to list its answers. Card updates arrive on project.<id>.chat.choice.updated and user.<uid>.chat.choice.updated. Every choice prompt’s text also carries a markdown copy of its decisions, so bridged conversations, notifications and clients with the experiment off still show them. Each project gets a Decisions page (/projects/:id/decisions) and a Decisions header button with a Waiting on you count badge, backed by GET /api/v1/projects/{id}/decisions (tab=pending|made, limit=0..200, cursor, q, conversation=<chatKey>, agent=<uuid>, and on tab=made user, provenance=direct|interpreted|confirmed|overridden, from, to), with agents, answerers and conversations facets and unfiltered counts (plus filteredCounts on tab=pending while filtered). GET /api/v1/users/me/decisions rolls up Decisions across up to 200 most recently active member projects the caller can read (truncatedProjects: true beyond the cap; project=<id>, sort=project|recent, and a projects facet), backing /profile/decisions and scion decisions list --all-projects (see the CLI reference). Every query filters by the conversations the caller can read (readable project topics plus the caller’s own DMs with the project’s agents — never another user’s private DMs, even for project admins), and decisions and answers outlive chat retention via the stored chat_key, so purged prompts stay visible (promptAvailable: false) and answerable (surface: "decisions_page"). Stored decisions and answers are kept while it is off. It replaces web.chat_choices and web.project_decisions, which are retired; a stored override for either is ignored and pruned on the next admin change, so a Hub that had them on must run scion hub experiments enable web.decisions. See Messaging — Multiple-Choice Decisions.
  1. Open or link a tracking issue.
  2. Add an entry to pkg/experiments/registry.go with every field set. Default: false unless there is a deliberate decision otherwise. Set ReviewBy about 90 days out.
  3. Web gates use isFeatureEnabled(CONST), with the constant exported from feature-flags.ts. Server gates use requireExperiment or s.experimentEnabled.
  4. The hub decides hub behaviour. Browser-side values can be edited by the user, so a web gate is presentation only. If the experiment changes what the hub does, register it with LayerServer and check it in the hub with requireExperiment or s.experimentEnabled; the UI gate only mirrors that check. PR reviewers check this, because no test can detect it automatically. GET /api/v1/experiments carries only experiments with LayerWeb, so a UI gate that mirrors a hub check only works if the experiment is registered with both LayerWeb and LayerServer.
  5. Keep both paths working and tested (unit tests for flag on and flag off). E2E tests pin the flag through window.__SCION_FEATURES__. Tests of the experiments machinery itself use fixture names, never a production experiment, so that retiring one does not touch them.
  6. If Default: true and the experiment has LayerWeb, add the name to DEFAULT_ON_FLAGS (server-only names must not be listed). The consistency test enforces this.

Changing an experiment’s default needs a code change to the registry, in a PR that references the tracking issue, and, for an experiment with LayerWeb, an update to DEFAULT_ON_FLAGS to match (add the name when the default becomes true, remove it when it becomes false); the consistency test enforces this. A server-only (LayerServer-only) experiment is never listed in DEFAULT_ON_FLAGS. Admin overrides are kept across the change — an admin who already set an explicit value for that experiment is unaffected by a default flip.

Retiring an experiment is either a graduation (the feature becomes permanent) or an abandonment (the feature is deleted). Either way:

  • Delete the registry entry, add the name to compiledRetired, and remove it from DEFAULT_ON_FLAGS if it was there.
  • Delete the code that only ran in the other state: the flag-off path for a graduation, the feature itself for an abandonment.
  • Stored overrides for a retired name need no migration: they are ignored on read and pruned on the next admin write. Retired names are never reused.
  • In code and tests, the only references to the name left are its compiledRetired entry and one web test asserting that the name resolves OFF.
  • Put a release note in the PR description.

Follow the full experiment-removal checklist.

The web client resolves a flag in this order, highest first:

# Source Present in production? Purpose
1 Values in window.__SCION_FEATURES__ before server flags are first applied (“pinned”) Only setFeatureFlag('web.native_chat*'), which is never an experiment name; otherwise E2E init scripts Deterministic tests
2 Server value from GET /api/v1/experiments (admin override, else the registry default) Yes, signed-in users only Hub-wide control
3 localStorage scion:feature:<name> Yes, for unregistered names, failed fetches, and signed-out loads (set only through devtools) Unregistered or in-development flags; fallback when the fetch fails
4 Compiled DEFAULT_ON_FLAGS Yes Fetch failure

The map of web-layer experiments is only served to signed-in users, from GET /api/v1/experiments, with compiled defaults applying when that request fails. Devtools localStorage overrides no longer apply to registered experiments, except on page loads where the experiments request fails. The server value wins so that an admin’s hub-wide choice is a dependable control, not something a single browser can silently override.

Each registry entry has a ReviewBy date. Once it passes, the Experiments tab shows a “Review overdue” warning on that row. The owner either extends ReviewBy with a reason in a PR, or retires the experiment. There is no CI check that fails a build after the date passes — review is a documented process, not a time bomb.