Experiments
An experiment is an unfinished or risky feature that ships in the binary, is off or on per hub, and is controlled by admins from Admin → Server Config → Experiments. Each experiment is declared once in code with a name, a description, a default, and the layers it gates (the web UI, the hub server, or both). From that tab, admins can enable or disable any registered experiment for every user of the hub; the change takes effect on the next page load.
For admins
Section titled “For admins”The Experiments tab lists every experiment this hub version knows about: its title, description, stage badge, layer badges (UI, Server, or both), its default, and a switch for the current effective value. Admins can also list, enable, disable, or reset experiment overrides from the CLI with scion hub experiments (list, enable NAME [NAME ...], disable NAME [NAME ...], and reset [NAME ...] or reset --all), which requires hub.experiments.update (or a hub-boundary access token with the hub_experiments:update scope). Managing experiments in the Web UI needs two permissions: hub.config.read to open the Server Config page at all, and hub.experiments.update to view and change the list on the Experiments tab itself. An admin who holds only hub.config.read sees the tab but gets a message asking for hub.experiments.update; granting roles should account for both.
- Toggling an experiment stores an explicit override for that experiment and applies it immediately — there is no separate save step. The row then shows “· overridden”, even if the new value matches the default. “Reset to default” (shown only when an override is set) removes the override, so the experiment follows the registry default again.
- Default means the value used when no admin has set an override for that experiment.
- Next-page-load semantics. Changes apply to all users of this hub. Users see them the next time they load or refresh the page — there is no live push to already-open tabs.
- Overrides for experiments unknown to this hub version are kept. The hub stores overrides by name, and different hub versions may know different experiments. If this hub doesn’t recognize a stored name, it leaves that override untouched (so a rolling deploy or a rollback never silently discards a kill switch) and lists it on the tab as a one-line note naming each one. Remove a stale entry on its own, without resetting every experiment, with
PUT /api/v1/admin/experimentsand body{"overrides":{"<name>":null},"expected_revision":<revision>}, using therevisionfromGET /api/v1/admin/experiments; a stale revision returns 409, so re-read and retry. - The admin value applies on every page load where the experiments request succeeds. If that request fails for a given page load (a transient error, a mixed-version replica during a rollout, an expired session), the browser falls back to its localStorage override, then to the compiled default, for that page load only; the next successful load restores the admin value.
- If the stored experiment settings become unreadable: every server-layer experiment turns off (regardless of its default or any override), every web-only experiment returns to its registry default (so a disabled default-on UI experiment can reappear until the row is reset), an error is logged on the hub, and the tab shows a banner with a “Reset all to defaults” button that clears every stored override, including overrides for experiments not known to this hub version.
For agents
Section titled “For agents”An agent can see which experiments are on with scion whoami --full, which reads GET /api/v1/experiments live (see scion whoami). That includes hub.artifacts (and web-only experiments such as web.artifact_review_editor, the graphical review editor for Markdown artifacts, and web.artifact_manage_access, the unified Manage access dialog on the artifact page; see Artifacts). The environment variable SCION_HUB_EXPERIMENTS covers only the experiments dispatched to agents (currently hub.conduit, hub.conduit_deliver and hub.exec_schedules), fixed when the agent starts; it is not the list of enabled experiments.
Server experiments with notes
Section titled “Server experiments with notes”hub.reincarnate_harness_switch(Reincarnate harness switch, server layer, alpha, off by default). Enablesscion reincarnate <agent> --harness <harness-config>on the Hub. The request deletes and rebuilds the agent home while keeping the agent identity and workspace; naming the current harness-config still resets the home. Run--dry-runfirst to review the HOME RESET warning. There is no rollback or confirmation prompt. Requires a Runtime Broker that advertises home-reset support for the agent’s runtime. See Switching harness-config for runtime support, credentials, dropped settings and failure recovery.hub.async_agent_start(Asynchronous agent start, alpha, on by default). Lets the Hub answer a start, resume, or restart of an existing agent (POST /api/v1/agents/{id}/startand/restartwith"async": true, whichscion startandscion resumesend and then wait on) as soon as a Runtime Broker that advertisesasyncStartaccepts the launch, while the broker finishes starting the container in the background; requireshub.async_agent_launch(also on by default). While any live Hub instance in thehub_instancesregistry lacksasync_start_v1, starts stay synchronous across the fleet. Nodes on a version from before thehub_instancesregistry do not appear in that table, so when upgrading a multi-node Hub from such a version, disablehub.async_agent_startfirst and remove the override after every node has upgraded. Before downgrading a Hub node to an older binary, disablehub.async_agent_startand waitlaunch_timeoutplus 15 seconds for in-flight starts to settle. See Server Configuration — Asynchronous agent start.hub.flat_runtime_brokers(Flat Runtime Brokers, alpha, off by default). Lets a Runtime Broker register and serve a single runtime target with a stable identity (runtime_targetinsettings.yaml), pins new agents to that target, and shows the Brokers area in the web UI. Every legacy Runtime Broker reports a read-only, publishable conversion plan (conversionPlanPresentation, the publishable form of whatscion runtime-broker convert --planprints) on its heartbeats whatever the experiment state; the Hub stores it only while the experiment is on, andGET /api/v1/runtime-brokers/{id}/conversion-planreturns the latest stored report (broker.read;404while the experiment is off). A legacy Runtime Broker’s Overview tab in the Brokers area shows that plan read-only in Plan a move to flat Runtime Brokers (proposed instances, profile map, findings, capacity split, and bound agents and projects), andscion runtime-broker convert(--plan,--apply,--status,--rollback) plans, executes, checks, or rolls back the conversion on the host. See Multi-Broker Architecture — Flat Runtime Brokers andscion runtime-broker convert.hub.conduit_deliver(Message delivery over Conduit, alpha, off by default). Delivers messages to Cloud Run agents over their Conduit session instead of through runtime exec, so no IAP exec connector is needed. It requireshub.conduit. An agent started before it was turned on keeps using exec until it restarts, and Docker and Kubernetes agents keep using exec. Only message text and interrupts go over Conduit: a message with attachments, text over 512 KiB, or harness delivery settings outside the route’s bounds uses exec where the runtime has it. Before an agent’s Conduit session is up, delivery retries, then uses exec, or fails before anything is pasted. Before you turn it on for a shared or production hub, rerun the Conduit load test (criterion 19) with this traffic.hub.exec_schedules(Exec schedules, alpha, off by default). Exec-only scheduled events and recurring schedules: run a command (argv) in an agent’s container at a scheduled time or on a cron schedule, over the agent’s Conduit session, with no message and no LLM turn. It requireshub.conduitand NTP-synchronised agent clocks (sciontoolrefuses starts pastnot_afterplus a 10-second clock tolerance). Agents launched while it is on advertise theexec-startandexec-statuscontrol routes; an agent started before it was turned on does not until it restarts (skippedwithagent executor unavailable). Create one-shot exec events withscion schedule create --type exec --agent <name> (--in|--at) [--timeout 60s] -- <argv...>(POST /api/v1/projects/{projectId}/scheduled-eventswitheventType: "exec") and recurring exec schedules withscion schedule create-recurring --name <name> --cron <expr> [--time-zone <IANA>] --type exec --agent <name> [--timeout 60s] -- <argv...>(POST /api/v1/projects/{projectId}/scheduleswitheventType: "exec",agentIdoragentName,command, and optionaltimeoutSecondsfrom1to3600). In the Web UI, the project schedule and scheduled-event forms also offerexecwhile the experiment is on. A recurring fire that arrives while a previous run of that schedule is still running isskipped(previous run in progress) via the Hub’s per-schedule database overlap lease (leaseUntil = startedAt + timeout + 120s) andsciontool’s live-process guard; completed fires updaterunCount,errorCount(failed fires only), andlastRunAt/lastRunStatus/lastRunError, andscion schedule getandscion schedule historydisplayexecSummary(outcome,exitCode,durationMs,outputBytes,startedAt,finishedAt). Creating, resuming (POST .../resume), or editing future dispatch requires a user or agent that may attach to the agent (agent.attach), checked again on every fire; onPATCH /api/v1/projects/{projectId}/schedules/{id},eventTypecannot be changed to or fromexec, dispatch-changing edits require liveagent.attachon the stored target when it exists, and deleting the target pauses recurring exec schedules withstatusReason: "target_deleted"(re-targeting a schedule whose target was deleted requires a replacementcommandorpayload). Command output is not stored, andcommandis revealed inpayloadonly to callers whose current credential holds liveagent.attachon the target—otherwise redacted tocommandRedacted: true,argc, andcommandSha256(matchingagentcontrol.ExecArgvDigest: the lowercase hex SHA-256 of the canonical UTF-8 JSON array encoding ofcommandwith no HTML escaping and the trailing newline trimmed; also recorded ascommand_sha256on exec audit events). Turninghub.exec_schedulesoff refuses new exec creations,PATCHedits, and resumes (whilepauseanddeletestill work), skips future fires, and continuesexec-statusoutcome recovery for already-admitted runs; turninghub.conduitoff also refuses reconnecting agent sessions, so outcome recovery continues only over sessions that stay connected, and runs whose session drops (or across a Hub restart withhub.conduitoff) endabandonedat lease expiry. See Scheduling & Scheduled Events — Exec Schedules.hub.port_view(Port page view, alpha, off by default). Opening an agent’s exposed port from the Hub serves the page through a short-lived page link, so the page’s relative scripts (including modules), styles, images, same-app requests and WebSockets load. Pages opened this way can only use relative URLs: URLs that start with/do not work. Browser storage, cookies and web workers are not available to the page. The link lastsserver.hub.port_proxy.view_ttl(default 2 hours); opening the port from the Hub again gives a new one. It works over either port proxy transport; a page’s WebSocket works only over Conduit (thehub.conduitexperiment). The page link grants access, so do not share it. Keep it off on a Hub behind an authenticating front proxy (for example IAP) until it has been verified with one.hub.chat_retention(Chat retention, alpha, off by default). Enables thechat_retentionadmin settings section (GET/PUT /api/v1/admin/chat-retentionand the Chat retention panel in Admin → Server Config;404while the experiment is off) and the"chat-retention"background sweep. By defaultchat_retention.enabledisfalseandchat_retention.messages_retention_daysisnull(keep messages forever). Whenhub.chat_retentionis on andenabledistrue, the sweep purges the messages of conversations and topics deleted more thandeleted_conversation_grace_days(default30) ago, and, whenmessages_retention_daysis set (7–36500days), messages older than that window, in paced chunks (sweep.chunk_size,pause_ms,tick_budget_seconds,tick_interval_minutes). Take a verified database backup before enabling this experiment in production. See Admin Settings — Chat Retention.hub.agent_edit_held(Edit running agents (apply on next start), alpha, off by default). LetsPATCH /api/v1/agents/{id}(and the project-scoped PATCH and the agent Configure dialog) keepconfigedits that cannot be applied immediately inpendingConfig(or discard them withDELETE /api/v1/agents/{id}/pending-config) instead of returning409. When pending edits are folded (at the next start, restart, resume, or wake, or on aconfigPATCH in a phase with no container), each pending key’s recorded saver is re-checked against that principal’s currentagent.updategrant and active account: keys whose saver may no longer edit the agent, whose account is inactive or missing, or that were saved with a delegated credential or by a principal kind not re-checked at fold time are discarded (not applied), audited asagent_pending_config_discarded(saver_unauthorized,saver_account_inactive,saver_unresolvable, orsaver_delegated), and named in warnings, while held edits by non-admin users wait while maintenance mode is on. If a saver re-check cannot be decided because of a lookup or authorization fault, nothing is dropped or written and the start or PATCH answers503 unavailable. See API Reference — Agents.hub.lifecycle_events(Agent lifecycle events, alpha, off by default). Writes one transactionalagent.lifecycleevent per agent lifecycle transition (created,started,stopped,completed,deleted,error) to an internal outbox table and delivers it asynchronously to thescion-analyticsCloud Logging log (or stdout on a loopback SQLite dev-auth workstation server). Requires a pinnedserver.hub.hub_id(SCION_SERVER_HUB_HUBID) and Cloud Logging enabled (server.hub.logging.cloud_logging: truewithcloud_project_id); enable the experiment on every Hub replica together. Includes an additive database migration that adds lifecycle cursor attribution columns (last_lifecycle_actor_*,last_lifecycle_principal_tier,last_lifecycle_run_id) toagentswith no backfill. See Server Configuration — Agent lifecycle events.hub.flat_cloudrun_target(Flat Cloud Run runtime target, alpha, off by default). Lets a flat Runtime Broker register and be pinned to a Cloud Run runtime target (target typecloudrun). On a co-located host (server.broker.instanceswithenableHub), a flatcloudruninstance activates at startup using only its ownruntime_target.credentials_file(must be a regular file, not a symlink or directory, and not group- or world-writable), resolves the GCP project number via Cloud Resource Manager forcloudrun:<projectNumber>/<region>scope identity, verifiesserver.project_storage(backend: nfswithhomes: nfs), and constructs acloudrun-instancesruntime isolated from ambient ADC and globalruntimes:/profiles:; on a remote host (enableHuboff), a flatcloudruninstance is not activated (cloudrun targets require a colocated host in this release). It requireshub.flat_runtime_brokers, and the Hub checks it in two places: a new flat registration whose target type iscloudrun, and agent create on a flat Runtime Broker whose target type iscloudrun. With it off, both answer412 experiment_disabledwithdetails.experimentset tohub.flat_cloudrun_target; withhub.flat_runtime_brokersalso off, the refusal nameshub.flat_runtime_brokers. The target type must be exactlycloudrun: a spelling that differs only by case or surrounding whitespace (for exampleCloudRun) is not thecloudruntype. It is refused at registration and at agent create with422 runtime_target_type_unsupportedwhether this experiment is on or off; onlyhub.flat_runtime_brokersbeing off is reported before it.details.runtimeTargetTypeechoes the refused value cut to at most 64 bytes. An existingcloudrunRuntime Broker still re-registers and its existing agents still start, so turning the experiment off stops new agents without stranding the Runtime Broker. Docker and Kubernetes targets do not need it. It is a server-only experiment and is not sent to agents. GA gate: it stays alpha and off by default until agent secrets are delivered by a means other than plain Cloud Run instance environment variables.hub.artifact_sponsor_review_grant(Automatic review access for the delegating user, alpha, off by default). When an agent creates a new artifact in a project, gives the agent’s recorded delegating user a review grant on it, if that user is active and holds a role that reads in the artifact’s home project. The grant counts only while they keep that role, until someone changes its level, and can be removed like any grant. Existing artifacts get nothing, and turning the experiment off stops new grants without removing those already made. Requireshub.artifacts. See Artifacts — Grants.hub.artifact_archive(Artifact zip download, alpha, off by default). Adds Download all (.zip) to an artifact’s Files tab and the route behind it,GET /api/v1/artifacts/{id}/versions/{seq}/archive(404while the experiment is off). It needshub.artifacts. The archive streams through the Hub on every storage backend, bounded by the publish limits (max_files,max_bundle_bytes) and by 4 concurrent archives (2 per user or agent). See Artifacts — Zip download.hub.artifacts_open_review_check(Warn before publishing over an open artifact review, alpha, off by default). Tracks open web review editor sessions (POST /api/v1/artifacts/{id}/review-sessions,POST .../{sessionId}/heartbeat, andDELETE .../{sessionId};404while the experiment is off) and refuses a new non-review publish version onPOST /api/v1/artifactsandPOST /api/v1/artifacts/{id}/versionswith409review_openwhen another user has an open review on the current version, unless"force": trueis set in the publish manifest (which publishes the version, returns the open reviews inwarnings, and logsversion_forcedin the artifact decision log). Requireshub.artifacts; keep it off until the Hub drops reviewers who lost access to the artifact. See Artifacts — Open reviews.hub.project_self_leave(Leave a project, alpha, off by default). Lets a user leave a project they were added to directly:POST /api/v1/projects/{id}/leave(optionally{"dryRun": true}, a preview that changes nothing) removes the caller’s own direct project roles in one step, and revokes their access tokens limited to that project when they keep no other access. The last usable direct owner and a member only through a group are refused with409. Project_capabilitiesreportcanLeaveandleaveBlockedReason(last_ownerorgroup_derived). When no project access remains, the Hub suspends and holds the agents the caller started in the project, together with every agent those agents created (see Agent Suspension When a Member Leaves a Project). The CLI command isscion project leave. In the web UI, Leave project is in the project page’s header menu, the project settings Danger Zone, and the user’s own row in the Members editor; it opens a dialog that previews the dry run and can delete the user’s agents first (see Leaving a Project). See API Reference — Projects. While it is off, the route answers404and thecanLeaveandleaveBlockedReasoncapabilities are not reported, so the web UI does not offer the action.hub.image_tag_setting(Image tag setting, alpha, off by default). Lets theimage_tagsetting choose the tag of the bundledscion-*agent images whose tag islatestor absent:versionselects the Hub binary’s release version (latestfor any non-release build, nightly included), and any other value is a literal tag. Images with another tag, a digest, or a registry host are left alone, except exactly<image_registry>/scion-<name>(latestor untagged), the form theimage_registryrewrite produces, which is retagged too. While it is off,image_tagis ignored and image references are unchanged: a Hub settings save that setsimage_tag(database or settings-file mode) is refused with412 experiment_disabled, and a value from the environment or a settings file other thanlatestis reported with a startup warning (latest, the opt-out, is logged at Info). While it is on, the Hub applies the tag to the images it sends and checks, and sends it with each dispatch (create, start, resume and restart), so the Runtime Broker applies it to the images it resolves from templates and harness configs. Maintenance builds and the system image build keeplatest. The Hub process’sSCION_IMAGE_TAGis the environment layer; a Runtime Broker’s own environment is not consulted. Agents need images built for the tag (see “Build images for a release” inimage-build/README.md). See Image tag.hub.image_tag_version_default(Image tag version default, alpha, on by default). Whenimage_tagis unset, a release build of the Hub (vX.Y.Z,vX.Y.Z-preview.NorvX.Y.Z-rc.N) uses its own version as the tag of the bundledscion-*agent images whose tag islatestor absent, with the same retagging rules ashub.image_tag_setting, so agents run the images published for that release; development, nightly and other builds keeplatest. An explicit--image scion-<name>:latest, or a barescion-<name>, is retagged too. Any setimage_tagtakes precedence (image_tag: latestopts out), even whilehub.image_tag_settingis off. The version comes from the build’s version stamp only, so a build of a tagged commit with local changes counts as that release. Turn it off withscion hub experiments disable hub.image_tag_version_defaultor under Admin → Server Config → Experiments; while it is off, an unsetimage_tagleaves image references unchanged. See Image tag.hub.runtime_orphan_reconcile(Runtime orphan report, alpha, off by default). Reports runtime objects (containers, pods, Cloud Run instances) that no Hub agent owns. On each Runtime Broker heartbeat, the Hub classifies the inventory entries it cannot match to one of that broker’s agents, but only on targets the broker listed completely: a project-filtered broker (multi-hub mode) or a target whose listing failed records nothing. The reasons areno_agent(the project is known and no agent has the entry’s name or agent ID, for example afterscion delete --force),superseded(an agent with the same name but another ID exists),deleted(the agent is soft-deleted),other_broker,unknown_project(the project is not in this Hub’s database) andunlabeled(no project or name). Agents with a delete in flight or a queued start are skipped. Runtime Brokers report each entry’s agent ID; entries from older brokers are matched by name only. Each new orphan logs one warning; thescion.hub.runtime_orphansgauge (scion_hub_runtime_orphansin Prometheus form) counts them bybroker,kindandreason;GET /api/v1/admin/runtime-orphanslists them with first and last seen times (filtersbroker,projectandkind; requireshub.health.readon a browser session,404while the experiment is off). An entry missing from three complete inventories in a row is removed. Report only: nothing is stopped or deleted. While it is off, nothing is classified or recorded.lifecycle.idle_suspend(Idle suspend and auto-wake, alpha, off by default). An agent the Hub suspended on its own (its suspend reason isidleorstalled, shown byscion listand the API’ssuspendReason) is resumed automatically when a direct message, a web chat message, a mention, a notification or a scheduled message arrives for it, with no--wakeneeded. The sender only needs permission to message the agent, not permission to resume it. Agents a user suspended (reasonuser), agents suspended before suspend reasons were recorded, and held agents are never woken this way. Direct messages, web chat primaries and scheduled messages wait for the resume and then deliver; mentions, web chat secondaries and notifications are accepted at once and delivered after a background resume. While it is off, the suspend reason is still recorded and shown, and nothing is woken without an explicit wake.hub.cloudrun_native_identity(Native agent identity on Cloud Run Instances, alpha, off by default). Dispatched by the Hub with each agent create, start, and restart. On each dispatch while the experiment is on, the Hub checks the Runtime Broker’s stored per-profile service account report (mappingsReportVersion3,mappingsComplete, confirmed within 15 minutes, withcloudRunNativeIdentity: trueonruntimes.<name>.cloudrunprofiles,serviceAccountMappingsentries ofsource: "cloudrun_allowlist", andcloudRunBlockAccount) before contacting the broker, and refuses an unmapped or unsupported identity with422 identity_not_mapped(checkedBy: "hub_report"anddetails.reasoncloudrun_account_not_attachable,cloudrun_block_account_missing,cloudrun_passthrough_unavailable,cloudrun_native_identity_unsupported,cloudrun_broker_upgrade_required,cloudrun_profile_unknown, orcloudrun_report_not_current). When the precheck passes on a nativecloudrunprofile, the Hub recordsnative_cloud_run: trueonappliedConfigbefore calling the broker (cleared only after a confirmed stop or delete), sends the native marker (gcp_identity.native_cloud_runon create,SCION_METADATA_NATIVE_CLOUDRUN=1in resolved env on start and restart), stripsgcp:token:*scopes from the agent’s Hub tokens, and refusesPOST /api/v1/agent/gcp-tokenandPOST /api/v1/agent/gcp-id-tokenwith403 forbidden("the hub does not issue GCP tokens for agents on this Cloud Run Instances profile; use the instance's metadata server"), including whilenative_cloud_runremains recorded if the experiment is later turned off. On the Runtime Broker, modeassignruns the instance as the Hub-assigned service account (requiring the Hub’s native marker and an entry in globalruntimes.<name>.cloudrun.assignable_service_accounts, refusing whenSCION_INIT_FIREWALLis enabled or invalid, and settingSCION_METADATA_MODE=passthroughinside the container so the instance’s metadata server serves that account directly); modeblock, an unset mode, or a hub-defaultpassthroughruns the instance as globalruntimes.<name>.cloudrun.block_service_accountwithSCION_METADATA_MODE=blockinside the container (400ifblock_service_accountis unset); explicit or project-defaultpassthroughoncloudrunandassignorblockoncloudrun-instancesor a flat Cloud Run instance are refused with400. Existing instances of the agent’s run carry thescion-identityhash label and are deleted and recreated on start or restart rather than started in place. See Permissions — Hub-Default GCP Identity.hub.cloudrun_ephemeral_agents(Ephemeral Cloud Run agents, server layer, alpha, off by default). Gatespersistence: "ephemeral"on agent creation (config.persistenceonPOST /api/v1/agents, fixed at creation and recorded in thescion.dev/persistenceannotation; an omitted or empty value defaults to"persistent", and any other value answers400 validation_error). An ephemeral agent gets a fresh memory-backed Agent home and workspace on every start — nothing written inside the container survives stop. While the experiment is off, creating an agent withpersistence: "ephemeral"answers412 experiment_disabled(details.experiment: "hub.cloudrun_ephemeral_agents"), and starting an existing ephemeral agent is refused on the Runtime Broker (HubAgentDefaults.Experimentscarrieshub.cloudrun_ephemeral_agentson each dispatch). Creating or starting an ephemeral agent on a Runtime Broker that does not advertise theephemeralPersistencecapability (only Cloud Run Instances and flatcloudrunRuntime Brokers advertise it) answers422 persistence_unsupported(details.persistence: "ephemeral",details.runtimeBrokerId), and creating one on a flatcloudrunRuntime Broker also requireshub.flat_cloudrun_target(412 experiment_disabled). Changingconfig.persistenceviaPATCHis refused (TXfixed). See Agent Configuration — Environment keys and API Reference — Agents.hub.welcome_project(Welcome project, alpha, off by default). Adds an opt-in Welcome project step (step: "welcome") to the setup wizard before First workspace, backed byPOST /api/v1/system/welcome/preview,POST /api/v1/system/welcome/install, andPOST /api/v1/system/welcome/skip(404while the experiment is off). Preview resolves the allow-listed GitHub repositories (scion-frontiers/agent-team,addyosmani/agent-skills, andmicrosoft/playwright-cli) to commit SHAs, downloads their archives fromgithub.com/codeload.github.com, verifies each tarball’s PAX global header commit ID against the resolved commit SHA (archive commit id does not match the requested commit), and vendors each template’s referenced skills intoskills/<name>/with local paths, returningpreview_id,expires_at,source,manifest(agent_team_repo,agent_team_sha,skills,templates,files_sha256,skipped_entries), andfile_count(422 welcome_project_failedon failure). Install ({"preview_id": "<id>"},400 welcome_preview_invalidif unknown, used, or expired) re-hashes the staged files againstmanifest.files_sha256, creates a Hub-managedwelcomeproject (orwelcome-1whenwelcomeis taken, with annotationscion.welcome/manifest), imports the staged templates at project scope, starts thecoordinatoragent when a Runtime Broker and harness credentials are ready (or recordspending_credentialsorpending_broker), and records the final state (started,pending_credentials,pending_broker,skipped, orfailed;409 welcome_project_doneon a repeat) in thewelcome_projecthub_settingsledger. See Onboarding Wizard — Welcome project.hub.system_one(System One message guardrails, server layer, alpha, off by default). Enables Hub-scoped System One provider instance administration and probing (GET/POST /api/v1/admin/system-one/instances,GET/DELETE /api/v1/admin/system-one/instances/{name}, andPOST /api/v1/admin/system-one/instances/{name}/probe, backed byscion admin system-one instance create|list|delete|probe), with runtime and probe evaluations bounded byserver.hub.system_one.max_eval_time(SCION_SERVER_HUB_SYSTEMONE_MAXEVALTIME, default3s, maximum30s). Outbound provider calls enforce HTTPS and refuse loopback, RFC 1918, link-local, cloud metadata, and unspecified IP addresses at connect time (allowPrivateNetwork: trueis refused). While it is off, the System One admin routes answer404and stored provider instances are kept.web.decisions(Decisions, alpha, off by default). Lets an agent ask a user to decide between a few options (scion message <recipient> [message] --choices <json>or--choices-file <path|->, with--updateto replace an earlier version of a decision key), and surfaces those Decisions across Web Chat, project and profile Decisions pages, the notification bell, andscion decisions. A choice prompt (message typechoice-prompt) carries up to 10 decisions; each has a key that is unique per agent (one the agent picks inid, such asD234, or a number1..Nthe Hub assigns), a markdownquestion(up to 2,000 runes) and 2 to 10choices(each a string label or{"id", "label", "description"}). For a single-decision prompt (answerMode: "single"), clicking a choice records the answer (POST /api/v1/decisions/{rowId}/answer, body{"choice_id": "<id>", "idempotency_key"?: "<key>"}) and sends achoice-responseto the agent. For a prompt with 2 to 10 decisions (answerMode: "set"), selecting choices stages them locally until every open decision has a selection, and Submit answers submits the set in one transaction (POST /api/v1/choice-prompts/{messageId}/answers, body{"answers": [{"decision_row_id": "<uuid>", "choice_id": "<id>"}, ...], "idempotency_key"?: "<key>"}) and dispatches one combinedchoice-response(carryingpromptMessageId,batchId, andanswers) to the agent (callingPOST /api/v1/decisions/{rowId}/answeron a multi-decision prompt is refused with409 batch_required, and a set submit whose actionable decisions changed is refused with409 answer_set_changed). Reposting a key replaces the earlier version only when the agent asks for it (--update/choices_update: true; without it,409 choice_key_exists). The Hub checks the experiment on sends carrying choices (refused with400 experiment_disabledwhile it is off, never silently dropped), on all decision, answer and roll-up routes (GET /api/v1/decisions/{rowId},POST /api/v1/decisions/{rowId}/answer,POST /api/v1/choice-prompts/{messageId}/answers,GET /api/v1/agent/decisions— withstate=pending|decided|all,limit,cursor, andqof 2 to 256 characters matching the question, key or effective answer label underhub.chat.search.timeout(503 SEARCH_TIMEOUTon expiry) —GET /api/v1/agent/decisions/{key},POST /api/v1/agent/decisions/interpreted,GET /api/v1/projects/{id}/decisions, andGET /api/v1/users/me/decisions, all of which answer404while it is off), on the coalesceddecisions_waitingnotification bell item, and in chat history, which omits the decision fields while it is off; the web layer toggles the cards, pages, nav entries, and Decisions profile settings card. In Web Chat a prompt shows as a card: the recommended choice is outlined; a multi-decision card selects nothing until the user clicks (each user can have recommended choices preselected with thepreselectRecommendedChoicesprofile setting), counts the open decisions still without a selection (N remaining), and keeps Submit answers disabled until that reaches 0, with a tooltip naming the missing keys; a replaced decision links to its new version (Jump to v2); a viewer who cannot answer (the asking agent does not accept messages from them, or the conversation is read-only for them) sees the card disabled with the reason; and achoice-responseshows as a compact line that a set response expands to list its answers. Card updates arrive onproject.<id>.chat.choice.updatedanduser.<uid>.chat.choice.updated. Every choice prompt’s text also carries a markdown copy of its decisions, so bridged conversations, notifications and clients with the experiment off still show them. Each project gets a Decisions page (/projects/:id/decisions) and a Decisions header button with a Waiting on you count badge, backed byGET /api/v1/projects/{id}/decisions(tab=pending|made,limit=0..200,cursor,q,conversation=<chatKey>,agent=<uuid>, and ontab=madeuser,provenance=direct|interpreted|confirmed|overridden,from,to), withagents,answerersandconversationsfacets and unfilteredcounts(plusfilteredCountsontab=pendingwhile filtered).GET /api/v1/users/me/decisionsrolls up Decisions across up to 200 most recently active member projects the caller can read (truncatedProjects: truebeyond the cap;project=<id>,sort=project|recent, and aprojectsfacet), backing/profile/decisionsandscion decisions list --all-projects(see the CLI reference). Every query filters by the conversations the caller can read (readable project topics plus the caller’s own DMs with the project’s agents — never another user’s private DMs, even for project admins), and decisions and answers outlive chat retention via the storedchat_key, so purged prompts stay visible (promptAvailable: false) and answerable (surface: "decisions_page"). Stored decisions and answers are kept while it is off. It replacesweb.chat_choicesandweb.project_decisions, which are retired; a stored override for either is ignored and pruned on the next admin change, so a Hub that had them on must runscion hub experiments enable web.decisions. See Messaging — Multiple-Choice Decisions.
For developers: launching an experiment
Section titled “For developers: launching an experiment”- Open or link a tracking issue.
- Add an entry to
pkg/experiments/registry.gowith every field set.Default: falseunless there is a deliberate decision otherwise. SetReviewByabout 90 days out. - Web gates use
isFeatureEnabled(CONST), with the constant exported fromfeature-flags.ts. Server gates userequireExperimentors.experimentEnabled. - The hub decides hub behaviour. Browser-side values can be edited by the user, so a web gate is presentation only. If the experiment changes what the hub does, register it with
LayerServerand check it in the hub withrequireExperimentors.experimentEnabled; the UI gate only mirrors that check. PR reviewers check this, because no test can detect it automatically.GET /api/v1/experimentscarries only experiments withLayerWeb, so a UI gate that mirrors a hub check only works if the experiment is registered with bothLayerWebandLayerServer. - Keep both paths working and tested (unit tests for flag on and flag off). E2E tests pin the flag through
window.__SCION_FEATURES__. Tests of the experiments machinery itself use fixture names, never a production experiment, so that retiring one does not touch them. - If
Default: trueand the experiment hasLayerWeb, add the name toDEFAULT_ON_FLAGS(server-only names must not be listed). The consistency test enforces this.
Changing the default
Section titled “Changing the default”Changing an experiment’s default needs a code change to the registry, in a PR that references the tracking issue, and, for an experiment with LayerWeb, an update to DEFAULT_ON_FLAGS to match (add the name when the default becomes true, remove it when it becomes false); the consistency test enforces this. A server-only (LayerServer-only) experiment is never listed in DEFAULT_ON_FLAGS. Admin overrides are kept across the change — an admin who already set an explicit value for that experiment is unaffected by a default flip.
Retiring an experiment
Section titled “Retiring an experiment”Retiring an experiment is either a graduation (the feature becomes permanent) or an abandonment (the feature is deleted). Either way:
- Delete the registry entry, add the name to
compiledRetired, and remove it fromDEFAULT_ON_FLAGSif it was there. - Delete the code that only ran in the other state: the flag-off path for a graduation, the feature itself for an abandonment.
- Stored overrides for a retired name need no migration: they are ignored on read and pruned on the next admin write. Retired names are never reused.
- In code and tests, the only references to the name left are its
compiledRetiredentry and one web test asserting that the name resolves OFF. - Put a release note in the PR description.
Follow the full experiment-removal checklist.
Precedence and localStorage
Section titled “Precedence and localStorage”The web client resolves a flag in this order, highest first:
| # | Source | Present in production? | Purpose |
|---|---|---|---|
| 1 | Values in window.__SCION_FEATURES__ before server flags are first applied (“pinned”) |
Only setFeatureFlag('web.native_chat*'), which is never an experiment name; otherwise E2E init scripts |
Deterministic tests |
| 2 | Server value from GET /api/v1/experiments (admin override, else the registry default) |
Yes, signed-in users only | Hub-wide control |
| 3 | localStorage scion:feature:<name> |
Yes, for unregistered names, failed fetches, and signed-out loads (set only through devtools) | Unregistered or in-development flags; fallback when the fetch fails |
| 4 | Compiled DEFAULT_ON_FLAGS |
Yes | Fetch failure |
The map of web-layer experiments is only served to signed-in users, from GET /api/v1/experiments, with compiled defaults applying when that request fails. Devtools localStorage overrides no longer apply to registered experiments, except on page loads where the experiments request fails. The server value wins so that an admin’s hub-wide choice is a dependable control, not something a single browser can silently override.
Review cadence
Section titled “Review cadence”Each registry entry has a ReviewBy date. Once it passes, the Experiments tab shows a “Review overdue” warning on that row. The owner either extends ReviewBy with a reason in a PR, or retires the experiment. There is no CI check that fails a build after the date passes — review is a documented process, not a time bomb.