Skip to content
Scion

Week of September 28 -- October 4, 2026

This week replaced raw keystroke messaging with a dedicated agent keys API: it was built, put into service and then made the only path for sending keystrokes. A timezone overhaul pinned servers, storage and APIs to UTC and made the Hub the only source of an agent’s TZ. Agent delete became asynchronous and failure-aware, and asynchronous create became available as an opt-in. Kubernetes gained per-agent NFS workspaces, Workload Identity and a new Substrate runtime. Security work continued throughout, with user access token boundaries, frozen delegation ceilings, project access granted only through role bindings, and fixes for a script-execution path and a leaked database password.

Highlights

Section titled “Highlights”

1. Agent Keys Replace Raw Messaging

Section titled “1. Agent Keys Replace Raw Messaging”

Keystroke delivery moved out of the message pipeline into its own API. POST /api/v1/agents/{id}/keys, and its project-scoped route, has bounded bodies, operation IDs, per-principal and per-target rate limits, running-phase checks and content-free audit records for every outcome. scion keys works inside agent containers and calls the API directly. Legacy raw messages went through the keys path for a short time. Then raw delivery was removed entirely: every message ingress (Hub, DMs, broadcasts, schedules, Runtime Brokers, plugins) now rejects a raw field with 422 raw_input_removed, and the --raw alias is gone.

2. UTC Everywhere, With Per-User Display Timezones

Section titled “2. UTC Everywhere, With Per-User Display Timezones”

Hub and Runtime Broker processes, the SQLite store, agent.log, access-constraint windows and API timestamps are all UTC now. tzdata is embedded in binaries and images. Each hub-dispatched agent gets its TZ from one Hub resolver: the agent’s pin, then the Hub env store (user, project, hub, Runtime Broker), then agent_defaults.default_timezone. Runtime-profile timezones and CRON_TZ= prefixes are gone. Users choose a display timezone and get a 24-hour clock across the web UI. The CLI shows a zone on every time and adds --tz and --utc flags, and maintenance operations normalize legacy stored timestamps. The new Times and Timezones reference covers it end to end.

3. Asynchronous, Failure-Aware Agent Lifecycle

Section titled “3. Asynchronous, Failure-Aware Agent Lifecycle”

Agent delete returns 204 when it finishes and 202 while teardown is still running. A failed delete leaves a marker that blocks start, restart, wake and reincarnate until the delete is retried or forced. The web UI shows “Deleting…”, “Delete interrupted” and “Delete failed” states, with Retry and Force actions. With server.hub.async_agent_launch on, create returns at once while the Runtime Broker launches the agent, and scion start and resume wait and print progress. Each agent also records whether it is meant to be running. Stops on offline Runtime Brokers are queued, agents whose container disappeared are moved to error, and a per-run run_id stops a stale delete from removing a recreated agent with the same name.

4. Kubernetes Workspaces, Identity and the Substrate Runtime

Section titled “4. Kubernetes Workspaces, Identity and the Substrate Runtime”

On Kubernetes NFS, worktree-per-agent projects give each agent its own git worktree, and clone-per-agent and the new empty-per-agent mode give each agent its own directory. The GCP identity mode assign now uses Workload Identity with operator-supplied service account mappings. Pods support priority classes, opting out of eviction and per-runtime shared-dir storage, and many start, cleanup and Secret-handling fixes make them more reliable. A new substrate runtime runs each agent as an Agent Substrate actor on GKE, with enforced privilege drop and egress allow-listing.

⚠️ Breaking Changes

Section titled “⚠️ Breaking Changes”
  • Raw message delivery removed (#2434, #2231, #2233, #2125): Every message ingress rejects a raw field with 422 raw_input_removed, and the CLI --raw alias is gone. Use POST .../keys or scion keys.
  • Hub is the only TZ source; cron is UTC-only (#2282, #2312, #2344, #2319, #2400): The Runtime Broker ignores TZ from local templates, Runtime Broker settings and persisted agent config, and logs a warning for each value it drops. profiles.<name>.timezone is removed, and a server-config PUT that includes it returns 422. CRON_TZ= and TZ= prefixes return 400. Existing prefixed schedules are paused once at upgrade; edit them to UTC, then resume them.
  • Agent delete can return 202 (#2391, #2399, #2351): A failed delete blocks lifecycle actions with 409 delete_in_progress. API clients that can’t tell 202 from 204 should poll until the agent is gone.
  • Project access comes only from role bindings (#2297, #2274, #2369): Project.OwnerID no longer grants access. Members groups are system-managed, and only hub admins can change them through the group API.
  • Sign-in requires a provider-verified email (#2071): Google, GitHub and OIDC sign-in reject unverified emails.
  • Runtime Broker registration and caps (#2063, #2142, #2115, #2114): Registering a Runtime Broker needs broker.create, granted through the hub-member role. On boot, existing max_agents_per_broker bindings scoped to a Runtime Broker become per-Runtime Broker settings and are now enforced, so upgrading can impose or tighten a cap. New Hubs default the cap to 100.
  • GCP Secret Manager names are hub-prefixed (#2123, #2166, #2226): New secrets are named scion-<hub hash>-…. Run scion hub secret migrate-names and update IAM conditions to the new prefix. In terraform-ha, the legacy hub-scope grant and the pre-created OIDC key are removed.
  • Kubernetes GCP identity (#2248, #2364): An explicit block on Kubernetes returns 400. assign now requires kubernetes_service_account_mappings (GSA to KSA) on the runtime or profile.
  • Reserved env targets (#2372): Hub secret and env writes to SCION_* and GCE_METADATA_* are rejected, and existing values are dropped at dispatch.
  • User access token storage (#2210): Tokens store an explicit boundary_kind, and project_id becomes nullable. Correct or delete any row with a non-UUID project_id before upgrading.
  • Hook token metrics retired (#2057): scion.hook.tokens.* is replaced by scion.usage.tokens{token_type}. Hook usage is recorded only when SCION_USAGE_SOURCE=hooks. Update dashboards and alerts.
  • Scheduled dispatch needs an unscoped credential (#2066): Scoped user access tokens can no longer create or change scheduled dispatch_agent work.
  • CLI and config defaults (#2153, #2445, #2456, #2453): scion list rejects positional arguments; use the new filter flags. harness-config sync and push default to project scope; use --global for ~/.scion/harness-configs. An invalid workspace storage config stops the Hub at startup. Server config saves that send an edited masked placeholder return 400.
  • Harness model changes (#2327, #2384): gemini-cli agents without a model now resolve the medium alias, so default agents switch models on restart. Antigravity thinking-level cut points changed.
  • hubclient ignores groveId (#2040): Request types honor only projectId.

🔐 Security & Access Control

Section titled “🔐 Security & Access Control”
  • Agent-written files could run scripts with a viewer’s session (#2009, #2340): Workspace, shared-dir, WebDAV, chat attachment and port-forward responses now carry a sandbox CSP and nosniff. Agent Set-Cookie and CSP headers are never relayed. (Credit: miller79)
  • Database password in logs (#2035): The Hub logged its full DSN, including the password, on every boot. Passwords are now masked in every DSN form.
  • User access token boundaries (#2299, #2407, #2436, #2143, #2300): Tokens store a versioned permission ceiling and are checked against the caller’s live authority when minted and on every request. Every bearer request checks the token’s boundary, target scope, current project access and live user authority, and any error denies. Lists, grants and user-to-agent messages also respect the boundary. scion hub token scopes shows which scopes you can mint.
  • Frozen delegation ceilings (#2206, #2355): Agent create records where the agent’s authority came from and caps the child’s role at the creating credential’s ceiling. Each delegation hop needs a live delegator, and cycles, excess depth and lookup errors deny.
  • Host-path and root-context hardening (#2244, #2236): Workspace paths are resolved through symlinks and checked against the project before mounts, syncs or chown. The filesystem root, home directories and system directories are refused. Root-context file operations use no-follow, fd-anchored primitives.
  • Secret handling (#2085, #2072, #2150, #2171, #2272, #2256): Runtime secret reads fail closed and require an active-member originator. Admins can limit agent-written secrets to profile scope. On Kubernetes, the transport credential moves into a per-agent Secret, and per-agent Secrets are cleaned up.
  • Credential checks (#2078, #2200, #2278, #2209): Agent token auth requires a successful credential-status check. Cloud token mint rechecks the service account assignment. Credentials minted for a create that never ran are revoked. sa_assign is now a separate scope.
  • Ownership integrity (#2414, #2435, #2334): The last owner of a project can’t be deleted, and role bindings are removed with their user. Ownership transfers can’t be undone by a concurrent PATCH. Project stop-all is authorized from role bindings.
  • Opaque pagination cursors (#2106, #2225): List cursors are sealed with AES-256-GCM and bound to the endpoint, filter and caller.
  • Messaging authorization (#2050, #2099, #2122): Agents can send keystrokes only within their own project. Direct conversations validate their recipients.
  • Port access for oversight (#2379): Project owners and admins can open members’ exposed ports. Attach, exec and env access are still denied. (Credit: miller79)
  • Credential attribution (#2090, #2091, #2092, #2292): Token purpose and label metadata appear in request logs, authorization decisions and audit records. Access-boundary changes have a transactional audit history.

💬 Chat & Messaging

Section titled “💬 Chat & Messaging”
  • Silent message loss fixed (#2107, #2081, #2119, #2147, #2370): Messages over about 16 KB were dropped; they are now streamed into tmux buffers. Agent reply rows no longer get stuck pending. Agents resumed after a Hub restart get their Message Broker subscriptions back. A full event-bus buffer returns a retryable 503. A broad correctness pass records every message failure exactly once.
  • @mentions and deferred delivery (#2083, #2079): Agents now receive @mentions from other agents. Messages to a reincarnating agent are stored and return 202 deferred instead of being dropped.
  • Cmd/Ctrl+K quick palette (#2069, #2104, #2084, #2169, #2207): Agents, Threads, People, Recent Files and Documents groups with fuzzy matching and in-page previews. It is on by default, with a touch-friendly header button.
  • gs:// links (#2250, #2271): gs:// URIs in agent messages render text, markdown and images inline, fetched with the sending agent’s service account. This is behind the web.gcs_links experiment.
  • Large-DM offload (#2132): Large DM bodies can be delivered as a short stub with a fetch command. It is off by default.
  • Chat additions and fixes (#2128, #2117, #2136, #2096, #2080, #2253, #2329, #2331, #2310): “Mark unread”, “Open terminal” and “Open in graph” actions are added, and owner/repo#N references auto-link. /stop stopped deleting agents, and /status and /spawn work again. Send with interruption is honored.

🤖 Agent Lifecycle

Section titled “🤖 Agent Lifecycle”
  • Async delete and create (#2391, #2446, #2466, #2420, #2360, #2455): Covered in the highlights above. scion start gains --no-wait and --wait-timeout.
  • Run intent and queued stops (#2422, #2293): Agents record their intended state separately from their observed phase. Stops on offline Runtime Brokers return 202 and are queued. Agents whose container disappeared move to error with container_missing.
  • Reincarnate improvements (#2037, #2177, #2193, #2441): Reincarnate now works for shared-workspace and hub-managed agents. --handoff-template prints the handoff template. --broker --dry-run reports whether an agent could move to another Runtime Broker.
  • Correct runtime routing (#2423, #2305, #2254): Stop, restart, delete, exec and logs go to the runtime recorded for the agent. An unregistered runtime returns 503 runtime_unavailable.
  • Configuration and images (#2076, #2074, #2163, #2183, #2430): Stopped agents’ model, image and env can be edited. Hub settings control image and imagePullPolicy. Model tier aliases resolve across harnesses. Auto-expose ports follow a single precedence.
  • Skill resolution (#2294, #2316, #2353, #2452): GitHub skill refs resolve concurrently, use per-credential cooldowns under rate limits, and fail with typed skill_resolution_failed errors.
  • Transport credential refresh (#2347, #2454, #2382): In-agent clients share the refreshed credential instead of the expired bootstrap one. sciontool doctor reports its source and expiry.

☸️ Kubernetes & Runtimes

Section titled “☸️ Kubernetes & Runtimes”
  • Substrate runtime (#2376): Agents run as Agent Substrate actors on GKE. Manifests and operations docs are under deploy/substrate/.
  • Per-agent and empty workspaces (#2314, #2333, #2365, #2390, #2397, #2409, #2419, #2418): NFS gives each agent its own worktree or clone. Non-git projects can use workspaceMode=per-agent for a private, initially empty directory; unsupported targets return 412.
  • Scheduling and storage (#2276, #2324, #2380, #2393, #2356): priority_class_name, safe_to_evict: false, shared_dir_storage_class and size, and a per-runtime shared_dir_storage_backend. The Runtime Broker reconciles NFS mounts automatically.
  • Reliability (#2235, #2222, #2318, #2354, #2352, #2281, #2263, #2108, #2156): Exec into pods on new nodes is retried. Failed starts clean up their pods and Secrets. Non-root pods no longer hit “Permission denied”. Root agents get /root. Intermittent ECHILD start failures are fixed.
  • Multi-runtime Runtime Brokers (#2254): Each distinct Kubernetes cluster, context or namespace is registered as its own runtime.

🕒 Timezones

Section titled “🕒 Timezones”
  • Server and storage (#2285, #2252, #2308, #2313, #2357): Processes, SQLite, logs and API timestamps are UTC. A make time-literals gate fixed 30 server-side sites.
  • Display and configuration (#2241, #2303, #2257, #2267, #2373, #2374, #2395, #2377, #2387): A per-user display timezone, a 24-hour clock across the UI, an admin Default Timezone and a Timezone row with Pin and Unpin on the agent Configure page. The CLI gains --tz and --utc.
  • Maintenance (#2403, #2388): utc-timestamp-normalize rewrites legacy stored timestamps. applied-config-tz-cleanup converts saved agent TZ values into pins.

📊 Usage Telemetry

Section titled “📊 Usage Telemetry”
  • Canonical usage contract (#2051, #2057): gen_ai.api.calls and scion.usage.tokens{token_type}, stamped with project and agent identity. The Hub dashboard counts cumulative increases, so Claude usage is attributed correctly.
  • Native harness usage (#2065, #2082, #2087, #2113, #2463): Codex, OpenCode, Antigravity, Copilot and gemini-cli now publish usage from their native events, with cache writes, failed calls and per-event model attribution.

☁️ Runtime Brokers, Deployment & Operations

Section titled “☁️ Runtime Brokers, Deployment & Operations”
  • Per-Runtime Broker settings and enforced caps (#2126, #2141, #2145, #2097): GET/PUT /api/v1/runtime-brokers/{id}/settings with maxAgents, resolved from the Runtime Broker setting, then the entitlement, then the hub default. The cap and its source appear in the web UI and in scion hub projects info.
  • Runtime Broker availability (#2046, #2450, #2070, #2098): Multi-instance Hubs restore offline providers. Heartbeats no longer wait on agent listing. Registration failures mark /healthz degraded, and runtime outages return a retryable 503.
  • Terraform HA module set (#2149): Shared Cloud SQL, Filestore, GKE Autopilot and Artifact Registry, plus per-hub roots for several namespaced Hubs in one GCP project.
  • Hub-wide experiments (#2121, #2152, #2191, #2214): An experiments registry, admin API and an Experiments tab on the server config page.
  • Single-node VM (#2100, #2059, #2151, #2120, #2124, #2350): An optional hybrid GKE runtime, checksummed releases, custom-mode default networks, and binary-tier updates from the config page.
  • CLI additions (#2153, #2219, #2445): scion list adds --owner, --broker, --harness and lineage filters. scion config get reads dotted profile and runtime keys. Hub-mode start resumes stopped agents.
  • Global directory protection (#2465): provide --project run outside a project no longer registers the Runtime Broker’s global directory, and the slug global is reserved.

🎨 Web UI

Section titled “🎨 Web UI”
  • Mobile web (#2287, #2322, #2283, #2461, #2358, #2239, #2245): 16px inputs, 44px touch targets, long-press action sheets, PWA icons, a pinned app frame, and layouts that fit 320px screens and landscape.
  • Bounded, faster agent lists (#2223, #2232, #2228, #2336, #2341, #2277, #2396, #2451, #2439): Server-sorted, paged lists. First load on a 500-agent hub dropped from about 21 s to 3.3 s. Authorization inputs are reused across a list, and view=compact roughly halves response size. Tree and graph views cap at four pages of 500.
  • Multi-role project members (#2273, #2320, #2330, #2449): One editor row per principal, with one built-in role plus custom roles and last-owner guards. (Credit: miller79)
  • Navigation (#2306, #2361, #2229, #2220): Cmd/Ctrl+K jumps to an agent in terminal and graph views. Each user’s open terminals are restored. Graph nodes stay in place when an agent is deleted.
  • Rendering performance (#2185, #2179, #2175, #2176): About 70% fewer DOM elements, cached graph layout and faster file-list rendering.

🧪 CI & Testing

Section titled “🧪 CI & Testing”
  • CI offload (#2402): The full test suite runs post-merge, nightly and on demand instead of on each PR.
  • New gates (#2443, #2378, #2459): An ent codegen drift job and a fixture-coverage gate. The 405 Allow lint is now blocking, and harness provision tests run in Build & Test.
  • Harness images (#2174, #2172, #2157): Every harness is built by Cloud Build, with a coverage check, and --target rebuilds a single harness.

📖 Documentation

Section titled “📖 Documentation”
  • Times and Timezones reference (#2458): The UTC API contract, display zone, agent TZ chain and maintenance migrations.
  • Operations (#2166, #2326, #2215, #2130, #2162): The Cloud Run secret-name migration, NFS endpoint changes, telemetry log queries and IAP audit logging.
  • Messaging and keys (#2243, #2411): Platform skills use scion keys. Agent messages use structured markdown.