Week of October 5 -- 11, 2026

This week Conduit grew from a relay hosted in the Hub into a working transport for agent terminals, Runtime Broker traffic and user tunnels, including SSH, all behind its experiment flag. Agent create, start, stop and delete were made consistent under races: a create or start that loses to a delete no longer revives the agent, operations are scoped to the run they target, and starts fail closed when credentials or settings cannot be resolved. Access now follows project membership. When a user’s access to a project ends, the agents they started there are suspended. Sharing a Runtime Broker needs its owner’s consent, and schedules run under recorded authority. scion reincarnate can now move agents between Runtime Brokers and change their settings, and the health dashboard was rebuilt around a single attention policy.

Highlights

1. Conduit Carries Terminals, Runtime Broker Traffic and User Tunnels

Behind the hub.conduit experiment, the Hub now hosts the conduit relay at GET /api/v1/conduit, and sciontool dials it with backoff. Over the week conduit gained the pty stream kind (served through tmux), per-agent PTY routing, and Runtime Broker conduit sessions, which run alongside the control channel and can carry Runtime Broker requests. The Hub can also ask an agent to rotate its token over the session. Users can now run scion tunnel <agent> (--port N | --ssh) [--stdio] to reach an agent’s TCP ports or an embedded sciontool SSH server (shell, exec, sftp, loopback forwarding) that is reachable only through conduit. scion tunnel status lists running tunnels. Sessions are admitted only for the agent’s current run. User streams are re-checked when access changes and closed with 4401 once access is withdrawn, and relay sessions have a lifetime cap with a GoAway handoff. scion attach, start -a and resume -a now send the Hub’s PTY preflight and reconnect automatically after retryable closes.

2. Agent Lifecycle That Holds Up Under Races

Every synchronous create, start, restart and create-rollback path that loses to a delete now answers 409 delete_in_progress. This covers cross-node, DM wake and A2A paths, in place of the old 200, 201 or 404 answers. Delete and stop are scoped to a run on Docker, Podman, Apple, Cloud Run and Kubernetes, and so is pre-clean on Kubernetes, so they never remove another run’s container or pod, and the Hub refuses to delete a row while the Runtime Broker holds a different run. Pod recovery start claims are on, so a held claim answers 409 start_in_progress. Synchronous create, start, stop, suspend and stop-all are detached from the client request with longer write deadlines, so a slow but successful operation no longer surfaces as a 502. Starts now fail closed: the Hub answers 503 when it cannot resolve an agent’s credentials or stored environment, and a start whose env-based harness auth is missing its keys fails before any container is created. Agents report an initializing activity until the harness accepts input, and messages sent in the meantime are buffered instead of lost. scion start --wait-ready waits for that point.

3. Access Follows Membership and Recorded Authority

When a user’s access to a project ends, the Hub suspends the agents they started there and every agent those agents created. Suspended agents keep their workspaces but are refused until a Hub admin lifts the hold. Linking a Runtime Broker to a project now requires broker.update on that Runtime Broker. Members can dispatch only to providers linked by the Runtime Broker’s owner, linked by an active super-admin, or set to auto-provide. Scheduled dispatches and scheduled messages run under the authority recorded at the schedule’s last revision, so existing schedules must be paused and resumed once. User access token scoping now covers inbox, conversation, project configuration, Hub configuration, integration and GitHub App operations. Conversations stay in the project they were created in, and federated callers are refused on non-agent writes.

4. Reincarnate Moves and Reconfigures Agents

scion reincarnate --broker <target> now actually moves empty-per-agent agents between Runtime Brokers on a shared Kubernetes export. It runs a dry-run handshake first, moves quota to the target, and rolls back to the source if the start fails. New --service-account, --role, --model, --thinking-level, --harness-auth and --image flags change the next generation, and --dry-run shows old → new values. Empty-per-agent agents can now also be rebuilt in place on local-disk runtimes (Docker, Podman and Apple container). --shared-dir-backend NAME=nfs|local switches a shared dir’s recorded storage backend without copying data, and the next start refuses an empty target dir unless --allow-empty-shared-dir is given. scion reset-auth --reissue-scopes recomputes delegated agents’ roles and scopes, one agent at a time or across the whole Hub with --all --apply. The model and limits of created, stopped, failed or suspended agents can now be edited, and PATCH responses say whether each change applies on the next container or at the next reincarnation.

⚠️ Breaking Changes

Assistant-reply mirror retired (#2582): sciontool no longer mirrors each turn’s final text, and the Hub answers assistant-reply with 200 {status: dropped}. Telegram Commentary, the ShowAssistantReply settings and the Google Chat relay are removed. Agents reply on purpose with scion message.
Chat notifications replaced by an unread badge (#3106, #3109): The Hub no longer creates MENTION/DM_RECEIVED notification rows, and the web Messages envelope and inbox tray are gone. The chat mode selector and tab title show the count from GET /api/v1/chat/unread-count.
Removed members’ agents are suspended (#2801, #3158): Suspended agents are refused on start, messaging, token minting, secrets, schedules and agent creation until a Hub admin calls POST /api/v1/agents/{id}/hold/lift. For the first boot after upgrading, server.hub.membership_sweep_report_only logs and audits the agents the sweep would hold, without holding them.
Runtime Broker sharing needs owner consent (#2665): The providers API, project register, hub link and auto-link on agent create all require broker.update. Only super-admins can set broker.auto_provide or link the embedded Runtime Broker.
Schedules need re-authoring (#2736, #2816, #2848): Existing dispatch_agent and message schedules fail, or do not fire, until they are paused and resumed or recreated. Writes from a credential whose authority cannot be recorded get 403, and creating schedules with a user access token is refused.
409 delete_in_progress and conflict (#2578, #2597, #2783, #2815, #2844, #2729, #2752): Synchronous create, start, restart and POST /agents on an existing agent answer 409 when a delete wins. Delete answers 409 conflict while the Runtime Broker holds a different run, even with force=true. A managed create whose post-create write fails rolls back with 500.
Agent starts fail closed (#2542, #3271, #3297, #3352, #3177): Starts answer 400 when an agent’s GCP service account is unverified or inadmissible, or when no Kubernetes service account maps it (identity_not_mapped). They answer 503 when the Hub cannot resolve the agent’s credentials or read its stored environment settings. A start whose env-based harness auth cannot be satisfied fails on the Runtime Broker before any container is created; that error names the harness, auth type and expected env keys, never their values. Auth types that declare required_files are never rejected.
Local-only CLI mode removal begins (#3227, #3287, #3277): A repo links to the local workstation Hub without prompting. scion hub unlink only clears the link, Runtime Brokers ignore hub.enabled, and scion delete --cleanup-only replaces scion --no-hub delete.
CLI assistant mode removed (#3131): SCION_CLI_MODE=assistant and cli.mode: assistant are ignored with a warning.
Unified log level (#3034, #3058, #2950, #3044, #3067): SCION_LOG_LEVEL, with per-component levels such as info,hub.auth=debug, controls the server, CLI and sciontool. SCION_DEBUG is a deprecated alias. Agents no longer inherit --debug, so use scion start --agent-log-level. /auth/debug is served only with --enable-debug-endpoints, and that flag is refused in hosted mode. server.log_format and the SCION_TELEMETRY_DEBUG/LOCAL_* variables are now inert.
Decision audit table dropped (#2986): On startup the Hub runs DROP TABLE IF EXISTS decision_audits, which takes an ACCESS EXCLUSIVE lock on PostgreSQL. Mutation audits are unchanged.
Authorization fails closed (#2755): The Hub refuses to start with incomplete session-only route metadata, and Hub-admin routes that declare a permission answer 500 when no authorization service is configured, instead of falling back to the admin role check.
Unlabeled git projects are clone-per-agent (#3016): A git project with no workspace sharing mode label, or a raw shared-plain label, now gives each agent its own clone. Only the shared label selects the shared checkout. Push shared-checkout work before upgrading.
Non-interactive confirmations answer No (#2920, #2949): Without a terminal, confirmations default to No unless --yes is passed, and prompt text goes to stderr.
Strict configuration writes (#2637, #3121): A server-config PUT with keys the Hub would not save is rejected with 422, and nothing is applied. DB-backed sections merge onto the stored row, and concurrent writes get 409.
Template commits are compare-and-swap (#3174, #3197, #3132): A lost race returns 409 template_conflict, and the CLI retries. Templates move to content-addressed blobs, and legacy rows migrate on their next commit.
User and service account deletes guarded (#2652, #3231, #3301): Deleting a user who still owns agents answers 409 and lists the agents. Deleting a GCP service account that is still referenced answers 409 sa_in_use; --force clears project and per-profile defaults, but never the Hub default. The last active access-constraint admin cannot be suspended or deleted.
Project input formats (#2591, #2607, #2592): Clone URLs that carry a username, password, token, query or fragment are refused, so move clone tokens to project secrets or the GitHub App. Client-supplied slugs must already be in slugified form.
Tokens and permissions (#3099, #3127, #3187, #3156, #3178, #3074): hub.federation.read/update, hub.teams_manifest.update, user.list and agent.log_append are reserved and removed from built-in roles. User access tokens need user_skill_injection:update to change their own injected skills. Bulk agent auth reset needs a user session, and GCP service account lists need project read or Hub membership. Federated callers are refused on non-agent writes. admin-status reports isAdmin only for Hub admins and super-admins.
Upgrade order and migrations (#2646, #2500, #2804, #2805, #2780, #3236, #3319): Upgrade the Hub first, or together with Runtime Brokers; upgrade Kubernetes Runtime Brokers together with the Hub, because older ones ignore the per-run Secrets the Hub now creates. Agents using file-based harness auth must re-authenticate once. Run auto-expose-env-normalize once per existing Hub. On large PostgreSQL deployments, pre-create the broker_dispatch (state, updated_at) index with CREATE INDEX CONCURRENTLY before upgrading. Re-import the Hub telemetry dashboard. The Helm chart sets POD_NAME; a value set through hub.extraEnv must be unique per pod. The Hub refuses to start when a later schema on the search_path shadows its tables; fix the search_path, or set server.database.allow_shadowed_schema to log a warning and continue.
Kubernetes resource defaults (#2596, #2640): CPU, memory and storage requests are added only when no request or limit is set, and the old 2 CPU / 4Gi / 10Gi limits are no longer applied when enforce_resource_defaults is false.
Output and exit codes (#2682, #2706, #2852, #2983): Group-send --format json returns an object with counts, results and retry_recipient, and partial delivery exits 3. The health summary drops stall_config. On non-GCS Hub storage, creating an agent on a remote Runtime Broker with no local path, in a project with no git remote, fails with 412. deploy.sh sets hub_name from HUB_NAME.
Explicit --project beats SCION_PROJECT_ID (#2548): Inside agent containers an explicit project flag now wins, and -g global resolves the Hub’s Global project.

🔐 Security & Access Control

User access token scoping (#2647, #2639, #2579, #2694, #2798, #2757, #2739, #2900): Every token is checked against its boundary at request time. Inbox, conversation, notification, messaging policy, template, project configuration, Hub configuration, integration, GitHub App and observability operations each require a matching permission or selector. Integration secrets and GitHub App configuration need an interactive session.
Conversations stay in their project (#3066, #2951, #3013, #3036): Lists and lookups return only readable conversations. Replies, attachments, read markers and thread references stay inside their conversation, and live DM events reach only participants.
Tokens for deleted users rejected (#2522): Hub JWT auth returns 401 user_not_found, and /auth/validate reports valid: false for deleted or suspended users.
Agent tokens bound to the run (#2756): Agent tokens carry a run_id claim, and an observe setting logs mismatches with the agent’s current run.
Delegation and service account authority (#2523, #2524, #2557, #2648, #2813, #2829, #3190, #3306): Agent create records its delegation edge atomically, restore reactivates it, and reincarnate requires delegation authority in the target project. A boot migration and an admin API give delegation edges recorded before provenance a bounded ceiling. The Hub records who authorized each service account assignment and checks that this source still holds gcp_service_account.assign.
Relationship grants need live project access (#2601, #2657, #2505): Owner and ancestor grants for users, tokens and federated users require current, Hub-recorded project access.
Ownership and membership (#2514, #2540, #2495, #2501, #2635, #2609, #2659): The last-owner rule counts only usable owners and checks the state after the change. Project members groups cannot be role-binding principals. Orphaned memberships are removed, and principal IDs are validated and stored in canonical form.
Passthrough on the embedded Runtime Broker (#2545): On single-node deployments, an unscoped local platform administrator may use passthrough on the co-located Runtime Broker. Other Runtime Brokers still require ownership.
Agent visibility and registration (#2662, #2593, #2633, #2468): Agent lists return only agents the caller can read. Runtime Broker registration and secret rotation require user credentials, and a Runtime Broker may rotate only its own secret.
Path and command handling (#2584, #2571, #2570, #2626, #2610, #3014, #3268): Downloads and template caches write only inside their root, resource file paths must be canonical, and project delete removes only a direct child of a projects root. Deleting an agent’s files no longer touches a same-named agent in the global project. sciontool init runs root commands through a fixed system search path.
Fixed text in errors and logs (#2701, #2658, #2576, #2707, #2929, #2946, #2909): Workspace, start and sync failures send fixed client text and log details on the server side. Staged secret values are masked line by line in provisioner errors.
Web and sign-in (#2831, #2823, #2837, #3241, #3298, #3168): The CLI loopback callback page renders inserted values as plain text. The SPA shell is served no-store, and the CSP drops cdn.webawesome.com. Invite-only Hubs report invite_only, and the test sign-in route adds audit rows and rate limiting.
Cloud Storage links use verified identity (#3028): gs:// links in agent messages are fetched only when the agent’s service account is verified, available to its project and still assigned.
Harness-config import validation (#3110): Import and reimport refuse unusable provisioner blocks with 422 harness_config_unusable.

🤖 Agent Lifecycle

Race-safe create, start and delete (#2536, #2487, #2562, #2469, #2537, #2705, #2797, #2669, #2719, #3075, #3062, #3024, #2989, #3030, #3084, #3119, #3116): A create or start that lands after a delete no longer revives the agent. Rollback removes only the run’s own containers, and deferred deletes are fenced with notAfter. An in-doubt delete whose queued intent later succeeds is finalized.
Start claims and quota (#2628, #2641, #2508): Every start runs under a claim and reserves a quota slot before dispatch, restoring both on failure. Queued stops on HTTP-only Runtime Brokers are applied from heartbeats.
Detached operations (#2490, #3165, #3104, #3207, #2917, #2980): Synchronous launches, stop, suspend, stop-all and sync-from survive client disconnects and have longer write deadlines. Stop-time workspace sync-back is bounded.
Readiness and wake (#3332, #3284, #2555, #2554): Agents show “starting” until the harness is ready, and chat, DMs and wake buffer messages until then. A suspended agent can be woken from chat with “Wake and send”, which needs the agent lifecycle permission.
Stranded runs and missing containers (#3247, #2759, #2718, #3255, #3224, #3212, #3226): The Hub adopts an older run that a Runtime Broker still reports after a quiet period. Exec on a removed container moves the agent to phase error (container_missing). Restart repairs agent home ownership, and start recreates a missing agent’s workspace inputs.
Long tasks and clone depth (#2976, #2959): scion start and create accept --task-file <path|-> up to 96 KiB, and a clone_depth setting replaces the fixed depth-1 clone.
Provisioned agents shown clearly (#2507, #2907): Agents that were created but never started show “created (not started)” with a start hint.
Harness fixes (#2525, #3303, #3242, #3296, #3344, #3264, #3273): SCION_THINKING_LEVEL maps to Claude effort levels. Claude’s Vertex AI guess no longer overrides an API key. waiting_for_input clears once the prompt is answered.

☸️ Kubernetes & GCP Identity

Per-profile identity reports (#3152, #3193, #3117, #3095, #3220, #3219, #3285, #3282): Runtime Brokers report which Google service accounts each Kubernetes profile can use, and through which Kubernetes service account. scion project service-accounts show and the create and configure pickers show each account’s mapping state.
Identity options (#2598, #2620, #3000, #3021, #2999, #3052, #3048): Projects can set a default service account per Runtime Broker profile. --service-account accepts an ID, email or display name. A block identity runs as a zero-privilege service account, and missing mappings return identity_not_mapped or identity_ksa_mismatch.
Shared dirs and NFS (#2575, #2580, #2617, #2825, #3161): Agents on Docker and Kubernetes can edit each other’s NFS shared-dir files through shared groups. shared_dir_storage_backends picks local or nfs per dir. NFS workspaces clone private repos with the project’s git credential, which is stored in the per-agent Secret.
Pod behavior (#2515, #2551, #2481, #2893, #3126, #2956, #2887, #3351): Delete and pre-clean are run-scoped. Stopping an EmptyDir workspace agent warns about unpushed work. OOMKilled, preempted and evicted pods are classified with clear reasons. Pods get GOMAXPROCS and GOMEMLIMIT from their limits.

💬 Chat & Messaging

Scheduled send, experimental (#2807, #2901, #2845): Behind web.chat_scheduled_send, topics and DMs offer Schedule send, with a 60-minute late cutoff and recovery actions for failed messages.
Faster chat (#2497, #2491, #2485, #2513): A cold /chat load drops from 19 to 6 API calls, and the DM list drops from about 241 store queries to 5.
Message IDs and warnings (#2912, #2737, #3290, #2811): Delivered envelopes carry message_id. scion message prints the created ID, says “accepted” until delivery is confirmed, and warns about attachments the Hub could not read.
Group sends (#2700, #2693, #2583): Group sends and broadcast --all cap concurrency at 6. A2A returns input-required for replies that need input.
Chat plugins (#2572, #2608, #2670, #2667, #2868, #2855, #2850): Discord, Telegram, Teams and Slack plugins act as the linked user again. Discord question-button answers reach the agent. DMs to deleted agents are reported undelivered.
Chat UI (#2568, #2547, #3205, #3210, #3257, #3228, #3286, #3323, #2467): Undelivered thread messages are marked. Thread and scroll position survive mode switches. Pull to refresh works on the spaces rail, and the unread badge counts only spaces you belong to. Web chat replies no longer vanish after Discord or Telegram use. (Contributor: G. Hussain Chinoy)

📦 Artifacts (Experimental)

Publishing and versions (#2612, #2742, #2746, #2787, #2791, #2792): Behind hub.artifacts: scion artifact publish and get, content-addressed storage, bundles, versions, HTML and Markdown views, and a project Files area.
Messages and review (#2808, #2858, #3166, #3138): Messages carry up to 10 scion://artifact/ references, shown as chips with previews. CriticMarkup reviews come with a web Review mode.
Sharing and retention (#2871, #2879, #2854, #2930, #3054, #3363): Grants, moves between projects, per-artifact expiry, time-limited share links, scion artifact share and scion artifact delete.

📊 Health & Observability

Health dashboard redesign (#2772, #2779, #2803, #2862, #2918, #2938, #3042): A single policy derives overall status and a ranked Needs attention list from Hub checks, Runtime Broker self-health reports, NFS, dispatch, integrations and agent error rates. A configurable monitoring dashboard link appears on the page.
Hub-instance registry (#3078, #3144, #3100, #3160, #3191): Each Hub replica registers itself, and the health summary reports per-replica DB pool status and fleet status.
Monitoring and tracing (#2780, #2745, #2775, #3233): An importable Cloud Monitoring dashboard covers Hub metrics. server.hub.perf_trace logs per-request phase timings. Cloud Logging write failures are counted, with alert policies.
Session metrics (#2703, #2673, #3022, #2939, #3113, #3170, #3259): Native usage for Claude, Copilot, Codex and Gemini reaches persisted session metrics, without losses or double counts around shutdown.

☁️ Runtime Brokers, Deployment & Operations

Headless Runtime Broker registration (#2702, #2713, #2643, #2878): scion hub brokers join-token create mints a single-use join token for scion runtime-broker join. --broker-name lets several Runtime Brokers share one host.
GKE Hub via Helm behind IAP (#2689, #2529, #2527, #2651): New Terraform modules install the scion-hub chart behind a global load balancer with IAP.
Settings apply live (#3293, #3249, #3244, #3304, #2840): Layer-1 settings apply without a restart, and save responses list applied_keys and pending_restart.
Runtime Broker Hub connections (#3223, #3281, #3299, #3276, #3199, #3173, #3151, #3150): Connection start, stop and reinitialize run one at a time, retry with backoff and no longer race on shared fields.
Server lifecycle (#3291, #3256, #3277, #3212, #3226): scion server stop waits for exit, and SIGTERM shuts down gracefully. scion server start opens onboarding, and starts the Hub and web without image_registry. In workstation mode, co-located agents can use the harness login files in the user’s home, for built-in harness configs only.
Admin user provisioning (#2735, #2778): scion hub users provision and POST /api/v1/users pre-register invited users, with an optional display name.
Harness configs (#2646, #2715, #2695, #2590): A deleted built-in harness config or template stays deleted across Hub restarts and upgrades. Remote Runtime Brokers can download Hub-stored harness configs.
Health status (#2504, #2709, #2691): /healthz reports healthy, degraded or unhealthy, and CLI tools treat degraded as up while naming the failing checks.
Empty env values are unset (#2836, #2895): Exported but empty SCION_*, SCION_SERVER_* and SCION_SEED_* variables no longer override configured values.
Flat Runtime Brokers, experimental (#3045): Behind hub.flat_runtime_brokers, each Runtime Broker has one runtime target, and agents stay pinned to it.
Other (#2606, #2712, #2727, #3194, #3125, #2902): Cloud Run runtimes deliver Hub secrets. Co-located agents behind IAP reach the Hub locally. Port forwarding answers 504 when an agent port sends no response headers within server.hub.port_proxy.response_header_timeout (default 60 s). Starter Hub TLS renewals reload Caddy, and agent images include Helm v3.17.3.

💻 CLI

CLI improvements (#2561, #2843, #2916, #3057, #2941, #2940, #2814, #2812): scion attach gives actionable close messages. --expires accepts 2h and 90m. Hub commands run outside a project. stop --all and suspend --all cap concurrency at 6.
Template sources (#2891, #2903): POST /api/v1/templates/{id}/reimport refreshes a Hub template imported from a GitHub folder URL, and scion harness-config install records its source URL.
Consistent output (#2710, #2519, #2817, #2754, #2781): Secret, env and Hub secret lists honour --format json. Empty Hub responses return errors instead of panicking.

🎨 Web UI

Faster loads (#2577, #2653, #2492, #2793, #2794, #2810, #2861): Agent views use the compact list (79% smaller responses) with a 30-second delta probe. Shoelace loads only from the bundle, and fingerprinted assets are cached.
Navigation and actions (#2795, #2957, #3065, #2726, #3213, #3148): The Configuration tab has a Reincarnate action. Ctrl/Cmd+1-3 switch views. Terminal bulk actions and multi-pane layouts were improved.
Status wording (#2680, #2738, #2790): waiting_for_input now reads “Waiting on Parent” across the web UI, notifications and sciontool.
Mobile and layout (#2905, #2908, #2896, #3347, #3217, #2559, #2558, #2528): Browser Back steps through chat panels on phones. Detail headers wrap long names, and Created/Updated dates display. (Credit: miller79)

📖 Documentation

Guides (#2474, #2623, #3145, #3079, #3215, #3159): Raw message removal migration, cross-Runtime Broker moves, a Kubernetes GCP identity checklist, reset-auth scope re-issue and a rollback reference.
Corrections (#2806, #2880, #2860, #2977, #3073): Lifecycle API responses, multi-replica storage requirements and recovering deleted built-in harness configs.