Deploy on GCP (Cloud Run + GKE)

This guide walks through deploying a production-ready Scion HA Hub on Google Cloud Platform. The architecture uses Cloud Run for the Hub (with IAP authentication), GKE Autopilot for agent dispatch, and Cloud SQL for durable state.

What you will set up:

Component	GCP Service	Purpose
Hub	Cloud Run (HA, multi-instance)	Control plane, API, web UI
Database	Cloud SQL PostgreSQL	Durable state (Ent AutoMigrate)
Agent Runtime	GKE Autopilot	Isolated agent pods
Auth	Identity-Aware Proxy (IAP)	Zero-trust access control
Secrets	Secret Manager + CSI Driver	Secure secret distribution
Storage	GCS	Templates, artifacts, hub data
Images	Artifact Registry	Container image repository

Terraform alternative

This guide walks through the manual gcloud/kubectl steps. If you’d rather provision this same architecture declaratively — including support for multiple hubs sharing one project’s shared infrastructure (network, Cloud SQL, Filestore, GKE Autopilot, Artifact Registry) — see Multi-Hub HA with Terraform and the Terraform module set at deploy/terraform/README.md in the repository. It automates most of the steps below; the two approaches are not meant to be mixed against the same project.

---

0. Prerequisites & Deployer Identity

GCP Project

You need an active GCP project with billing enabled. Throughout this guide, replace $PROJECT_ID with your project ID and $PROJECT_NUMBER with your project number.

export PROJECT_ID="your-project-id"
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
export REGION="us-central1"          # Hub + Cloud SQL region
export GKE_REGION="us-west2"         # GKE cluster region (can differ from Hub)

Enable APIs

gcloud services enable \
  run.googleapis.com \
  sqladmin.googleapis.com \
  secretmanager.googleapis.com \
  container.googleapis.com \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com \
  iap.googleapis.com \
  iam.googleapis.com \
  --project=$PROJECT_ID

CLI Tools

Tool	Install	Verify
gcloud	cloud.google.com/sdk/install	gcloud version
kubectl	gcloud components install kubectl	kubectl version --client
gke-gcloud-auth-plugin	See below	gke-gcloud-auth-plugin --version
psql	sudo apt-get install -y postgresql-client	psql --version
scion CLI	Per Scion repo README	scion version

gke-gcloud-auth-plugin in Containers

In containerized, managed, or non-interactive environments (where standard gcloud components install is disabled), you must configure the Google Cloud SDK apt repository and install the plugin natively:

sudo apt-get install -y google-cloud-cli-gke-gcloud-auth-plugin

Otherwise, use the gcloud component manager:

gcloud components install gke-gcloud-auth-plugin

Authenticate

For Human Deployers (Interactive)

gcloud auth login
gcloud config set project $PROJECT_ID
gcloud auth application-default login

For Automated Deployers / Service Accounts (Non-Interactive / Cross-Project)

If an automated CI/CD agent or a service account from another project is executing this deployment:

1. Authenticate using the service account’s key file:

   gcloud auth activate-service-account --key-file=KEY_FILE_PATH
   gcloud config set project $PROJECT_ID

2. Note that interactive authentication commands (gcloud auth login and gcloud auth application-default login) are disabled and should be skipped.

---

Deployer Permission Requirements

The deployment identity (whether human or service account) must bypass two separate permission walls: project-level IAM bindings and service-account-level IAM bindings. Ensure the deployer has the following consolidated roles on the target project:

Role	Scope	Why It’s Needed
roles/resourcemanager.projectIamAdmin	Project-level	Needed to assign project-level IAM policies to service accounts (Section 2)
roles/iam.serviceAccountAdmin	Service-account-level	Needed to set service-account-level policy bindings (such as Token Creator in 2c, 2d and Workload Identity in 2g)
roles/run.admin	Service-level	Needed to deploy Cloud Run services and bind service-level permissions (run.invoker)
roles/editor	Project-level	General resource provisioning (Cloud SQL, GCS, Secret Manager, GKE)

Granting Cross-Project Deployer Access

If deploying from a service account in a different “Infrastructure” project (e.g., scion-deployer@infra-project.iam.gserviceaccount.com), grant the required roles to that service account on the target project before proceeding:

DEPLOYER_SA="scion-deployer@infra-project.iam.gserviceaccount.com"

for ROLE in \
  roles/resourcemanager.projectIamAdmin \
  roles/iam.serviceAccountAdmin \
  roles/run.admin \
  roles/editor; do
  gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$DEPLOYER_SA" \
    --role="$ROLE" \
    --quiet
done

---

1. Infrastructure Setup

1a. Artifact Registry

gcloud artifacts repositories create scion \
  --repository-format=docker \
  --location=$REGION \
  --project=$PROJECT_ID

Set the registry prefix (used throughout):

export IMAGE_REGISTRY="$REGION-docker.pkg.dev/$PROJECT_ID/scion"

1b. Cloud SQL — PostgreSQL Database

gcloud sql instances create scion-hub-db \
  --database-version=POSTGRES_16 \
  --edition=ENTERPRISE \
  --tier=db-f1-micro \
  --region=$REGION \
  --project=$PROJECT_ID \
  --database-flags=max_connections=200

Enterprise Edition Required for Micro Tier

You must explicitly provide --edition=ENTERPRISE in the instance creation command. If omitted, some projects/organizations default to ENTERPRISE_PLUS, which does not support the cost-effective db-f1-micro tier.

Why max_connections=200?

Cloud Run can spin up multiple revisions during deploys. With the Hub, Discord, and agent services all sharing one Cloud SQL instance, 200 connections provides headroom. The default of 100 is too low — you will hit SQLSTATE 53300 (too many connections) during redeploys.

Create the database user and database:

# Set password (choose your own secure password)
export DB_PASSWORD="YourSecurePassword"

gcloud sql users create scion \
  --instance=scion-hub-db \
  --password=$DB_PASSWORD \
  --project=$PROJECT_ID

# The database is created automatically by Ent AutoMigrate on first Hub boot.
gcloud sql databases create scionhub \
  --instance=scion-hub-db \
  --project=$PROJECT_ID

1c. GCS Bucket

export BUCKET_NAME="scion-hub-$PROJECT_ID"

gsutil mb -l $REGION -p $PROJECT_ID gs://$BUCKET_NAME/

1d. GKE Autopilot Cluster

gcloud container clusters create-auto scion-agents \
  --region=$GKE_REGION \
  --project=$PROJECT_ID \
  --enable-secret-manager

Tip

The --enable-secret-manager flag installs the Secrets Store CSI Driver add-on for GKE, allowing agent pods to mount secrets directly from Secret Manager as files. If you forget this flag, you can enable it later with gcloud container clusters update.

GKE uses different CSI names than upstream

Component	Upstream (will NOT work)	GKE Managed (correct)
CSI Driver	secrets-store.csi.x-k8s.io	secrets-store-gke.csi.k8s.io
Provider	gcp	gke
:::

Shared Directories and Volume Capabilities Failure

When creating a project through the Hub web UI, Scion includes a default “scratchpad” shared directory. Under the hood, this mounts a Kubernetes PersistentVolumeClaim with ReadWriteMany (RWX) access mode.

• The Catch: GKE Autopilot’s default storage class (standard-rwo) only supports ReadWriteOnce (RWO). Attempting to dispatch an agent will fail with an opaque scheduling error: VolumeCapabilities is invalid: specified multi writer with mount access type, and the Hub will output pods not found.
• The Fix (Option A - Easiest): If you do not require shared directories, navigate to your project settings in the Hub UI after installation and remove or disable the default shared directory (e.g., delete the scratchpad entry).
• The Fix (Option B - Production): Set up Google Cloud Filestore or a compatible NFS server, configure the Filestore CSI driver, and define the custom storage class in your GKE runtime profile to support native ReadWriteMany volumes.
• The Fix (Option C - Shared NFS export): Set server.shared_dir_storage.backend to nfs with a static PVC (pv_name) bound to your Filestore or NFS export. Pods then mount shared directories from that single claim by subPath instead of requesting a new RWX PVC per directory.

Create the agent namespace:

# Get credentials for kubectl
gcloud container clusters get-credentials scion-agents \
  --region=$GKE_REGION --project=$PROJECT_ID

kubectl create namespace scion-agents

Verify:

kubectl get namespace scion-agents
# Expected: scion-agents   Active

# Verify CSI driver is running
kubectl get pods -n kube-system | grep csi-secrets-store
# Expected: csi-secrets-store-gke-* pods Running

1e. Secret Manager — Kubeconfig for Hub

The Hub needs a kubeconfig to dispatch agents to GKE. Store it in Secret Manager:

# Generate a kubeconfig for the scion-agents cluster
KUBECONFIG=/tmp/gke-kubeconfig.yaml gcloud container clusters get-credentials \
  scion-agents --region=$GKE_REGION --project=$PROJECT_ID

# Store in Secret Manager
gcloud secrets create scion-gke-kubeconfig \
  --data-file=/tmp/gke-kubeconfig.yaml \
  --project=$PROJECT_ID

# Clean up local file
rm /tmp/gke-kubeconfig.yaml

Note

The kubeconfig contains the cluster endpoint and CA cert but not credentials. The Kubernetes client on Cloud Run falls back to GCE metadata auth using the Cloud Run service account’s ambient credentials. This works because the Hub SA has roles/container.developer (granted in the IAM section below).

1f. Generate Session Secret

The Hub uses a 32-byte hex session signing key:

export SCION_SERVER_SESSION_SECRET=$(openssl rand -hex 32)
echo "Session secret: $SCION_SERVER_SESSION_SECRET"
# Save this — you'll use it in the deploy command.

Infrastructure Verification Checklist

echo "=== Infrastructure Verification ==="
echo "Artifact Registry:"
gcloud artifacts repositories list --location=$REGION --project=$PROJECT_ID \
  --format="value(name)" | grep scion && echo "  OK" || echo "  MISSING"

echo "Cloud SQL:"
gcloud sql instances describe scion-hub-db --project=$PROJECT_ID \
  --format="value(state)" 2>/dev/null && echo "  OK" || echo "  MISSING"

echo "GCS Bucket:"
gsutil ls -b gs://$BUCKET_NAME/ >/dev/null 2>&1 && echo "  OK" || echo "  MISSING"

echo "GKE Cluster:"
gcloud container clusters describe scion-agents --region=$GKE_REGION \
  --project=$PROJECT_ID --format="value(status)" 2>/dev/null && echo "  OK" || echo "  MISSING"

echo "Kubeconfig Secret:"
gcloud secrets describe scion-gke-kubeconfig --project=$PROJECT_ID \
  --format="value(name)" 2>/dev/null && echo "  OK" || echo "  MISSING"

---

2. IAM Setup

Critical Section

The majority of deployment failures trace back to missing IAM bindings. Every binding below was discovered through real deployment failures.

2a. Create Service Accounts

# Hub runtime SA — runs the Cloud Run Hub service
gcloud iam service-accounts create scion-hub-runner \
  --display-name="Scion Hub Runner" \
  --project=$PROJECT_ID

# Transport SA — identity used to mint IAP tokens for GKE agents
gcloud iam service-accounts create scion-transport \
  --display-name="Scion Transport Token Minter" \
  --project=$PROJECT_ID

# Discord runtime SA — runs the Discord Cloud Run service
gcloud iam service-accounts create scion-discord-runner \
  --display-name="Scion Discord Runner" \
  --project=$PROJECT_ID

2b. Hub Runner SA — Project-Level Roles

SA_HUB="scion-hub-runner@$PROJECT_ID.iam.gserviceaccount.com"

for ROLE in \
  roles/cloudsql.client \
  roles/storage.objectAdmin \
  roles/secretmanager.admin \
  roles/container.developer; do

  gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SA_HUB" \
    --role="$ROLE" \
    --quiet
done

Role	Purpose
roles/cloudsql.client	Connect to Cloud SQL via Unix socket
roles/storage.objectAdmin	Read/write GCS bucket (templates, artifacts)
roles/secretmanager.admin	Required. Read, write, and create Secret Manager secrets (e.g. API keys saved by users via the Hub web UI). Traditional viewer/accessor roles are too restrictive and cause “Internal Error” failures when writing secrets.
roles/container.developer	Dispatch agent pods to GKE, create K8s resources

2c. Hub Runner SA — Token Creator on Transport SA

Most commonly missed binding

Without this binding, the Hub cannot mint IAP identity tokens for GKE agents. The failure is silent — no error in Hub logs, but SCION_TRANSPORT_TOKEN is simply not injected into GKE pods. The agent then fails with "Invalid IAP credentials: empty token".

This binding is on the transport SA, not at the project level.

SA_TRANSPORT="scion-transport@$PROJECT_ID.iam.gserviceaccount.com"

gcloud iam service-accounts add-iam-policy-binding \
  $SA_TRANSPORT \
  --member="serviceAccount:$SA_HUB" \
  --role="roles/iam.serviceAccountTokenCreator" \
  --project=$PROJECT_ID

Why this exists: When dispatching a GKE agent, the Hub mints an OIDC identity token on behalf of the transport SA (via gcpTransportMinter.MintIDToken). This token is injected as SCION_TRANSPORT_TOKEN into the agent pod. The agent uses this token as Authorization: Bearer for all Hub API calls through IAP. The serviceAccountTokenCreator role authorizes the Hub SA to mint tokens as the transport SA.

2d. Hub Runner SA — Token Creator on Itself

The Hub generates signed GCS URLs for template files when dispatching agents. This operation requires the iam.serviceAccounts.signBlob permission, which requires the Hub Runner SA to have the Token Creator role on itself:

gcloud iam service-accounts add-iam-policy-binding \
  $SA_HUB \
  --member="serviceAccount:$SA_HUB" \
  --role="roles/iam.serviceAccountTokenCreator" \
  --project=$PROJECT_ID

Without this binding, template generation will fail with: Permission 'iam.serviceAccounts.signBlob' denied on resource.

2e. Hub Runner SA — IAM Security Reviewer (Optional, for Policy Troubleshooter)

When you enable GCP IAM check mode (gcp_iam_check_mode: enforce), Scion verifies that human and agent callers possess iam.serviceAccounts.actAs permission on assigned service accounts via the GCP Policy Troubleshooter v3 API.

For Policy Troubleshooter to evaluate permissions across your project (or your entire organization if you assign SAs from multiple projects), the Hub Runner SA must be granted the IAM Security Reviewer role (roles/iam.securityReviewer):

# Grant Security Reviewer at the project level
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_HUB" \
  --role="roles/iam.securityReviewer" \
  --quiet

# (Optional) For multi-project or organization-wide SAs, grant at the org level instead:
# export ORG_ID="your-gcp-organization-id"
# gcloud organizations add-iam-policy-binding $ORG_ID \
#   --member="serviceAccount:$SA_HUB" \
#   --role="roles/iam.securityReviewer"

Without this permission, Policy Troubleshooter checks will return an indeterminate or unknown status (such as ACCESS_STATE_UNKNOWN_INFO_DENIED), which results in a fail-closed denial of service account assignment in Scion.

The Hub’s identity needs access to run the assignment check for each target service account; until it has that access, service account assignment is denied. If you assign service accounts from other GCP projects and have not granted the role at the organization level, grant it on each of those projects too (replace $SA_PROJECT_ID with the project that holds the service account):

gcloud projects add-iam-policy-binding $SA_PROJECT_ID \
  --member="serviceAccount:$SA_HUB" \
  --role="roles/iam.securityReviewer"

Without it, assigning a service account from that project is refused with a message that names the project and gives this command. Registering or verifying such an account returns the same message as a warning, so you can grant the role before the first assignment. If the check cannot run because of the Hub’s identity, the admin health summary reports it (see Hub Identity Access for the Assignment Check).

2f. Transport SA — IAP Access

The transport SA’s identity is used in the IAP token. IAP must allow this identity to access the Hub:

# Grant IAP access at project level
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_TRANSPORT" \
  --role="roles/iap.httpsResourceAccessor" \
  --quiet

Why roles/iap.httpsResourceAccessor and not just roles/run.invoker?

When Cloud Run has native IAP enabled, roles/run.invoker alone is NOT sufficient. The request passes through the IAP layer first, which checks roles/iap.httpsResourceAccessor. Without it, GKE agents get 403: Access denied even with a valid token.

Also grant Cloud Run invoker (needed for the underlying service):

# Run this command AFTER the scion-hub service is created in Section 3
gcloud run services add-iam-policy-binding scion-hub \
  --member="serviceAccount:$SA_TRANSPORT" \
  --role="roles/run.invoker" \
  --project=$PROJECT_ID \
  --region=$REGION

Tip

Make sure your deployer identity has roles/run.admin before running this, as it modifies service-level IAM.

2g. Discord Runner SA — IAP Access

The Discord service calls Hub APIs through IAP (for registration, message routing, etc.):

SA_DISCORD="scion-discord-runner@$PROJECT_ID.iam.gserviceaccount.com"

# Cloud SQL access (for Discord's database interactions)
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_DISCORD" \
  --role="roles/cloudsql.client" \
  --quiet

# IAP access to call Hub APIs
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$SA_DISCORD" \
  --role="roles/iap.httpsResourceAccessor" \
  --quiet

2h. GKE Workload Identity — Agent Pod Secret Access

Agent pods need Workload Identity (WI) bindings to access Secret Manager for CSI-mounted secrets:

# Use the default compute SA or create a dedicated one
export GKE_GSA="${PROJECT_NUMBER}-compute@developer.gserviceaccount.com"

# Annotate the KSA in scion-agents namespace
kubectl annotate serviceaccount default \
  --namespace=scion-agents \
  iam.gke.io/gcp-service-account=$GKE_GSA \
  --overwrite

# Grant the WI binding (KSA → GSA)
gcloud iam service-accounts add-iam-policy-binding \
  $GKE_GSA \
  --role=roles/iam.workloadIdentityUser \
  --member="serviceAccount:$PROJECT_ID.svc.id.goog[scion-agents/default]" \
  --project=$PROJECT_ID

# Ensure the GSA can access secrets
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$GKE_GSA" \
  --role="roles/secretmanager.secretAccessor" \
  --quiet

Caution

The WI binding is namespace-scoped — it binds scion-agents/default specifically. Pods dispatched to a different namespace (e.g. default) will fail with PermissionDenied: secretmanager.versions.access denied.

2i. GKE Workload Identity for GCP Identity Mode Assign

GCP identity mode assign on the Kubernetes runtime authenticates agent pods via Workload Identity, not via the sciontool metadata emulator used on Docker. Scion does not create, annotate, or bind Kubernetes ServiceAccounts (KSAs) for this — pre-provision each one yourself, then tell the broker about it in its own settings.yaml.

This section covers the cluster side: the Workload Identity binding and the broker mapping. Registering the GSA with the Hub and granting the Hub token-creator on it come first; see the setup checklist for the full order.

Cluster and node pool prerequisites (once per cluster, if not already done for 2h):

# Enable Workload Identity Federation on the cluster.
gcloud container clusters update CLUSTER_NAME \
  --location=REGION \
  --workload-pool=$PROJECT_ID.svc.id.goog

# Enable GKE_METADATA on every node pool agent pods run on.
gcloud container node-pools update NODE_POOL_NAME \
  --cluster=CLUSTER_NAME \
  --location=REGION \
  --workload-metadata=GKE_METADATA

Autopilot clusters

GKE Autopilot has Workload Identity Federation enabled on every node by default and has no node pools to update — skip the gcloud container node-pools update command above on Autopilot.

For every GCP service account (GSA) you plan to assign to agents on this cluster:

# Choose (or create) a KSA for this GSA, in the namespace agents dispatch to.
export AGENT_GSA="agent-worker@$PROJECT_ID.iam.gserviceaccount.com"
export AGENT_KSA="agent-worker-ksa"
export AGENT_NAMESPACE="scion-agents"

kubectl create serviceaccount $AGENT_KSA --namespace=$AGENT_NAMESPACE \
  --dry-run=client -o yaml | kubectl apply -f -

# Annotate the KSA with the GSA it is bound to.
kubectl annotate serviceaccount $AGENT_KSA \
  --namespace=$AGENT_NAMESPACE \
  iam.gke.io/gcp-service-account=$AGENT_GSA \
  --overwrite

# Grant the WI binding (KSA -> GSA), scoped to this namespace and KSA.
gcloud iam service-accounts add-iam-policy-binding \
  $AGENT_GSA \
  --role=roles/iam.workloadIdentityUser \
  --member="serviceAccount:$PROJECT_ID.svc.id.goog[$AGENT_NAMESPACE/$AGENT_KSA]" \
  --project=$PROJECT_ID

If this agent also uses gke: true volumes

Once a pod authenticates as $AGENT_KSA via Workload Identity, CSI Secret Manager mounts and GCS FUSE volumes on that pod also authenticate as $AGENT_GSA — not as the default KSA’s GSA from 2h. Grant $AGENT_GSA whatever those volumes need directly (for example roles/secretmanager.secretAccessor for CSI secrets, or bucket-level object access for FUSE), the same way 2h’s GSA needed it. 2h’s existing grant does not extend to this GSA.

Then add the GSA-to-KSA mapping to the runtime broker’s own global settings — never a project’s own settings.yaml, which this mapping deliberately ignores — under the kubernetes-typed runtime entry, or under a profile in that same global source to override the runtime-level mapping for agents created under that profile only:

runtimes:
  remote:
    type: kubernetes
    kubernetes_service_account_mappings:
      agent-worker@PROJECT_ID.iam.gserviceaccount.com: agent-worker-ksa

Where “global settings” lives depends on your deployment mode:

• File-only mode (no database configured): this is ~/.scion/settings.yaml on the broker host, as shown above.
• Broker in the same process as a database-backed Hub (the Cloud Run + Cloud SQL setup in this guide, which starts the Hub and the Runtime Broker together): as §3c below explains, runtimes and profiles are persisted to the database on first boot and the database then takes over as the source of truth for those sections — editing the settings.yaml secret afterward and redeploying has no effect on them, the same way it has no effect on admin_emails. Set or update the mapping with PUT /api/v1/admin/server-config (the Hub’s Settings admin page sends this same request) instead, sending the complete runtimes and/or profiles object you want in effect — a mapping added only to the settings.yaml secret after first boot is silently ignored, and every assign dispatch then fails with “no mapping” even though the file looks correct.
• Standalone Runtime Broker (running in a separate process from the Hub, even when the Hub uses a database): the broker reads only its own ~/.scion/settings.yaml; the Hub database’s runtimes and profiles do not apply to it.

A Kubernetes dispatch with GCP identity mode assign whose GSA has no entry here fails at dispatch time with an actionable error naming this setting — it does not fall back to the emulator or to the pod’s default identity. This mapping is authoritative: if the create or start request itself also names an explicit kubernetes.serviceAccountName that differs from the mapped KSA for that GSA, the dispatch is rejected rather than silently resolved either way. A serviceAccountName set only in a template (not on the request) is not a conflict — it is overridden by the mapping instead, the same way any other request-level value already overrides a template value.

Caution

Each mapping is namespace-scoped by the WI binding’s member string ($PROJECT_ID.svc.id.goog[NAMESPACE/KSA]), the same as 2h. An agent dispatched to a different namespace than the one you bound needs its own KSA, annotation, and binding — the kubernetes_service_account_mappings entry only names the KSA, not its namespace.

The namespace an assign dispatch runs in comes only from the broker’s global settings: the selected runtime entry’s namespace, then the runtime’s default namespace. Provision the mapped KSA in that namespace:

runtimes:
  remote:
    type: kubernetes
    namespace: scion-agents
    kubernetes_service_account_mappings:
      agent-worker@PROJECT_ID.iam.gserviceaccount.com: agent-worker-ksa

Profiles have no namespace setting of their own. A profile that needs a different namespace selects its own runtime entry, which sets that namespace and the mapping for the KSA provisioned there:

runtimes:
  remote:
    type: kubernetes
    namespace: scion-agents
    kubernetes_service_account_mappings:
      agent-worker@PROJECT_ID.iam.gserviceaccount.com: agent-worker-ksa
  remote-team:
    type: kubernetes
    namespace: team-agents
    kubernetes_service_account_mappings:
      agent-worker@PROJECT_ID.iam.gserviceaccount.com: team-worker-ksa
profiles:
  team:
    runtime: remote-team

A create or start request that names a namespace explicitly is accepted only when it equals that resolved namespace.

A project’s own settings.yaml can still set runtimes.<entry>.namespace or runtimes.<entry>.context for the entry an assign dispatch selects, and the pod would be placed with the project’s value. Such a dispatch is refused with 400, naming the runtime entry and both values; remove the project’s override, or select another runtime entry.

A broker whose server configuration forces a runtime (ForceRuntime, which skips profile resolution) does not consult profiles for this lookup either. It reads the mapping and the namespace from the runtime entry whose key is the forced runtime’s type name, for example runtimes: kubernetes:. The forced Kubernetes runtime places pods in its own namespace, so that namespace must equal the entry’s resolved namespace; otherwise the dispatch is refused with 400. The entry’s context is not consulted in this case.

IAM Verification

echo "=== Hub Runner SA Roles ==="
gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:$SA_HUB" \
  --format="table(bindings.role)"

echo ""
echo "=== Token Creator Binding on Transport SA ==="
gcloud iam service-accounts get-iam-policy $SA_TRANSPORT \
  --project=$PROJECT_ID \
  --format="yaml(bindings)"

echo ""
echo "=== Transport SA IAP Access ==="
gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:$SA_TRANSPORT" \
  --format="table(bindings.role)"

Complete IAM Reference

Service Account	Role	Scope	Purpose
scion-hub-runner	roles/cloudsql.client	Project	Cloud SQL connections
scion-hub-runner	roles/storage.objectAdmin	Project	GCS bucket read/write
scion-hub-runner	roles/secretmanager.admin	Project	Read, write, create secret values
scion-hub-runner	roles/container.developer	Project	GKE agent dispatch
scion-hub-runner	roles/iam.serviceAccountTokenCreator	On itself	Generate GCS signed URLs
scion-hub-runner	roles/iam.serviceAccountTokenCreator	On scion-transport SA	Mint IAP tokens for agents
scion-hub-runner	roles/iam.securityReviewer	Project/Org, plus each project that holds an assigned SA	Policy Troubleshooter (required for SA assignment when gcp_iam_check_mode: enforce)
scion-transport	roles/iap.httpsResourceAccessor	Project	Allow IAP access to Hub
scion-transport	roles/run.invoker	Hub service	Allow Cloud Run invocation
scion-discord-runner	roles/cloudsql.client	Project	Cloud SQL connections
scion-discord-runner	roles/iap.httpsResourceAccessor	Project	Call Hub API through IAP
GKE pods GSA	roles/secretmanager.secretAccessor	Project	CSI secret mounts
GKE pods GSA	roles/iam.workloadIdentityUser	On itself, for KSA	WI auth for pods

---

3. Hub Deployment (Cloud Run)

3a. Configure custom OAuth Client ID

For programmatic access to an IAP-protected Cloud Run service (which our GKE agent pods require), you must set up a custom OAuth Client ID:

1. In the GCP Console, go to APIs & Services > Credentials.
2. Click Create Credentials and select OAuth client ID.
3. Set the Application type to Web application.
4. Set the Name to scion-hub-iap.
5. Under Authorized redirect URIs, add the following native IAP redirect path: https://iap.googleapis.com/v1/oauth/clientIds/YOUR_CLIENT_ID:handleRedirect (Note: replace YOUR_CLIENT_ID with the actual client ID string generated after creation).
6. Export the Client ID for subsequent steps:

   export IAP_CLIENT_ID="your-client-id-string.apps.googleusercontent.com"

---

3b. Build Container Images

Docker Build Chain Architecture

Scion agent image building follows a strict dependency chain: $$\text{core-base} \longrightarrow \text{scion-base} \longrightarrow \text{harnesses (gemini-cli, etc.)}$$

1. core-base: Built on node:24-trixie-slim (Debian 13). Contains core tools (Go compiler, Git vendored from the Chainguard git image, unix packages, GCS FUSE).
2. scion-base: Copies repository code (cmd/, pkg/, etc.) and builds the scion and sciontool binaries on top of core-base.
3. Harnesses: Pulls scion-base and adds target agent packages (like @google/gemini-cli).

Compilation Constraint: -tags no_embed_web

By default, the Hub’s Go build process compiles and embeds the frontend web assets (web/dist/) into the final binary. However, compiling the frontend takes time and is not required for non-hub components.

Any container or binary build for a non-hub service (such as the A2A bridge, Discord broker, custom harnesses, or utility binaries) that imports from the pkg/ module must compile with -tags no_embed_web. If this tag is omitted, compilation will fail with errors in web/embed.go because the required frontend build artifacts do not exist.

Single-Architecture AMD64 Speed Up

Avoid Emulation Build Timeouts

The official cloudbuild-scion-base.yaml compiles multi-platform (linux/amd64,linux/arm64). Under Cloud Build, arm64 compiles run under slow QEMU emulation on standard amd64 machines, which routinely times out after 30 minutes. Since GKE Autopilot runs on standard amd64 nodes by default, you can completely bypass this limit and accelerate your build by building amd64-only images.

Additionally, because gcloud builds submit --tag does not support --build-arg, we must compile the dependent images using small, custom Cloud Build --config YAML files to feed the BASE_IMAGE build argument cleanly.

1. Clone the repository

git clone https://github.com/GoogleCloudPlatform/scion.git
cd scion

2. Build the Hub Image

gcloud builds submit . \
  --tag="$IMAGE_REGISTRY/hub:latest" \
  --project=$PROJECT_ID \
  --ignore-file=.dockerignore \
  --machine-type=e2-highcpu-8 \
  --quiet

Use --ignore-file=.dockerignore

The default .gcloudignore excludes the web/ frontend directory. Because the Hub’s Dockerfile compiles frontend assets and embeds them directly inside the Go binary, forgetting --ignore-file=.dockerignore will produce a Hub container missing its entire Web UI.

3. Build the Core Base Image

gcloud builds submit image-build \
  --tag="$IMAGE_REGISTRY/core-base:latest" \
  --dockerfile=core-base/Dockerfile \
  --project=$PROJECT_ID \
  --quiet

Submit image-build, not the repository root

core-base/Dockerfile copies image-build/lib/ (the shared toolchain and base-contract scripts it runs, also used by thick-prep/Dockerfile). Docker cannot read anything outside the build context, so the context has to be image-build/ and the Dockerfile path is relative to it.

4. Build the Scion Base Image

Create a single-arch temporary build file to inject the custom BASE_IMAGE cleanly:

cat <<EOF > /tmp/scion-base-build.yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [
    'build',
    '--build-arg', 'BASE_IMAGE=$IMAGE_REGISTRY/core-base:latest',
    '-t', '$IMAGE_REGISTRY/scion-base:latest',
    '-f', 'image-build/scion-base/Dockerfile',
    '.'
  ]
images:
- '$IMAGE_REGISTRY/scion-base:latest'
EOF

gcloud builds submit . --config /tmp/scion-base-build.yaml --project=$PROJECT_ID --quiet
rm /tmp/scion-base-build.yaml

5. Build the Gemini-CLI Harness Image

Create a single-arch temporary build file:

cat <<EOF > /tmp/gemini-cli-build.yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [
    'build',
    '--build-arg', 'BASE_IMAGE=$IMAGE_REGISTRY/scion-base:latest',
    '-t', '$IMAGE_REGISTRY/scion-gemini-cli:latest',
    '-f', 'harnesses/gemini-cli/Dockerfile',
    '.'
  ]
images:
- '$IMAGE_REGISTRY/scion-gemini-cli:latest'
EOF

gcloud builds submit . --config /tmp/gemini-cli-build.yaml --project=$PROJECT_ID --quiet
rm /tmp/gemini-cli-build.yaml

---

3c. Configure and Store settings.yaml

To eliminate extremely fragile shell escaping and prevent nested quotation failures, do not embed settings.yaml within startup commands. Instead, write a clean local settings.yaml, save it as a Secret Manager secret, and mount it natively as a file on Cloud Run.

No Cloud Run Profile Allowed

If the Hub is deployed on Cloud Run with GKE as the agent runtime, the settings.yaml must NOT include a Cloud Run (cr) runtime/profile block. Having both runtimes in the file causes transient routing issues where the broker incorrectly attempts to spin up agents via Cloud Run, outputting the error: cloudrun: PullImage not yet implemented. Keep the profile focused exclusively on Kubernetes.

1. Write the Configuration File

Create a local settings.yaml file:

schema_version: "1"                    # REQUIRED — must be first key
image_registry: REGISTRY_PLACEHOLDER   # Overwritten with $IMAGE_REGISTRY
active_profile: remote                 # Default dispatch runtime is GKE

# === RUNTIME DEFINITIONS ===
runtimes:
  remote:
    type: kubernetes
    gke: true                          # Enables CSI Secret Manager mounts

# === PROFILE MAPPINGS ===
profiles:
  remote:
    runtime: remote

# === SERVER CONFIGURATION ===
server:
  database:
    driver: postgres
    url: postgres://scion:DB_PASSWORD_PLACEHOLDER@/scionhub?host=/cloudsql/PROJECT_ID:REGION:scion-hub-db

  auth:
    mode: proxy
    proxy:
      provider: iap
      iap:
        audience: /projects/PROJECT_NUMBER/locations/REGION/services/scion-hub
    transport:
      mode: iap
      oidc_audience: IAP_CLIENT_ID_PLACEHOLDER
      platform_auth_sa: scion-transport@PROJECT_ID.iam.gserviceaccount.com

  # === OIDC IDENTITY PROVIDER ===
  # Enables the Hub to act as an OIDC provider, publishing JWKS endpoints
  # and minting identity tokens for agents to authenticate to external resources.
  oidc:
    enabled: true                      # Enabled to support agent identity tokens
    token_lifetime: 15m                # Lifetime of minted OIDC identity tokens

  # === OIDC FEDERATION ===
  # Enables inbound federation from trusted external OIDC providers.
  federation:
    enabled: false                     # Disabled by default, configure if needed
    trusted_issuers: []

  secrets:
    backend: gcpsm
    gcp_project_id: PROJECT_ID

  hub:
    hub_id: scion-hub-ha-prod           # Stable identity — see note below
    admin_emails:
      - your-admin@example.com          # Admin email addresses
    hub_name: scion-hub-ha

  storage:
    provider: gcs
    bucket: BUCKET_NAME_PLACEHOLDER

  broker:
    enabled: true
    host: 127.0.0.1

Set a stable hub_id

server.hub.hub_id is a short string (recommended: lowercase letters, digits, hyphens — e.g. a project or environment slug such as scion-hub-ha-prod) that identifies this Hub instance. All Cloud Run replicas must resolve to the same value, and it should be unique among hubs that share a GCP project or bucket. Without an explicit hub_id, the Hub falls back to an implicit ID (a hash of the Cloud Run service name, or a per-host, hostname-derived value elsewhere). That value changes if the service is renamed or recreated under another name, and it differs for any process that does not run under the same service. The HA preflight therefore requires it to be pinned.

The hub ID is permanent for the hub’s lifetime: changing it changes the name prefix of every Secret Manager secret this hub writes (scion-<sha256(hub_id)[:12]>-...), the GCS prefix (hubs/<hub_id>/...) and hub-scoped database rows, orphaning anything namespaced under the old value — see IAM Permissions and Secret Naming for the naming scheme. Set it once and keep every redeploy and revision on the same value. If server.hub.hub_id is missing on an HA deployment, the Hub refuses to start (hosted HA deployment requires an explicit server.hub.hub_id).

Critical: Distinguishing IAP Audiences

Configuring IAP requires two different audience formats used in separate contexts:

1. server.auth.proxy.iap.audience (Cloud Run native IAP audience path):

   • Format: /projects/PROJECT_NUMBER/locations/REGION/services/SERVICE_NAME
   • Purpose: Used by the Hub to validate incoming IAP-signed JWTs (from browsers and human API calls).
   • Where to find: GCP Console → Security → Identity-Aware Proxy → Select your backend service → click the three dots → select Signed Header JWT Audience.

2. server.auth.transport.oidc_audience (IAP OAuth Client ID):

   • Format: PROJECT_NUMBER-xxxx.apps.googleusercontent.com
   • Purpose: Used as the audience minted into OIDC tokens for dispatched agents and brokers to authenticate and traverse Google IAP. IAP requires the OAuth client ID format for validating programmatically minted OIDC tokens, not the Cloud Run resource path.
   • Where to find: GCP Console → Security → Identity-Aware Proxy → Select your backend service → click the three dots → select Edit OAuth Client.

Using the wrong format for either field will cause startup verification or agent authentication to fail.

Proxy-Authorization Support

Cloud Run native IAP fully supports the Proxy-Authorization: Bearer <Google OIDC ID token> header. Dispatched agents and brokers can use either Authorization or Proxy-Authorization for the outer transport layer to pass through IAP. This prevents collisions if your client needs to use the standard Authorization header for internal Hub authentication.

Replace placeholders with your live values:

sed -e "s|REGISTRY_PLACEHOLDER|$IMAGE_REGISTRY|" \
    -e "s|DB_PASSWORD_PLACEHOLDER|$DB_PASSWORD|" \
    -e "s|PROJECT_ID|$PROJECT_ID|g" \
    -e "s|PROJECT_NUMBER|$PROJECT_NUMBER|g" \
    -e "s|REGION|$REGION|g" \
    -e "s|IAP_CLIENT_ID_PLACEHOLDER|$IAP_CLIENT_ID|" \
    -e "s|BUCKET_NAME_PLACEHOLDER|$BUCKET_NAME|" \
    settings.yaml > /tmp/settings-final.yaml

Understanding admin_emails Settings Precedence

• What it is: admin_emails defines the list of Google/IAP accounts granted the Administrator role in the Hub Web UI, enabling them to configure projects and dispatches.
• The Precedence Trap: Scion’s settings follow a strict precedence order: Embedded Defaults → YAML Config → PostgreSQL Operational Settings → Environment Variables.
• The Friction: When the Hub starts for the first time, YAML values are persisted directly to the Database. Any subsequent updates to admin_emails inside settings.yaml will be silently ignored because database-stored operational settings override them.
• The Workaround: Once the Hub database is initialized, you must add administrators through the Hub Admin Web UI/API, or execute a database update manually.

2. Create the Secret in Secret Manager

gcloud secrets create scion-hub-settings \
  --data-file=/tmp/settings-final.yaml \
  --project=$PROJECT_ID

rm /tmp/settings-final.yaml

---

3d. Deploy the Hub to Cloud Run

We can now deploy the Hub to Cloud Run cleanly. Notice that we mount both the kubeconfig and the settings.yaml files as secrets directly under /etc/scion/ and /home/scion/.scion/, keeping the start command extremely simple and robust.

gcloud run deploy scion-hub \
  --image="$IMAGE_REGISTRY/hub:latest" \
  --region=$REGION \
  --project=$PROJECT_ID \
  --service-account=scion-hub-runner@$PROJECT_ID.iam.gserviceaccount.com \
  --add-cloudsql-instances=$PROJECT_ID:$REGION:scion-hub-db \
  --set-env-vars="SCION_DEPLOY=$(date +%s),KUBECONFIG=/etc/scion/kubeconfig.yaml,SCION_K8S_NAMESPACE=scion-agents,SCION_SERVER_SESSION_SECRET=$SCION_SERVER_SESSION_SECRET,SCION_CLOUDRUN_CPU_ALWAYS_ALLOCATED=true" \
  --set-secrets="/etc/scion/kubeconfig.yaml=scion-gke-kubeconfig:latest,/home/scion/.scion/settings.yaml=scion-hub-settings:latest" \
  --min-instances=2 \
  --max-instances=3 \
  --cpu=1 \
  --no-cpu-throttling \
  --memory=512Mi \
  --port=8080 \
  --timeout=900 \
  --command="/usr/local/bin/scion" \
  --args="server,start,--hosted,--enable-hub,--enable-runtime-broker,--enable-web,--foreground,--web-port,8080" \
  --no-allow-unauthenticated \
  --quiet

--no-cpu-throttling keeps CPU allocated outside requests. The Runtime Broker in this service needs it to finish asynchronous agent launches after the create response. SCION_CLOUDRUN_CPU_ALWAYS_ALLOCATED=true tells the server that CPU is always allocated, which silences its startup warning; remove the variable if you deploy with CPU throttling. See Asynchronous agent create.

Why min-instances=2?

An HA Hub needs at least two running replicas, so that losing one instance (crash, host maintenance, scale-in) never leaves the Hub with zero warm replicas. With --min-instances=1 there is no failover peer, and the deployment is not HA (see HA overview). --max-instances=3 leaves room for Cloud Run to scale up under load.

More than one replica requires shared workspace storage

The settings.yaml above leaves server.workspace_storage at its default local backend. With that backend each replica keeps hub-managed project workspaces on its own container storage, so replicas can see different workspace content, and the content is lost when an instance is replaced. On Cloud Run the Hub also refuses workspace writes (file edits, WebDAV, git clone) with 503 Service Unavailable while the backend is local (see Ephemeral Storage & 503 Safety Gate).

Before running more than one replica, declare a shared volume on the Cloud Run service, mount it at /mnt/<volume_name>, and select it in settings.yaml.

For example, to use a Filestore (NFS) share, export the service with gcloud run services describe scion-hub --region=${REGION} --format export > service.yaml, add the volume and its mount, and apply the file with gcloud run services replace service.yaml:

spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/execution-environment: gen2   # required for NFS volumes
    spec:
      containers:
      - image: IMAGE_URL                 # keep the existing container entry
        volumeMounts:
        - name: VOLUME_NAME
          mountPath: /mnt/VOLUME_NAME    # must be /mnt/ + volume_name in settings.yaml
      volumes:
      - name: VOLUME_NAME
        nfs:
          server: FILESTORE_IP           # Filestore instance IP address
          path: /FILE_SHARE              # Filestore file share name
          readOnly: false

The mount path must equal /mnt/VOLUME_NAME, where VOLUME_NAME is the volume_name setting below. The Cloud Run volume’s own name can differ; using the same name for both just keeps it simple. NFS volumes require the second generation (gen2) execution environment. Do not use an in-memory volume here: it is local to each instance and not shared between replicas, even though the readiness check passes because the path is mounted.

server:
  workspace_storage:
    backend: cloudrun-volume
    cloudrun_volume:
      volume_name: VOLUME_NAME       # the volume declared on the service
      subpath_root: projects         # default

The Hub derives every workspace path from /mnt/<volume_name>. If the volume is not mounted there, GET /readyz returns 503 instead of the Hub writing workspaces to container storage. A missing volume_name stops the Hub at startup. See Workspace Storage for all backends and fields.

Cloud Run Timeout Warning

We explicitly set --timeout=900 (15 minutes). When dispatching the very first agent, GKE Autopilot triggers node provisioning to scale up from 0 nodes, which routinely takes 5-10 minutes. The default Cloud Run timeout (300 seconds) will prematurely kill the request, return a 503 Service Unavailable, and tear down the initiating container. Set the timeout to at least 900 seconds to prevent this.

---

3e. Enable Cloud Run Native IAP

Once the Hub service is deployed, activate Native IAP via the CLI:

gcloud run services update scion-hub \
  --iap=enabled \
  --region=$REGION \
  --project=$PROJECT_ID

Finalize Invoker Permissions

Grant invoker permissions to both the scion-transport service account and the native IAP service agent so the authentication pipeline completes smoothly:

# Allow IAP service agent to invoke Cloud Run
gcloud run services add-iam-policy-binding scion-hub \
  --region=$REGION \
  --project=$PROJECT_ID \
  --member="serviceAccount:service-$PROJECT_NUMBER@gcp-sa-iap.iam.gserviceaccount.com" \
  --role="roles/run.invoker" \
  --quiet

# Allow scion-transport SA to invoke Cloud Run
gcloud run services add-iam-policy-binding scion-hub \
  --region=$REGION \
  --project=$PROJECT_ID \
  --member="serviceAccount:scion-transport@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/run.invoker" \
  --quiet

---

3f. Verify Hub Health

Retrieve your live Hub URL:

export HUB_URL=$(gcloud run services describe scion-hub \
  --region=$REGION --project=$PROJECT_ID \
  --format="value(status.url)")
echo "Hub URL: $HUB_URL"

Critical: New Cloud Run URL Format & Hub Endpoint Resolution

Cloud Run service URLs are provisioned in two formats:

• Legacy: https://scion-hub-PROJECT_NUMBER.REGION.run.app
• New (default for newer projects): https://scion-hub-HASH-REGION.a.run.app

When the Hub is protected by IAP, it attempts to automatically resolve its own public URL from the IAP audience path. However, this automatic resolution only works for the legacy format.

If your printed Hub URL uses the new format containing a random hash (such as .a.run.app), the Hub’s auto-derivation will fail, causing agent dispatching and OIDC federation to break. You must set the SCION_SERVER_BASE_URL environment variable explicitly to resolve this.

Action Required: If your URL uses the new format, update your Cloud Run service to set this variable now:

gcloud run services update scion-hub \
  --region=$REGION \
  --project=$PROJECT_ID \
  --update-env-vars="SCION_SERVER_BASE_URL=$HUB_URL"

For more information on how the Hub resolves endpoints, see the Hub Endpoint Resolution Reference.

Verify that unauthenticated endpoints are successfully blocked by IAP:

curl -s -o /dev/null -w "%{http_code}" "$HUB_URL/"
# Expected: 302 (redirect to Google sign-in) or 403 (IAP block)

Verify authenticated /health check

Avoid /healthz Endpoints

Do not hit $HUB_URL/healthz. The HA Hub serves operational diagnostics strictly at /health (the legacy /healthz path returns a Google IAP 404 page, which can mislead administrators into thinking their entire IAP routing is broken).

TOKEN=$(gcloud auth print-identity-token --audiences="$IAP_CLIENT_ID")

curl -s "$HUB_URL/health" \
  -H "Authorization: Bearer $TOKEN"
# Expected: {"status":"healthy",...}

---

4. Discord Service Deployment (Cloud Run)

The Scion Discord service links Discord servers directly to the Hub API, enabling Slack/Discord-styled interactive chat dispatches.

4a. Discord Bot Setup (Developer Portal)

Before deploying, you must register a bot application with Discord:

1. Navigate to the Discord Developer Portal.
2. Click New Application and choose a name (e.g. scion-bot).
3. Under the Bot tab:
   • Click Reset Token and copy the generated Bot Token (save this as DISCORD_BOT_TOKEN).
   • Under Privileged Gateway Intents, enable Presence Intent, Server Members Intent, and Message Content Intent (required to read client messages).
4. Under the OAuth2 tab, note the Client ID (save this as DISCORD_APPLICATION_ID).
5. Open your Discord client, enable developer mode (User Settings > Advanced > Developer Mode), right-click your Discord Server, and copy its ID (save this as DISCORD_GUILD_IDS).
6. Generate the Bot Invite Link:
   • Go to OAuth2 > URL Generator.
   • Under Scopes, select bot.
   • Under Bot Permissions, select Send Messages, Read Message History, and View Channel.
   • Copy the generated URL at the bottom and load it in your browser to invite the bot to your target server.

---

4b. Build Discord Image

Ensure the build context is set to the repo root (.) and not extras/scion-discord/. The Discord service depends on parent modules within the workspace directory, so building from the subdirectory will throw compilation errors.

gcloud builds submit . \
  --tag="$IMAGE_REGISTRY/discord:latest" \
  --dockerfile=extras/scion-discord/Dockerfile \
  --project=$PROJECT_ID \
  --quiet

---

4c. Deploy Discord Service

gRPC Port Configuration

The Discord plugin is an internal gRPC service listening on port 50051 by default. Under Cloud Run, the TCP startup probe routes traffic to the container port (8080 by default), which causes immediate startup probe crashes and deployment failures. To resolve this:

1. You must inject the environment variable GRPC_PORT=8080 to instruct the service to bind to Cloud Run’s port.
2. You must enable HTTP2 support on Cloud Run using the --use-http2 flag so gRPC traffic flows correctly.

Required Environment Variables:

Variable	Description
DATABASE_URL	Cloud SQL PostgreSQL connection string
DISCORD_STANDALONE	Instructs the plugin to run in standalone listener mode (true)
DISCORD_BOT_TOKEN	Your secure Discord bot token
DISCORD_APPLICATION_ID	Your Discord client/application ID
DISCORD_GUILD_IDS	Comma-separated target Discord Guild (server) IDs
DISCORD_HUB_URL	The fully qualified public URL of your Hub ($HUB_URL)
DISCORD_TRANSPORT_MODE	Set to iap for Identity-Aware Proxy auth
DISCORD_TRANSPORT_AUDIENCE	The OAuth client ID of the Hub ($IAP_CLIENT_ID)

Deployment Command:

gcloud run deploy scion-discord \
  --image="$IMAGE_REGISTRY/discord:latest" \
  --region=$REGION \
  --project=$PROJECT_ID \
  --service-account=scion-discord-runner@$PROJECT_ID.iam.gserviceaccount.com \
  --add-cloudsql-instances=$PROJECT_ID:$REGION:scion-hub-db \
  --use-http2 \
  --port=8080 \
  --set-env-vars="GRPC_PORT=8080,DISCORD_STANDALONE=true,DATABASE_URL=postgres://scion:${DB_PASSWORD}@/scionhub?host=/cloudsql/$PROJECT_ID:$REGION:scion-hub-db,DISCORD_BOT_TOKEN=$DISCORD_BOT_TOKEN,DISCORD_APPLICATION_ID=$DISCORD_APPLICATION_ID,DISCORD_GUILD_IDS=$DISCORD_GUILD_IDS,DISCORD_HUB_URL=$HUB_URL,DISCORD_TRANSPORT_MODE=iap,DISCORD_TRANSPORT_AUDIENCE=$IAP_CLIENT_ID" \
  --min-instances=1 \
  --max-instances=1 \
  --cpu=1 \
  --memory=512Mi \
  --no-allow-unauthenticated \
  --quiet

The Discord service is a singleton, not HA

--min-instances=1 --max-instances=1 is intentional. The Discord plugin holds a single, unsharded Gateway connection per bot token; a second instance would receive and process every event twice, so the service runs as exactly one instance and is not replicated like the Hub. It is not covered by the HA guarantee. While it is down or restarting (for example during the redeploy checklist in Section 7a), the Hub keeps serving but the Discord integration is unavailable.

---

5. GKE Agent Dispatch Verification

By this point you should have:

• GKE Autopilot cluster with Secret Manager CSI add-on (Section 1d)
• scion-agents namespace created (Section 1d)
• Kubeconfig in Secret Manager (Section 1e)
• Workload Identity binding (Section 2g)
• Hub deployed with GKE support (Section 3d)

5a. Verify GKE Configuration

# Verify CSI driver is running
kubectl get csidrivers | grep secrets-store
# Expected: secrets-store-gke.csi.k8s.io

# Verify WI annotation on default KSA in scion-agents namespace
kubectl get serviceaccount default -n scion-agents -o yaml | grep gcp-service-account
# Expected: iam.gke.io/gcp-service-account: $PROJECT_NUMBER-compute@developer.gserviceaccount.com

# Verify the WI IAM binding
gcloud iam service-accounts get-iam-policy $GKE_GSA --project=$PROJECT_ID
# Expected: binding for serviceAccount:$PROJECT_ID.svc.id.goog[scion-agents/default]

5b. Settings.yaml Reference for GKE Dispatch

Our settings.yaml only exposes GKE (remote) to ensure zero profile/runtime resolution conflicts:

schema_version: "1"
image_registry: us-central1-docker.pkg.dev/your-project/scion  # WHERE agent images live
active_profile: remote                                           # DEFAULT dispatch target

runtimes:
  remote:
    type: kubernetes
    gke: true          # true = CSI Secret Manager mounts

profiles:
  remote:
    runtime: remote    # Maps to runtimes.remote (kubernetes)

gke Setting	Secret Storage	Requires CSI Add-on	Secret Values in Pod Spec
false	K8s native Secrets	No	Yes (in K8s Secret object)
true	CSI → Secret Manager direct	Yes	No (fetched at mount time)

Use gke: true for production (better security posture — secret values never appear in Kubernetes objects). Use gke: false if the CSI add-on is not installed or for quick testing.

Agent IAP Auth Flow

Understanding this flow is essential for debugging:

1. Hub dispatches agent pod to GKE
2. Hub mints IAP token via scion-transport SA
   └─ Requires: roles/iam.serviceAccountTokenCreator ON scion-transport SA
3. Token injected as SCION_TRANSPORT_TOKEN env var in pod
4. sciontool in the pod reads SCION_TRANSPORT_TOKEN → "injected" mode
5. sciontool uses token as Authorization: Bearer for Hub API calls
6. IAP validates the token against the OAuth client ID
   └─ Requires: scion-transport has roles/iap.httpsResourceAccessor
7. Hub processes the request

If step 2 fails silently (missing IAM binding): SCION_TRANSPORT_TOKEN is empty → sciontool skips OIDC → IAP rejects with "Invalid IAP credentials: empty token".

---

6. First Agent Test (E2E Verification)

6a. Dispatch a Test Agent

# Get IAP token
TOKEN=$(gcloud auth print-identity-token --audiences="$IAP_CLIENT_ID")

# Check Hub is reachable
curl -s "$HUB_URL/health" -H "Authorization: Bearer $TOKEN"
# Expected: {"status":"healthy",...}

# Dispatch a test agent
curl -s -X POST "$HUB_URL/api/v1/agents" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "e2e-test-1",
    "template": "gemini-cli",
    "task": "echo Hello from GKE"
  }'
# Expected: HTTP 201 with agent details

6b. Verify the Agent Pod

# Wait 30-60 seconds for pod to schedule on GKE Autopilot
# (Autopilot may need to provision a new node — can take 1-2 minutes)

kubectl get pods -n scion-agents
# Expected: spawn-test--e2e-test-1-xxxx in Running or ContainerCreating state

6c. Check Agent Logs

# Get the pod name
POD=$(kubectl get pods -n scion-agents -o name | grep e2e-test-1 | head -1)

# Stream logs
kubectl logs -f -n scion-agents $POD

# Look for key indicators
kubectl logs -n scion-agents $POD --tail=100 \
  | grep -i "transport\|IAP\|heartbeat\|error\|401\|403"

Success indicators:

Heartbeat succeeded
Transport token refresh scheduled

---

7. Maintenance & Troubleshooting

7a. Redeploy Checklist

When redeploying the Hub with a new image:

1. Scale Discord to min-instances=0 (prevents connection exhaustion):

   gcloud run services update scion-discord \
     --min-instances=0 \
     --region=$REGION --project=$PROJECT_ID --quiet

2. Sync workspace to latest before building:

   git fetch origin main && git rebase origin/main

3. Build with --ignore-file=.dockerignore (always):

   gcloud builds submit . \
     --tag="$IMAGE_REGISTRY/hub:latest" \
     --ignore-file=.dockerignore \
     --project=$PROJECT_ID --quiet

4. Deploy with --update-env-vars (not --set-env-vars):

   # CORRECT — preserves all other env vars:
   gcloud run services update scion-hub \
     --update-env-vars="SCION_DEPLOY=$(date +%s)" ...

5. Restore Discord to min-instances=1:

   gcloud run services update scion-discord \
     --min-instances=1 --max-instances=1 \
     --region=$REGION --project=$PROJECT_ID --quiet

7b. Secret Name Migration

Note

This path has not been run against a live hub; it was checked against the code and gcloud --help only.

Run scion hub secret migrate-names against a hub deployed with this guide from a one-off Cloud Run job, as described in §9 of docs/deploy/migrate-names-cloudrun.md. The job uses the serving revision’s image digest and runs as scion-hub-runner. It reaches this guide’s public-IP Cloud SQL instance through the same Cloud Run Cloud SQL connection as the hub, so it needs no VPC configuration. The DSN is supplied only through Secret Manager: the job mounts the scion-hub-settings secret with --set-secrets, and no operator reads or handles the DSN. Do not run migrate-names from a workstation against this hub’s database.

The job takes the hub ID from server.hub.hub_id in that settings file (see Set a stable hub_id in §3c), and the command checks it against the hub’s existing secret records. Sections 1, 4, 5, 6 and 8 of the runbook apply to this guide’s hubs, with the differences listed in its §9; section 7 (Terraform cleanup) does not. See Secrets: IAM Permissions and Secret Naming for what the command does in general, and --help for its flags.

---

Appendix A: Complete settings.yaml Reference

# === ROOT-LEVEL KEYS (not under server:) ===
schema_version: "1"                    # REQUIRED — must be first key
image_registry: us-central1-docker.pkg.dev/your-project/scion
active_profile: remote                 # Default dispatch runtime

# === RUNTIME DEFINITIONS ===
runtimes:
  remote:
    type: kubernetes
    gke: true                          # CSI secret mounts

# === PROFILE MAPPINGS ===
profiles:
  remote:
    runtime: remote

# === SERVER CONFIGURATION ===
server:
  database:
    driver: postgres
    url: postgres://scion:PASSWORD@/scionhub?host=/cloudsql/PROJECT:REGION:INSTANCE

  auth:
    mode: proxy
    proxy:
      provider: iap
      iap:
        audience: /projects/PROJECT_NUMBER/locations/REGION/services/scion-hub
    transport:
      mode: iap
      oidc_audience: PROJECT_NUMBER-xxxx.apps.googleusercontent.com  # OAuth client ID
      platform_auth_sa: scion-transport@PROJECT_ID.iam.gserviceaccount.com

  # === OIDC IDENTITY PROVIDER ===
  # Enables the Hub to act as an OIDC provider, publishing JWKS endpoints
  # and minting identity tokens for agents to authenticate to external resources.
  oidc:
    enabled: true                      # Enabled to support agent identity tokens
    token_lifetime: 15m                # Lifetime of minted OIDC identity tokens

  # === OIDC FEDERATION ===
  # Enables inbound federation from trusted external OIDC providers.
  federation:
    enabled: false                     # Disabled by default, configure if needed
    trusted_issuers: []

  secrets:
    backend: gcpsm
    gcp_project_id: your-project

  hub:
    hub_id: hub-ha-prod                # Stable identity — see Section 3c
    admin_emails:
      - admin@example.com
    hub_name: hub-ha-prod

  storage:
    provider: gcs
    bucket: scion-hub-your-project

  broker:
    enabled: true
    host: 127.0.0.1

---

Appendix B: Troubleshooting Quick Reference

1. IAP Token for Manual API Calls

IAP_CLIENT_ID="your-oauth-client-id.apps.googleusercontent.com"
TOKEN=$(gcloud auth print-identity-token --audiences="$IAP_CLIENT_ID")

2. JWT Token Debugging

# Decode a JWT to inspect claims
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool
# Check: aud, iss, email, exp

3. Hub API Endpoints

HUB="https://scion-hub-PROJECT_NUMBER.REGION.run.app"

# Health check
curl -s "$HUB/health" -H "Authorization: Bearer $TOKEN"

# List agents
curl -s "$HUB/api/v1/agents" -H "Authorization: Bearer $TOKEN" | python3 -m json.tool

# Dispatch agent
curl -s -X POST "$HUB/api/v1/agents" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"test-1","template":"gemini-cli","task":"echo hello"}'

4. Diagnostic Log Patterns & Opaque Failures

Volume Provisioning PVC Failure (pods "..." not found)

• Symptoms: Handlers return a generic dispatch timeout: failed to launch container: pods "test-project--test22" not found.
• Cause: Your Hub project includes a default scratchpad shared directory, which mounts a ReadWriteMany PVC. If GKE Autopilot only has the default standard-rwo storage class (which only supports ReadWriteOnce), GKE Autopilot fails to bind the volume and immediately destroys the pod.
• Diagnosis: Run kubectl get pvc -n scion-agents and kubectl describe pvc -n scion-agents. Look for: VolumeCapabilities is invalid: specified multi writer with mount access type.
• Resolution: Delete/disable the scratchpad shared directory from your project settings in the Hub UI, or provision a Google Cloud Filestore backend to support ReadWriteMany PVC mounts.

Pull Image Not Yet Implemented (cloudrun: PullImage not yet implemented)

• Symptoms: The Hub returns a PullImage not yet implemented error.
• Cause: The runtime broker has incorrectly attempted to route your dispatch request to a Cloud Run (cr) backend instead of your GKE (remote) backend.
• Resolution: Remove any cr / cloudrun profile or runtime mappings from your settings.yaml secret in Secret Manager, leaving GKE (remote) as the sole configured profile.

Administrative Email Changes Ignored

• Symptoms: You updated admin_emails in settings.yaml and redeployed, but the added user accounts still cannot perform administrative tasks in the Hub UI.
• Cause: Once the Hub database is initially booted, all YAML-configured Hub settings are written to the database. These database settings permanently supersede future YAML changes.
• Resolution: Explicitly update the settings through the Hub Admin UI, or manually modify the settings table inside your Cloud SQL PostgreSQL database.

Standard Error Resolution Table

Log Line	Root Cause	Fix
Heartbeat failed: ... empty token	Transport token not injected	Check serviceAccountTokenCreator IAM (Section 2c)
Heartbeat failed: ... 403: Access denied	Transport SA lacks IAP role	Grant iap.httpsResourceAccessor (Section 2e)
dial tcp 127.0.0.1:8080: connection refused	Hub endpoint is localhost	Upgrade Scion CLI to the latest version
driver name secrets-store.csi.x-k8s.io not found	Wrong CSI driver name on GKE	GKE requires secrets-store-gke.csi.k8s.io. Ensure gke: true is configured in settings.yaml.
PermissionDenied: secretmanager.versions.access	WI binding missing	Check GKE Workload Identity bindings (Section 2g)
SQLSTATE 53300	Too many DB connections	Scale dependent services (like Discord) to 0 instances before upgrading.