Observability

Scion provides comprehensive observability for agent containers and system components through the sciontool telemetry pipeline and OpenTelemetry log bridging. This guide covers how to monitor agent activity, collect logs, and integrate with cloud-native observability platforms like Google Cloud Logging and Trace.

Architecture Overview

Scion’s observability architecture follows a “forwarder” pattern where sciontool acts as a local collector inside each agent container, and system components (Hub and Broker) bridge their logs directly to a central backend.

┌─────────────────────────────────────────┐
│           Agent Container               │
│                                         │
│  ┌─────────────┐                       │
│  │   Agent     │ OTLP (localhost:4317) │
│  │  (Claude/   │───────┐               │
│  │   Gemini)   │       │               │
│  └─────────────┘       │               │
│                        ▼               │
│              ┌─────────────────┐       │
│              │   sciontool     │       │
│              │   forwarder     │       │
│              └────────┬────────┘       │
│                       │                │
│                       │ OTLP (Cloud)   │
└───────────────────────┼────────────────┘
                        │
                        ▼
              ┌─────────────────┐
              │  Cloud Backend  │
              │ (Logging/Trace) │
              └─────────────────┘
                        ▲
                        │ OTLP (Cloud)
              ┌─────────┴─────────┐
              │    System Logs    │
              │  (Hub & Broker)   │
              └───────────────────┘

Administrator Setup: Cloud Logging

To centralize logs and traces from all Scion components in Google Cloud, you must configure the OTLP endpoints and project identifiers.

Connecting Hub and Broker Logs

The Scion Hub and Runtime Broker use structured logging (slog) with an OpenTelemetry bridge. To enable log forwarding to Google Cloud:

1. Configure Environment Variables: Set the following on your Hub and Broker server processes:

   # Enable OTel log forwarding
   export SCION_OTEL_LOG_ENABLED=true
   
   # Set the GCP OTLP endpoint (standard for Cloud Trace/Logging)
   export SCION_OTEL_ENDPOINT="monitoring.googleapis.com:443"
   
   # Specify your GCP Project ID
   export SCION_GCP_PROJECT_ID="your-project-id"

2. Authentication: Ensure the service account running the Hub/Broker has the following IAM roles:

   • roles/logging.logWriter
   • roles/cloudtrace.agent
   • roles/monitoring.metricWriter

Direct Cloud Logging (Alternative)

As an alternative to the OTel pipeline, the Hub and Broker can send logs directly to Google Cloud Logging using the cloud.google.com/go/logging client library. This is simpler to set up when you only need log forwarding without traces or metrics:

# Enable direct Cloud Logging
export SCION_CLOUD_LOGGING=true
export SCION_GCP_PROJECT_ID="your-project-id"

# Optional: customize the log name (default: "scion-server")
export SCION_CLOUD_LOGGING_LOG_ID="scion-hub"

scion server start --enable-hub

Both approaches can be used simultaneously — OTel for the full telemetry pipeline and Cloud Logging for direct log delivery.

Google Cloud Error Reporting Integration

For platform operators running in Google Cloud, Scion’s logging handlers (GCPHandler and CloudHandler) natively integrate with Google Cloud Error Reporting to automatically discover, group, and track application errors:

1. Service Context Grouping: The loggers automatically inject a serviceContext group containing the system component name (e.g. scion-hub, scion-broker, or scion-server) and its build version on all log entries across all severity levels.
2. Automated Error Capture: On any log entry of severity ERROR or higher, the loggers automatically parse the caller’s stack frame and inject a stack_trace string along with a specific @type metadata annotation (set to type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent).
3. Preservation of Custom Annotations: These attributes allow GCP Error Reporting to instantly parse and group log entries. If your application or a custom hook logs a pre-structured error that already specifies a custom @type field, Scion preserves your custom annotation instead of overriding it.

This features works transparently with both the standard OTel log-bridging handler and the direct cloud.google.com/go/logging client library.

Configuring Agent Telemetry

Agents use sciontool as their init process, which includes an embedded OTLP forwarder. This forwarder must be configured to point to your cloud backend.

Via Settings File (Recommended)

The preferred approach is to configure telemetry in settings.yaml. Settings at the global level apply to all agents; project-level settings apply to a specific project. Templates cannot set telemetry (a template’s telemetry: block is ignored), and an individual agent can only turn it on or off; the project’s telemetry policy beats both. See the telemetry ladder.

# In ~/.scion/settings.yaml (global) or .scion/settings.yaml (project)
telemetry:
  enabled: true
  cloud:
    enabled: true
    # Required when exporting to Google Cloud — without it the metrics
    # dashboard has no project to query and stays empty.
    gcp_project_id: "my-gcp-project"
    endpoint: "monitoring.googleapis.com:443"
    protocol: grpc
  filter:
    events:
      exclude:
        - "agent.user.prompt"

See the Orchestrator Settings Reference for the full field reference, and Metrics & OpenTelemetry for how settings merge across scopes.

Via Hub Environment Variables

For hosted deployments, environment variables can be set at the Project or Broker level on the Hub. These are automatically injected into every agent container.

SCION_OTEL_ENDPOINT="monitoring.googleapis.com:443"
SCION_GCP_PROJECT_ID="your-project-id"
SCION_TELEMETRY_ENABLED="true"

Harness-Specific Configuration

Each harness integrates with sciontool’s telemetry pipeline differently depending on its native telemetry support:

• OpenTelemetry-native harnesses (e.g. opencode): These harnesses emit OTLP data directly. Point them at the sciontool forwarder on localhost:

  • gRPC (Default): OTEL_EXPORTER_OTLP_ENDPOINT="localhost:4317"
  • HTTP: OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"

• Claude Code (logs-first route): Claude Code emits structured OTLP log records (scope com.anthropic.claude_code.events) rather than spans or metrics. The sciontool receiver normalizes Claude’s event names to the canonical agent.* namespace (e.g. user_prompt → agent.user.prompt) and unconditionally redacts all Claude log bodies to protect prompt privacy, regardless of the configured filter policy. When Claude is the active harness and telemetry is enabled, the broker propagates the telemetry backend configuration (gcp or otlp) via the SCION_TELEMETRY_CLOUD_PROVIDER environment variable to sciontool to prevent accidental metrics export before credentials are available.

• Gemini CLI: The harness provisioner points Gemini CLI’s native OTel (logs and metrics) at the local sciontool receiver over gRPC on 4317, so it passes through redaction and identity stamping. Its usage (gen_ai.api.calls, scion.usage.tokens) is derived from the native gemini_cli.api_response and gemini_cli.api_error log events — see the usage source description under Automated Metrics Collection. Session metrics still come from session-file parsing — see Session Metrics (Gemini).

• Copilot and Grok Build: The harness provisioner always points the native OTel exporter at the local sciontool receiver (Copilot over HTTP on 4318, Grok Build over gRPC on 4317), never directly at SCION_OTEL_ENDPOINT, so their telemetry passes through redaction and identity stamping. SCION_COPILOT_OTEL_ENDPOINT and SCION_GROK_BUILD_OTEL_ENDPOINT override the endpoint for local debugging only; they bypass redaction entirely. See Supported Harnesses.

Reserved telemetry variables

When a harness provisioner configures native telemetry, the variables that control it are reserved: CLAUDE_CODE_ENABLE_TELEMETRY, GEMINI_TELEMETRY_*, COPILOT_OTEL_*, GROK_TELEMETRY_*, GROK_EXTERNAL_OTEL, and OTEL_*. If the runtime environment (for example, Project or Broker env on the Hub) sets any of them to a value other than the one the provisioner generated, the agent fails to start with a native telemetry policy conflict error naming the key.

These harness-specific env vars are injected at agent start time via the harness config’s env map and are separate from the Scion telemetry settings. Scion injects SCION_HARNESS into every agent container, and SCION_MODEL when the agent’s configuration names a model, to enable harness-aware telemetry attribution (see Native Metrics Pipeline for how a missing model is labelled).

Agent Logs

Agent logs are written to /home/scion/agent.log inside the container. The sciontool logging system writes to both stderr and this file.

Each agent.log line starts with a UTC RFC 3339 timestamp, followed by [sciontool] [LEVEL], an optional [TAG], and the message, for example 2026-10-02T00:30:15.123456789Z [sciontool] [INFO] [LIMITS_EXCEEDED] Agent stopped: .... Per-service <name>.lifecycle.log lines use the same timestamp in brackets: [2026-10-02T00:30:15.123456789Z] message. Timestamps are always UTC, whatever TZ the agent runs with; the agent process itself keeps its own TZ.

Cloud Log Viewer & Hub API

Scion provides a built-in Cloud Log Viewer in the Web UI to stream agent logs in real-time. This is backed by the Hub API, which retrieves logs directly from the active runtime broker or from the persisted agent.log file, ensuring comprehensive visibility into agent execution regardless of its current state.

Log Ownership and Permissions

The sciontool utility ensures that agent.log is owned by the scion user during initialization, even if sciontool is initially run as root. The log file is created with permissive 0666 permissions to ensure multiple processes can contribute to the log stream.

Log Levels

Scion uses four log levels: DEBUG (detailed diagnostics), INFO (normal operational events, the default), WARN (something an operator should notice) and ERROR (failures). The Hub and Broker server log, the CLI’s [DEBUG] lines on stderr, and sciontool’s agent.log all read their level from one shared setting, described in Controlling the Log Level.

Controlling the Log Level

SCION_LOG_LEVEL

SCION_LOG_LEVEL sets the level for the server, the CLI and sciontool. The value is a comma-separated list:

• The first entry, if it has no =, is the default level: debug, info, warn (or warning) or error. Names are case-insensitive. If you leave it out, the default level is info.
• Each further entry is a per-component level: <component>=<level>.

# Everything at DEBUG
SCION_LOG_LEVEL=debug scion server start --enable-hub

# Only warnings and errors
SCION_LOG_LEVEL=warn scion server start --enable-hub

# INFO overall, DEBUG for auth decisions
SCION_LOG_LEVEL=info,hub.auth=debug scion server start --enable-hub

# CLI: show only the [hubsync] debug lines of a CLI command
SCION_LOG_LEVEL=info,hubsync=debug scion list

Component names are the server’s subsystem names, such as hub.auth or broker.heartbeat, and the tags on the CLI’s [DEBUG] lines, such as hubsync. Matching is hierarchical on dots: hub=debug applies to hub.auth and hub.maintenance.pull-images unless a longer key such as hub.auth=info is also given. Component keys that match nothing are accepted and have no effect.

Server output written through Go’s standard log package (for example lines that start with Warning:) is logged at INFO level, so SCION_LOG_LEVEL=warn or error hides it.

If a level name is not valid, Scion prints a warning to stderr once and falls back: an invalid default level becomes info, and an invalid component entry is ignored.

Where each part of Scion applies the level:

Part	Where it applies
Hub and Broker server log	The default level and per-component levels filter the main server log on every sink: stdout, OpenTelemetry and direct Cloud Logging. OpenTelemetry export used to receive DEBUG records even at the default level; it now follows SCION_LOG_LEVEL like the other sinks. The HTTP request log and the message log are an access and audit trail: SCION_LOG_LEVEL can only make them more verbose (debug), so warn or error never drops their INFO entries.
CLI	[DEBUG] lines on stderr appear when the default level is debug. Tagged lines such as [hubsync] also appear when that tag’s component level is debug.
sciontool (agent.log)	The default level filters DEBUG, INFO and WARN lines. A tag on a line, or a subsystem attribute on a structured log record, is matched against the per-component levels.

The log level only changes logging. It does not enable any HTTP routes; diagnostic endpoints are controlled separately by scion server start --enable-debug-endpoints.

Precedence

When more than one source sets a level, the one with the highest precedence wins:

1. Flag: scion server start --debug sets the server’s default level to debug and keeps any per-component levels from the environment. scion --debug turns on all of the CLI’s [DEBUG] output for that command only; it does not change the environment of agents the command starts. sciontool --log-level <spec> takes the same syntax as SCION_LOG_LEVEL and replaces it completely.
2. Environment variable: SCION_LOG_LEVEL, or the deprecated SCION_DEBUG (see below).
3. Setting: the server.log_level setting (or SCION_SERVER_LOGLEVEL) has the lowest precedence of the explicit sources. The server applies it at startup and again on a file-mode settings reload or admin server-config save, and the change reaches every sink, including direct Cloud Logging. Clearing it reverts to the default. At startup the server logs one Log level resolved line with the level and the source that set it (default when the setting holds the default value). The line is logged at INFO, or at WARN when the level is warn, so it is hidden only at error. Its values are debug, info, warn and error; see the server configuration reference.
4. Default: info.

If both SCION_LOG_LEVEL and SCION_DEBUG are set, SCION_LOG_LEVEL wins.

SCION_DEBUG (deprecated)

SCION_DEBUG is deprecated and is planned for removal one release after its deprecation. It still works: any non-empty value means the same as SCION_LOG_LEVEL=debug, for the server, the CLI and sciontool. The one exception is scion commands run inside an agent, which ignore it (see Debugging an agent). When it is used, each process prints this warning to stderr once:

Warning: SCION_DEBUG is deprecated and will be removed in a future release; use SCION_LOG_LEVEL=debug instead.

Because SCION_DEBUG is now an alias, a Hub or Broker started with SCION_DEBUG set in its environment logs at DEBUG level, which it did not do before. The warning is not written to agent.log. sciontool hook and sciontool status run as short-lived subprocesses whose stderr their caller captures, so they never print it.

Switch scripts and environment files from SCION_DEBUG=1 to SCION_LOG_LEVEL=debug.

Debugging an agent

Agents do not inherit debug settings. A Hub or Broker running with --debug (or with a debug level in its own environment) and scion --debug on the command line no longer set SCION_DEBUG or SCION_LOG_LEVEL in the agents they start. Earlier releases set SCION_DEBUG=1 in every agent in those cases, which made scion commands inside agents print debug output. Agents started by an older Hub or Broker may still have SCION_DEBUG=1 in their environment, so scion commands inside an agent (agent CLI mode) ignore SCION_DEBUG and print no deprecation warning for it. They still honour SCION_LOG_LEVEL and --debug. sciontool in the agent still treats SCION_DEBUG as the deprecated alias.

To debug an agent, set SCION_LOG_LEVEL for it explicitly. The value uses the same syntax as above, and scion commands and sciontool inside the agent apply it.

• One agent, at start: pass --agent-log-level to scion start or scion resume. It sets SCION_LOG_LEVEL in that agent’s environment. An invalid value is rejected before the agent starts.

  scion start my-agent --agent-log-level debug
  scion start my-agent --agent-log-level info,hubsync=debug

  The value is saved in the agent’s applied configuration and kept on later restarts and resumes. To remove or change it, delete and re-create the agent. When the Hub reuses an agent that already exists (for example, resuming a suspended agent), it keeps the stored configuration and does not apply --agent-log-level; both scion start and scion resume print a warning in that case.

• Many agents, through the Hub: store SCION_LOG_LEVEL as a Hub environment variable at the scope you want. Use --always so it is injected without a template requesting it.

  # Every agent in the current project
  scion hub env set --project --always SCION_LOG_LEVEL=debug
  
  # Every agent you start
  scion hub env set --always SCION_LOG_LEVEL=debug
  
  # Turn it off again
  scion hub env clear --project SCION_LOG_LEVEL

• In agent configuration: add SCION_LOG_LEVEL to the env of an inline --config file (a template cannot set SCION_* keys). A value set this way overrides the Hub scopes; --agent-log-level overrides it in turn.

Telemetry Collection

The telemetry pipeline in sciontool collects and forwards OpenTelemetry (OTLP) data from agents. See the Metrics & OpenTelemetry guide for deep configuration details.

Telemetry Export Resilience

Trace and log batches make at most four pipeline export attempts, including the first, with exponential backoff while the caller deadline permits. Metrics make one pipeline export call per eligible flush. Calls for the same metric series are spaced at least 15 seconds after the preceding call completes. An unresolved metric snapshot stops after 20 pipeline calls or five minutes from its oldest admission, whichever comes first. An underlying SDK may make additional network calls within one pipeline call; the caller context bounds cooperative work.

An OTLP intake success means the agent’s local collector accepted the records. It does not confirm remote delivery. The collector retains at most 16 MiB of encoded payload, 4096 records or metric points, and 512 nonempty request entries across all signals, including pending, in-flight, and retry work. On capacity exhaustion it rejects the new request with HTTP 429 or gRPC ResourceExhausted and retry advice; it does not evict healthy admitted work. A permanent oversize request receives HTTP 413 or gRPC ResourceExhausted without retry advice. The collector has no disk-backed queue, so a process crash can lose admitted data.

Local delivery diagnostics use span, log-record, or admitted metric-point units. Delivered means a whole exporter batch returned success. Unconfirmed means admitted data reached a terminal permanent, partial, attempt-limit, age-limit, or shutdown outcome without full delivery confirmation; it is not proof that the backend lost those records. Dropped covers proven local discard. BackendRejected records the known rejected subset in an OTLP partial-success response. At a quiet point, Accepted = Delivered + Dropped + Unconfirmed + retained. Attempts counts exporter calls, and Failed counts failed batch outcomes. SDKErrors separately counts asynchronous Cloud Logging callbacks; one failed batch can produce both a Failed outcome and an SDKErrors callback without representing two failed batches. A later cumulative metric value may include an earlier unconfirmed baseline, but it cannot retroactively confirm that earlier admission.

The collector writes bounded delivery snapshots to local stderr and agent.log at startup, on the first export failure, at most once per minute during continued operation, and on shutdown. These remain visible while the telemetry destination is unavailable and are not sent through that destination. A final degraded snapshot with Unconfirmed records signals uncertain delivery after a failed drain.

The server admits at most 16 active HTTP and gRPC processing requests together. A gRPC client can queue a 17th call locally on the same connection before the server receives it; use a caller deadline to bound that wait. The server’s 15-second intake deadline starts only after it receives a stream. These are local collector limits, not end-to-end delivery or process-memory guarantees.

What’s Collected

Data Type	Source	Description
Traces	Agent OTLP	Span data for tool calls, API requests
Metrics	sciontool	Counters and histograms for tokens, tools, and latency
Correlated Logs	sciontool	Log records linked to traces for every hook event
Native Logs	Claude Code OTLP	Structured log records (bodies redacted); event names normalized to agent.* namespace
Hook Events	Harness hooks	Tool calls, prompts, model invocations converted to spans
Session Metrics	Gemini session files	Token counts, turn counts, tool statistics

Privacy Controls

By default, user prompts (agent.user.prompt) are excluded from telemetry to protect privacy. Additionally, sensitive attributes are automatically redacted or hashed.

• Redacted: prompt, user.email, tool_output, tool_input
• Hashed: session_id
• Claude log bodies: Unconditionally redacted ([REDACTED]) regardless of filter configuration. Claude Code emits free-form text in its OTLP log bodies; the receiver enforces redaction even if a custom policy omits log.body from its redaction list.

HTTP Request Logs

HTTP requests to Hub, Broker, and Web servers are logged as a dedicated structured stream, separate from application logs. Request logs use the google.logging.type.HttpRequest format and include project/agent IDs, a generated request ID, and trace context from incoming headers.

Enabling Request Log Output

Method	Configuration
File	Set SCION_SERVER_REQUEST_LOG_PATH=/path/to/requests.log
Cloud Logging	Automatic when SCION_CLOUD_LOGGING=true — uses log name scion_request_log
Stdout	Default when running in background mode (suppressed in --foreground mode)

Trace Context Propagation

The middleware generates a UUID request_id for every request and captures trace headers (X-Cloud-Trace-Context, traceparent, X-Trace-ID). These IDs are automatically attached to all application logs emitted during the request via logging.Logger(ctx), enabling end-to-end correlation between the request log entry and any downstream application log entries.

Cloud Logging Queries

Request logs appear under a separate log name from application logs:

-- All HTTP request logs
logName="projects/YOUR_PROJECT/logs/scion_request_log"

-- Slow requests (latency > 1s)
logName="projects/YOUR_PROJECT/logs/scion_request_log"
httpRequest.latency > "1s"

-- Failed requests for a specific project
logName="projects/YOUR_PROJECT/logs/scion_request_log"
httpRequest.status >= 400
labels.project_id = "my-project"

-- Correlate a request with its application logs
logName="projects/YOUR_PROJECT/logs/scion-server" OR logName="projects/YOUR_PROJECT/logs/scion_request_log"
jsonPayload.request_id = "YOUR_REQUEST_ID"

See the Local Development Logging guide for the full field reference and file output format.

Artifact Decision Log

The Hub writes one structured entry per artifact decision (publish, versions, grants, share links, moves, expiry, deletion, owner liveness marks and authorization refusals) to a dedicated log, scion-artifacts. With SCION_CLOUD_LOGGING=true it is a separate Cloud Logging log. Otherwise it goes to stdout. Entries carry the artifact id (from the parsed route, never the raw path), the outcome, a fixed reason code for refusals and failures, and, for decisions made on a request, the route template and the request_id of the matching request log line:

-- Every decision about one artifact
logName="projects/YOUR_PROJECT/logs/scion-artifacts"
labels.artifact_id = "ARTIFACT_ID"

-- Refusals and the request they answered
logName="projects/YOUR_PROJECT/logs/scion-artifacts" AND labels.outcome = "refused"

Reads answered with 404 are logged at DEBUG only. Share-link tokens, titles, notes, file contents and credentials are never logged. See Artifacts: Decision log for the actions, fields and reason codes.

Querying Logs by Subsystem

Hub and Broker logs include a subsystem attribute that identifies the internal subsystem that produced each log entry. This is separate from the top-level component field (which reflects the server mode: scion-hub, scion-broker, or scion-server) and provides finer-grained filtering.

Available Subsystems

Subsystem	Description
hub.agent-lifecycle	Agent create, start, stop, delete, and state transitions
hub.auth	Authentication and authorization decisions
hub.control-channel	WebSocket lifecycle for broker connections
hub.messages	Message routing from scion message to brokers
hub.notifications	Event-driven notification dispatch and subscription matching
hub.scheduler	Background recurring and one-shot scheduled tasks
hub.env-secrets	Environment variable and secret management
hub.templates	Template CRUD, hydration, and bootstrap
hub.workspace	Git worktree sync operations
hub.dispatcher	HTTP agent dispatch to brokers
hub.web	Web server requests and sessions. At debug, every web request is logged; requests that return an error status or are aborted are logged at info and more verbose levels.
broker.agent-lifecycle	Container provisioning, environment resolution, template hydration
broker.control-channel	Broker-side WebSocket connection to the hub
broker.messages	Message injection into agent tmux sessions
broker.heartbeat	Periodic broker status reports to hub
broker.env-secrets	Broker-side environment gathering and finalization

With the complete roll-out of Tier 3 subsystem logging, dotted logging tags now systematically cover all CRUD handlers across these subsystems, providing comprehensive filtering and tracing for all resource-management actions.

In combo server mode (scion-server), both hub.* and broker.* subsystem logs appear in the same stream. The dotted prefix distinguishes them without requiring separate processes.

Cloud Logging Query Examples

All examples assume your logs are in the scion-server log name. Adjust the logName filter to match your configuration.

Filter by Server Component

-- All hub logs (hub-only or combo mode)
logName="projects/YOUR_PROJECT/logs/scion-server"
labels.component="scion-hub"

-- All logs from combo server mode
logName="projects/YOUR_PROJECT/logs/scion-server"
labels.component="scion-server"

Filter by Subsystem

-- All hub subsystem logs
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "^hub\."

-- All broker subsystem logs
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "^broker\."

-- A specific subsystem
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem = "hub.notifications"

Agent Lifecycle Debugging

-- All agent lifecycle events across hub and broker
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.agent-lifecycle$"

-- Agent lifecycle for a specific agent
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.agent-lifecycle$"
jsonPayload.agent_id = "my-agent-id"

-- Only errors in agent lifecycle
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.agent-lifecycle$"
severity >= ERROR

Message Tracing

-- All message-related logs (hub routing + broker injection)
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.messages$"

-- Messages from a specific sender
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.messages$"
jsonPayload.sender = "agent-slug"

-- Messages to a specific recipient in a project
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.messages$"
jsonPayload.recipient = "target-agent"
jsonPayload.project_id = "my-project-id"

Agent Telemetry Logs

Logs that agents export through the sciontool telemetry pipeline go to the scion-agents log name, not scion-server. The exporter turns each OpenTelemetry resource attribute into a log label with the same key. Agent identity is carried only in these resource-attribute labels: there are no top-level agent_id or project_id labels on agent telemetry log entries. The only extra label is hub, which is set from SCION_HUB_NAME when present.

The identity labels are stamped by the sciontool receiver from the agent’s authoritative identity (see Identity Enforcement), so filter on these:

• labels."scion.project.id" for the agent’s project.
• labels."scion.agent.id" for the agent ID.
• labels."scion.agent.slug" for the agent slug.

-- Agent telemetry logs for a project
logName="projects/YOUR_PROJECT/logs/scion-agents"
labels."scion.project.id" = "my-project-id"

-- Agent telemetry logs for one agent, by slug
logName="projects/YOUR_PROJECT/logs/scion-agents"
labels."scion.agent.slug" = "my-agent-slug"

-- Agent telemetry logs for one agent, by ID
logName="projects/YOUR_PROJECT/logs/scion-agents"
labels."scion.agent.id" = "my-agent-id"

Auth and Security Auditing

-- All authentication and authorization events
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem = "hub.auth"

-- Auth failures only
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem = "hub.auth"
severity >= WARNING

Control Channel Monitoring

-- All control channel activity (hub + broker sides)
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.control-channel$"

-- Control channel errors (connectivity issues)
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "\.control-channel$"
severity >= ERROR

Operational Noise Reduction

-- All hub logs EXCEPT heartbeat noise
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "^hub\."
jsonPayload.subsystem != "broker.heartbeat"

-- Only high-priority subsystems (notifications, auth, agent lifecycle)
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem = "hub.notifications" OR
jsonPayload.subsystem = "hub.auth" OR
jsonPayload.subsystem =~ "\.agent-lifecycle$"

Combining with Time and Severity

-- Errors across all subsystems in the last hour
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem != ""
severity >= ERROR
timestamp >= "2026-03-03T00:00:00Z"

-- Debug-level broker logs for troubleshooting
logName="projects/YOUR_PROJECT/logs/scion-server"
jsonPayload.subsystem =~ "^broker\."
severity = DEBUG

Tip

Create saved queries in the Cloud Logging console for subsystem filters you use frequently. For alerting, use log-based metrics with a filter like jsonPayload.subsystem = "hub.notifications" AND severity >= ERROR to trigger alerts on notification dispatch failures.

Structured Messaging Pipeline

Scion includes a comprehensive structured messaging pipeline that provides reliable delivery of messages to and from agents. This pipeline is fully observable:

• Hub API Integration: Messages can be sent and retrieved via the new Hub API, allowing external systems to programmatically interact with agents.
• Web UI “Messages” Tab: An interactive interface in the dashboard allows administrators and users to trace message flows in real-time.
• Multi-Stage Broker Adapter: Ensures robust delivery of messages to the agent containers, including external notifications. You can monitor message flow health in logs using the hub.messages and broker.messages subsystems.

Stalled and Offline Agent Detection

The Hub runs two separate checks on running agents, each every 5 minutes. They set different activity values and have different causes.

• Offline (no heartbeat): An agent whose last heartbeat is more than 2 minutes old is set to the offline activity. The process, or its connection to the Hub, is gone. One cause is an agent that cannot refresh its auth token, which stops its heartbeat and other updates. Stop and restart such an agent to give it a new token; an agent that keeps its connection to the Hub refreshes its token on its own. Agents whose activity is blocked, completed or limits_exceeded are not marked offline.
• Stalled (heartbeat but no activity): A running agent whose heartbeat is still arriving (within the last 2 minutes) but that has sent no activity events within server.hub.stalled_threshold (default 5m) is set to the stalled activity. The process is alive but not making progress. Agents whose activity is blocked, waiting_for_input, completed, limits_exceeded or offline are excluded. With server.hub.auto_suspend_stalled on (default off), the Hub also suspends it; see Auto-Suspend of Stalled Agents.
• Notifications: A notification subscription can include the Stalled and Error triggers. The subscription set up during agent creation with notifications enabled includes both, along with Completed, Waiting on Parent and Limits Exceeded. scion start sets one up unless you pass --no-notify, and so does the web UI’s create form when Notify me on important agent state changes is checked. A subscription you add later, from the web UI or with scion notifications subscribe, starts with only Completed, Waiting on Parent and Limits Exceeded.
• Visibility: The Web UI shows offline and stalled agents with their own status badges, so they are not lost among active workloads.

Server Maintenance Logs

Server maintenance operations (like rebuild-server, rebuild-web, and pull-images) emit structured execution logs that are captured in real-time. These logs are accessible via the Web Dashboard’s Maintenance Panel, providing immediate visibility into the progress of updates and synchronization tasks. In addition to real-time streaming, historical logs for completed maintenance tasks are archived and associated with their respective run records, allowing administrators to review past operations for duration, completion status, and potential failure points.

Troubleshooting for Admins

Logs Not Appearing in GCP

1. Verify Endpoints: Ensure SCION_OTEL_ENDPOINT is set to monitoring.googleapis.com:443.
2. Check Credentials: Outside Kubernetes, an agent with no GCP identity configured runs in metadata mode block, where ADC gets no token. Such agents need either a service account assigned with --service-account (echo $SCION_METADATA_MODE inside the agent prints assign) or a GCP service account key file. For the key file, verify the scion-telemetry-gcp-credentials secret is registered with target ~/.scion/telemetry-gcp-credentials.json. Inside the agent, check echo $SCION_OTEL_GCP_CREDENTIALS — it should point to the file. See GCP Credentials for Agent Containers for setup.
3. Check Permissions: Verify the Workload Identity or Service Account has roles/logging.logWriter.
4. Inspect Agent Init: View the agent container logs (stderr) to see if sciontool reported a telemetry startup failure:

   [sciontool] ERROR: Failed to start telemetry: connection refused

5. Network Policy: If running in Kubernetes, ensure Egress is allowed to GCP APIs.

Missing Trace Correlation

If you see logs but they aren’t linked to traces in the Cloud Trace waterfall:

1. Ensure the agent is using the sciontool gRPC port (4317).
2. Verify SCION_OTEL_LOG_ENABLED=true is set on the system components.

Common Hub and Broker API Errors

1. no_runtime_broker (422 Unprocessable Entity)

• Symptom: Dispatched agents or CLI commands like scion start fail immediately with a no_runtime_broker error code.
• Observable: The Hub server logs contain ErrCodeNoRuntimeBroker or no_runtime_broker error messages.
• Root Cause: The Runtime Broker is disconnected, locked up, or saturated (failing to send heartbeats within the expected interval). High load or redundant synchronous workspace sweeps can block heartbeats.
• Operator Action:
  1. Check broker daemon health and restart the broker service if unresponsive.
  2. Wait 30–60 seconds for heartbeats to resume after a restart.
  3. Reduce concurrent agent start operations under high load.

2. Agent Token Verification Failures

• Symptom: Agents fail to start or refresh tokens with errors like failed to verify token: error in cryptographic primitive or repeat Token refresh failed … 401 in logs.
• Observable: Loop between user login and session_expired states.
• Root Cause: The Hub regenerated its JWT/session signing keys on restart, or there is a multi-replica key mismatch (replicas using different signing keys).
• Operator Action:
  1. Pin the session/JWT signing key securely in the deployment configuration. Ensure SCION_SERVER_SESSION_SECRET (or SharedSigningSecret) is set consistently across all Hub replicas and survives restarts.
  2. Have users run scion message <agent> "continue" to trigger resume, or perform a manual login refresh.

3. Duplicate sciontool Processes (Split-Brain Agent)

• Symptom: An agent gets stuck in the starting phase while its activity says completed, or has port conflicts.
• Observable: Agent logs show sciontool init starting as PID <different-pid> and Failed to start telemetry: ... address already in use or similar port conflicts.
• Root Cause: A duplicate sciontool process was spawned. This typically happens when test runners (like go test) or child processes inherit Hub/agent environment variables and invoke a second sciontool init.
• Operator Action:
  1. Inspect running processes inside the agent container and terminate the duplicate PID.
  2. Recreate the agent to clean up the telemetry socket and state files.

Conversation Model Migration: Messaging Divergence Diagnostics Board

During the migration to the new conversation model (Tranche G), Scion Hub runs active dual-routing and dual-read validation. To monitor this process in real-time, administrators can query a read-only diagnostics board endpoint:

GET /api/v1/admin/messaging/divergence

Endpoint Authorization

Access is strictly permission-gated. Callers must possess the hub.diagnostics.read permission (which is included in the hub-admin curated role set) to query the board.

Response Schema

The response returns process uptime and real-time divergence counters, along with critical machine-readable caveats explaining the exact context of the values:

{
  "hub_id": "replica-node-1",
  "process_start_time": "2026-08-30T00:00:00Z",
  "process_uptime": "12h30m5s",
  "matches": 14205,
  "mismatches": 12,
  "comparisons": 14217,
  "fallbacks": 5,
  "caveats": {
    "scope": "per_replica_since_boot",
    "scope_detail": "These counters live in process memory and reset when the replica restarts. They reflect only this replica's traffic, identified by hub_id.",
    "mismatch_composition": "The mismatches count conflates two unrelated signals: routing-key disagreement (ComputeDivergenceMatch) and prior-message conversation_id inconsistency (CheckConversationConsistency). This board cannot separate them.",
    "consistency_check_fails_open": "CheckConversationConsistency returns true (no mismatch increment) on ListMessages query errors, on empty resolved conversation IDs, and when insufficient data prevents lookup. A low mismatch count does not imply agreement — it is equally consistent with agreement, query errors, or insufficient lookup data.",
    "not_go_no_go": "This board is NOT the Tranche G go/no-go input. The offline recomputation report is the artifact that answers the go/no-go question.",
    "counter_snapshot": "All counters are read independently via separate atomic loads and may not correspond to a single instant. comparisons is computed as matches + mismatches from those independent reads, so the triple is arithmetically consistent but not a true snapshot. Ratios derived from these values (e.g. mismatch rate, fallback percentage) are approximate."
  }
}

Metrics Definitions

• matches: Number of times dual-routing matches (routing-key agreement).
• mismatches: Number of times disagreement is detected (routing-key disagreement or prior conversation consistency mismatches).
• comparisons: Arithmetic sum of matches + mismatches (internally consistent but approximate).
• fallbacks: Number of read-path fallback attempts to old routing configurations during the switch.

Related Guides

• Metrics & OpenTelemetry - Detailed telemetry configuration
• Hub Monitoring Dashboard - Cloud Monitoring dashboard for Hub metrics
• Hub Server - Hub integration for hosted mode
• Runtime Broker - Broker setup and configuration