Project Aegis: Enterprise streaming and agentic operations platform¶

Customer Engineering (CE) go-to-market walkthrough and interactive demo guide¶

• Target Audience: Chief Technology Officers (CTOs), Vice Presidents (VPs) of Data Infrastructure, Enterprise Architects, Chief Security Officers (CSOs), and Artificial Intelligence (AI) and Operations Leads
• Duration: 10 Minutes (Interactive 4-Stage Presentation and Walkthrough)
• Key Google Cloud Products: Google Cloud Managed Apache Kafka, Google Cloud Pub/Sub, Dataproc cluster aegis-spark-cluster on CLUSTER_TIER_STANDARD with vectorized Spark execution and optional CLUSTER_TIER_PREMIUM C++ Lightning Engine (Velox and Gluten), Cloud Dataflow with Apache Beam Streaming Engine, BigQuery Continuous Queries, Cloud Bigtable, BigQuery Agent Analytics, Cloud Run microservices, Gemini Enterprise Agent Platform (GEAP) with Gemini 2.5 Flash, Model Armor security guardrails, and Google Cloud Dataplex Knowledge Graph with OpenLineage integration.

Multi-stack reference architecture (3 deployable stacks)¶

Project Aegis supports three modular streaming technology stacks that share a common dual-sink persistence layer using Cloud Bigtable and BigQuery, Model Armor security guardrails, and the Gemini 2.5 Flash reasoning engine on GEAP.

The Heads-Up Display (HUD) header displays a persistent Active Stack indicator badge on every page showing the active architecture name (ACTIVE STACK: <STACK_NAME>), the target Google Cloud project (Project: <project_id>), and the 4-stage pipeline topology chips:

Executive command HUD tour (5 interactive modules)¶

The Project Aegis Command HUD organizes operations into 5 sequential modules accessible from the top segmented navigation bar:

Module 1: Executive deck (/slides)¶

Module 1 embeds the 16:9 executive presentation deck with interactive playback, slide thumbnails, and fullscreen presentation controls.

Module 2: Stream simulator (/simulator)¶

Module 2 provides real-time telemetry rate selectors, a 5-minute throughput readout, and lifecycle controls for the active streaming compute pipeline.

Module 3: End-to-end architecture and demo guide (/guide)¶

Module 3 presents the 7-stage architectural blueprint alongside executive KPI cards and direct deep links to live Google Cloud Console resources.

Module 4: Live operations and cognitive co-pilot (/grid)¶

Module 4 pairs the 15-asset Cloud Bigtable operational grid with the Gemini 2.5 Flash Co-Pilot panel for root cause diagnosis and Human-in-the-Loop (HITL) mitigation approval.

Module 5: BigQuery batch analytics and financial provenance (/analytics)¶

Module 5 delivers an interactive BigQuery GoogleSQL workspace for auditing fleet stress aggregations, anomaly spikes, and AI tokenomics ROI.

Quick start and single-command deployment¶

CEs can provision any of the three Aegis infrastructure stacks, container repositories, streaming engines, and Cloud Run microservices with Terraform.

1. Provision infrastructure with Terraform¶

# Navigate to the desired stack directory, such as oss, first-party, or low-code
cd terraform/stacks/oss

# Initialize provider and state
terraform init

# Apply infrastructure setup
terraform apply -auto-approve

2. Verify output endpoints and stack health¶

Upon deployment completion, Terraform outputs all live service endpoints and resource identifiers:

Outputs:

agent_service_url       = "projects/YOUR_PROJECT_ID/locations/us-central1/reasoningEngines/YOUR_AGENT_ID"
artifact_registry_repo  = "aegis-containers"
bigquery_dataset_id     = "analytics"
bigtable_instance_id    = "aegis-bigtable"
dataproc_cluster_name   = "aegis-spark-cluster"
dataproc_deps_bucket    = "YOUR_PROJECT_ID-dataproc-deps"
geap_agent_id           = "YOUR_AGENT_ID"
hud_backend_url         = "https://hud-backend-xxxx-uc.a.run.app"
hud_frontend_url        = "https://hud-frontend-xxxx-uc.a.run.app"
kafka_cluster_id        = "aegis-kafka-cluster"
kafka_topic_id          = "telemetry-raw"
project_id              = "YOUR_PROJECT_ID"
region                  = "us-central1"
service_account_email   = "aegis-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com"
telemetry_simulator_url = "https://telemetry-simulator-xxxx-uc.a.run.app"

Run the unified multi-stack verification Command-Line Interface (CLI) from the project root to verify Cloud Run services, streaming pipelines, Cloud Bigtable freshness, and BigQuery tables:

# Verify all deployed stacks, or pass --stack oss, first-party, or low-code
python3 _agents/scripts/verify_all.py --stack all

3. Launch authenticated HUD browser proxy¶

To open the interactive HUD frontend in your browser with Google Cloud Identity and Access Management (IAM) credential proxying, run the helper script from the root directory:

./RUN_PROXY.sh

End-to-end architecture overview¶

flowchart TD
    subgraph Ingestion ["1. Streaming Ingestion Plane"]
        A[Telemetry Simulator Microservice] -->|Streaming JSON Events @ 100 msgs/s| B[Managed Kafka or Cloud Pub/Sub: telemetry-raw]
        B --> C[Stream Processor: Dataproc Spark / Cloud Dataflow / BigQuery Continuous SQL]
    end

    subgraph Compute ["2. Windowed Stream Processing Engine"]
        C -->|10-Second Tumbling Window Aggregation| D[Windowed Metric Aggregator]
        D -->|Sub-millisecond State Writes| E[(Cloud Bigtable: aegis-bigtable)]
        D -->|Streaming Analytics Sink| F[(BigQuery: analytics.telemetry_events)]
        C -.->|DATAPROC_LINEAGE_ENABLED| G[Dataplex / OpenLineage Knowledge Graph]
    end

    subgraph Agentic ["3. GEAP Agentic Operations and Security"]
        E -->|SSE Stream / Anomaly Alert| H[HUD Backend FastAPI Service]
        H -->|Telemetry Payload| I[Model Armor Security Shield]
        I -->|Sanitized Prompt| J[GEAP: Gemini 2.5 Flash Agent]
        J -->|Chain-of-Thought RCA and Mitigation Plan| K[Operator HITL Review Panel]
        K -->|Approve and Execute| L[Industrial Actuator Tool: throttle_and_cool]
        L -->|Reset Physical Asset State| A
        J -->|Token Spend and ROI Logging| M[(BigQuery: analytics.rca_events)]
        J -->|Metrics and Logs| N[Cloud Monitoring and Cloud Logging]
    end

    subgraph Experience ["4. Executive Command HUD"]
        H -->|Server-Sent Events| O[HUD Next.js Frontend Dashboard]
        O -->|Interactive Anomaly and Control| H
    end

Stage 1: Real-time ingestion and vectorized processing (2.5 minutes)¶

Stage objective¶

Demonstrate how Google Cloud ingests high-throughput Industrial Internet of Things (IIoT) telemetry with low latency using Google Cloud Managed Apache Kafka or Cloud Pub/Sub and processes 10-second tumbling windows using a warm Dataproc cluster (aegis-spark-cluster) on CLUSTER_TIER_STANDARD with vectorized Spark execution (or optional CLUSTER_TIER_PREMIUM C++ Lightning Engine with Velox and Gluten vectorization), Cloud Dataflow, or BigQuery Continuous Queries, writing operational state to Cloud Bigtable in sub-milliseconds.

CE delivery script and talk track¶

Live demonstration commands (Stage 1)¶

1. Start the telemetry stream through the API or Module 2 UI¶

# Start synthetic telemetry stream across 15 assets at 100 msgs/sec
curl -X POST "https://hud-backend-xxxx-uc.a.run.app/api/start-stream" \
  -H "Content-Type: application/json" \
  -d '{"rate_msgs_per_sec": 100}'

2. Operate the streaming pipeline through the HUD or backend API¶

Terraform starts the streaming pipeline during terraform apply, and operators can also check status, stop, or restart the pipeline directly from Module 2: Stream Simulator (/simulator) or the backend Application Programming Interface (API):

# Check active streaming pipeline status
curl -X GET "https://hud-backend-xxxx-uc.a.run.app/api/pipeline/status"

# Start or restart the streaming pipeline
curl -X POST "https://hud-backend-xxxx-uc.a.run.app/api/pipeline/start" \
  -H "Content-Type: application/json"

3. Inspect live Cloud Bigtable operational state¶

# Read live row key metrics for Asset-04 from Cloud Bigtable
cbt -project YOUR_PROJECT_ID -instance aegis-bigtable lookup telemetry_metrics Asset-04

4. Inspect end-to-end millisecond latency metrics through the HUD backend API¶

Every asset payload returned by /api/telemetry/assets and /api/stream includes millisecond-resolution timestamps, specifically ingestion_timestamp_ms, db_insert_timestamp_ms, and pipeline_latency_ms:

# Query live asset states with millisecond ingestion and Bigtable insert timestamps
curl -s "https://hud-backend-xxxx-uc.a.run.app/api/telemetry/assets" | jq '.assets[0] | {asset_id, status, ingestion_timestamp_ms, db_insert_timestamp_ms, pipeline_latency_ms}'

Key technical takeaways for customer CTOs¶

1. Flexible Multi-Stack Ingestion and Compute: Choose between Managed Apache Kafka with Dataproc Spark (oss), Cloud Pub/Sub with Cloud Dataflow (first-party), or Cloud Pub/Sub Direct with BigQuery Continuous Queries (low-code).
2. Vectorized Spark Acceleration: Achieve high-throughput columnar execution using vectorized Spark reader properties on CLUSTER_TIER_STANDARD (2.2-debian12), with up to 300-400% performance improvements over standard Spark JVM execution when enabling optional CLUSTER_TIER_PREMIUM C++ Lightning Engine (Velox and Gluten) on allowlisted projects.
3. Sub-Millisecond Operational Storage: Cloud Bigtable handles continuous high-concurrency writes with single-digit millisecond latency and tracks end-to-end pipeline_latency_ms.

Stage 2: Chaos injection, Model Armor, and Gemini 2.5 Flash RCA (3 minutes)¶

Stage objective¶

Demonstrate how Aegis detects thermal and compute anomalies on industrial equipment in Module 4: Live Grid and AI Co-Pilot (/grid), passes alert payloads through Model Armor security shields to prevent prompt injection and Personally Identifiable Information (PII) leakage, and executes Gemini 2.5 Flash Chain-of-Thought Root Cause Analysis (RCA) in the sticky right-hand Co-Pilot panel.

CE delivery script and talk track¶

Live demonstration commands (Stage 2)¶

1. Inject a chaos anomaly through the API or HUD UI¶

# Triggers thermal/CPU anomaly on a random fleet asset
curl -X POST "https://hud-backend-xxxx-uc.a.run.app/api/simulator/inject-anomaly" \
  -H "Content-Type: application/json"

2. Trigger direct agent root cause analysis¶

# Test Agent Service Root Cause Analysis directly
curl -X POST "https://hud-backend-xxxx-uc.a.run.app/api/agent/mitigate" \
  -H "Content-Type: application/json" \
  -d '{
    "asset_id": "Asset-09",
    "cpu_utilization": 91.8,
    "temperature_c": 104.9,
    "pressure_psi": 115.0,
    "memory_utilization_pct": 85.0,
    "status": "CRITICAL",
    "additional_context": "Thermal sensor T-102 reporting rapid temp climb."
  }'

Key technical takeaways for CSOs and AI leads¶

1. Model Armor Defense-in-Depth: Protects LLM agents against adversarial prompt injections and sensitive data leaks before model execution.
2. Sub-Second Gemini 2.5 Flash Latency: Delivers multi-step Chain-of-Thought reasoning and structured remediation plans in under 800ms.
3. Side-by-Side Operational Ergonomics: Keeps the 15-asset Cloud Bigtable grid and the AI Agent Co-Pilot visible simultaneously without page scrolling.

Stage 3: Human-in-the-loop approval and autonomous closed-loop recovery (2 minutes)¶

Stage objective¶

Demonstrate HITL operational governance where the plant operator reviews the AI recommendations and the ROI Hero Strip metrics for AI Cost, Saved, and Net ROI, then clicks Approve and Execute Mitigation to trigger the agent's IndustrialActuatorTool, normalize the physical machine state, and watch the recovery flow back through the streaming pipeline into Cloud Bigtable.

CE delivery script and talk track¶

Live demonstration commands (Stage 3)¶

1. Approve and execute closed-loop mitigation through the API¶

# Execute Human-in-the-Loop approval and trigger IndustrialActuatorTool
curl -X POST "https://hud-backend-xxxx-uc.a.run.app/api/agent/approve" \
  -H "Content-Type: application/json" \
  -d '{
    "asset_id": "Asset-09",
    "approved_by": "Plant Lead Engineer",
    "incident_id": "INC-20260928-001"
  }'

Key technical takeaways for plant managers and operations leads¶

1. Safe Human-in-the-Loop Control: Balances autonomous AI speed with human authority over critical physical machinery.
2. Genuine Closed-Loop Verification: Neither the frontend nor the backend mutates Cloud Bigtable directly during remediation. The live streaming pipeline verifies recovery end to end.
3. Low Mean Time to Resolution (MTTR): Reduces MTTR from hours of manual troubleshooting to seconds.

Stage 4: Tokenomics, financial ROI, and data governance (2.5 minutes)¶

Stage objective¶

Demonstrate enterprise financial governance with BigQuery Agent Analytics token spend tracking, calculating Token ROI by comparing prevented downtime value to Gemini inference cost, and executing interactive GoogleSQL queries in Module 5: Batch Analytics (/analytics).

CE delivery script and talk track¶

Live demonstration commands (Stage 4)¶

1. Query BigQuery token spend and incident audit logs¶

-- Run in Module 5: Batch Analytics (/analytics) or BigQuery Console
SELECT
  event_id,
  asset_id,
  timestamp,
  tokens_used,
  cost_usd,
  status,
  root_cause
FROM
  `YOUR_PROJECT_ID.analytics.rca_events`
ORDER BY
  timestamp DESC
LIMIT 10;

2. Calculate fleet-wide prevented downtime ROI¶

SELECT
  COUNT(*) AS total_incidents_mitigated,
  SUM(tokens_used) AS total_tokens_consumed,
  ROUND(SUM(cost_usd), 6) AS total_gemini_cost_usd,
  ROUND(SUM(5000.0), 2) AS total_downtime_saved_usd,
  ROUND(SUM(5000.0) / NULLIF(SUM(cost_usd), 0), 0) AS roi_multiplier
FROM
  `YOUR_PROJECT_ID.analytics.rca_events`;

Key technical takeaways for Chief Financial Officers (CFOs) and data governance leads¶

1. Granular Cost Tracking: BigQuery records every LLM token and dollar fraction in analytics.rca_events for enterprise auditability.
2. Quantifiable ROI Proof: Proves the immediate economic return of autonomous AI mitigation over manual troubleshooting.
3. Automated Lineage Traceability: OpenLineage integration maps end-to-end data provenance into Google Cloud Dataplex Knowledge Graph.

Frequently asked questions and objection handling (CE playbook)¶

Q1: "Why use Cloud Bigtable instead of storing streaming telemetry directly in BigQuery?"¶

CE Answer: Bigtable provides sub-millisecond point reads and writes at high concurrency, making it ideal for live operational HUD displays, real-time alerting, and sub-second control loops. BigQuery serves as the analytical warehouse for partitioned ad-hoc SQL, multi-day trend analysis, and historical LLM tokenomics. Aegis uses both in a dual-sink architecture.

Q2: "How does vectorized Spark and the C++ Lightning Engine differ from standard PySpark?"¶

CE Answer: Standard PySpark executes inside JVM executors with object serialization and garbage collection pauses. Project Aegis enables vectorized columnar reader properties on CLUSTER_TIER_STANDARD (2.2-debian12) by default, and supports optional CLUSTER_TIER_PREMIUM C++ Lightning Engine (Velox and Gluten) on allowlisted projects to compile Spark execution plans into vectorized C++ code operating on columnar memory, delivering 2-4x speedups without JVM garbage collection overhead.

Q3: "How do I choose between the OSS, 1st-Party, and Low-Code stacks?"¶

CE Answer: Use the OSS stack (Managed Kafka + Dataproc Spark) when customers have existing Apache Kafka and PySpark investments they want to run on managed Google Cloud services. Use the 1st-Party stack (Pub/Sub + Cloud Dataflow) for serverless autoscaling Apache Beam pipelines with sub-second early window triggers. Use the Low-Code stack (Pub/Sub Direct + BigQuery Continuous Queries) when data teams prefer writing continuous streaming pipelines entirely in standard GoogleSQL.

Q4: "How does closed-loop actuation work safely with human-in-the-loop controls?"¶

CE Answer: Aegis enforces a strict two-tier actuation plane: the AI Agent performs cognitive diagnosis and generates structured remediation steps, while physical tool actuation (IndustrialActuatorTool.throttle_and_cool) requires explicit human approval through the authenticated HUD command center.

Summary checklist for demo success¶

• Run terraform apply in terraform/stacks/oss, terraform/stacks/first-party, or terraform/stacks/low-code to provision all Cloud Run services, Cloud Bigtable instances, and streaming resources.
• Run python3 _agents/scripts/verify_all.py --stack all to confirm endpoints, pipelines, Bigtable cells, and BigQuery tables pass health checks.
• Launch ./RUN_PROXY.sh to open the authenticated HUD on http://localhost:8080 and confirm the Active Stack badge in the top-right header.
• Confirm Module 2: Stream Simulator (/simulator) shows both the telemetry generator and stream processing pipeline in RUNNING state.
• Verify Module 4: Live Grid and AI Co-Pilot (/grid) displays all 15 assets updating live in nominal state.
• Test the Inject Anomaly, Run Gemini RCA, and Approve and Execute Mitigation closed-loop workflow before executive presentation.
• Check Module 5: Batch Analytics (/analytics) across all 3 SQL tabs to verify interactive BigQuery query execution.