Skip to content
DiffusionGemma

Path to Production: Serving dgem at Scale

This track takes dgem from your laptop to a production endpoint and keeps it running. Identifiers such as <PROJECT>, <REGION> and <ENDPOINT_ID> are placeholders. The cloud stages run a serving image that you build into your own Artifact Registry.

The journey

Section titled “The journey”
StepPageWhat you get
1Run on your laptopA first decision in minutes on Apple Silicon or a local NVIDIA GPU
2Use a remote GPUPoint the CLI, Studio and MCP at any hosted endpoint
3Deploy on Cloud RunYour own scale-to-zero GPU service: no cost when idle, a few minutes of cold start
4Production on Vertex AIAn always-warm endpoint with autoscaling replicas and zero cold start
5Gateway and routingOne URL for users, automatic failover from Vertex to Cloud Run, IAP
6Latency and capacityRequest design, engine settings and how to measure your own
7Operations runbookHealth, promotion, rollback, teardown and troubleshooting
8Monitoring and alertsHealth checks, dashboards, alerts and what to do when they fire

Hosting it yourself? Build the serving image first; Evaluate dgem on your own GPU covers GPU requirements, checking the install with the regression matrix, and a custom evaluation on your own labelled data.

Crawl → walk → run

Section titled “Crawl → walk → run”
StagePlatformUse it forCost modelCold start
CrawlApple Silicon (diffgemma, Metal, q4) or a local NVIDIA GPUDevelopment, writing policies, offline and private dataYour machineNone (Metal is a different engine; its latency doesn’t transfer to GPU serving)
WalkCloud Run GPU, 1× RTX PRO 6000Batch evaluation, research, bursty internal use, failoverPer instance-second; no cost when idleA few minutes from idle (lean image + Direct VPC egress)
RunVertex AI dedicated endpoint, g4-standard-48 + 1× RTX PRO 6000Production: always warm, IAM, autoscaling, monitoring, imagesPer replica-hour while deployedNone (min replicas ≥ 1)

The same image, policies, CLI, HTTP API and MCP server work on every stage. Latency depends on your policies, concurrency and network: measure it on your own deployment (method).

TL;DR recommendations

Section titled “TL;DR recommendations”
  • GPU: use a Blackwell GPU (RTX PRO 6000). The checkpoint is NVFP4 (4-bit); Blackwell runs FP4 natively, and the L4 was slower on every request type in our tests. Other GPUs are untested (requirements).
  • Production: a Vertex AI dedicated endpoint with minimum replicas ≥ 1, autoscaling on GPU duty cycle.
  • Failover and batch: a scale-to-zero Cloud Run GPU service using the lean image and Direct VPC egress.
  • One gateway (dgem serve) in front of both, routing vertex_first with automatic failover.
  • Requests: "samples": 1 by default; opt in to 4 only when you need agreement/standard error; never "auto" on latency-sensitive paths. Put all questions about one input in one request.
  • Change safely: validate every new serving image next to your current deployment before switching traffic (runbook).

Method

Section titled “Method”