Skip to content

Step 4: Production on Vertex AI

A Vertex AI dedicated endpoint is the run stage: always warm, autoscaling replicas and Cloud Monitoring, on the same container image as Cloud Run. The container is deployed with arbitrary custom routes (invokeRoutePrefix: "/*"), so every server route is available under /invoke/* (why). Deploy it with a service key so that only your own services can make decisions on it, and give people access through the gateway.

  • One g4-standard-48 + 1× RTX PRO 6000 replica per deployed model, with the vision tower on.
  • No cold start while minimum replicas ≥ 1. Deploying a new model takes roughly 10–15 minutes.
  • Latency and throughput depend on your policies (questions, samples, images, reasoning budget) and concurrency; measure your own deployment with scripts/serving_speed.py (see Latency and capacity).
  • APIs: aiplatform.googleapis.com, artifactregistry.googleapis.com, storage.googleapis.com.

  • Vertex AI serving quota for RTX PRO 6000 in your region:

    Terminal window
    gcloud beta quotas info describe CustomModelServingRTXPRO6000GPUsPerProjectPerRegion \
    --service=aiplatform.googleapis.com --project=<PROJECT> --format=json \
    | jq -r '.dimensionsInfos[] | select(.dimensions.region == "<REGION>") | .details.value'
  • Weights in gs://dgem-weights-<PROJECT>/dgemma/ (see Cloud Run §2); the endpoint mounts them as the model artifact.

  • A service account for the endpoint (default dgemma-gpu-sa@<PROJECT>.iam.gserviceaccount.com) with read access to the bucket.

  • Vertex AI requires an Artifact Registry or Container Registry image. Build the serving image into your project (GCP_PROJECT=<PROJECT> make cloudrun-build) and pin the commit tag or digest it prints.

  • A service key for the serving image, stored in Secret Manager as dgem-server-key. When the secret exists, the deploy script sets it as DGEM_SERVER_KEY on the model and the serving image refuses every POST without a matching X-DGem-Key header (GET /health and GET /ready stay open). Without the secret the script prints a warning and the serving image accepts any POST that reaches it. Give the key to the services that call the endpoint (your gateway, batch jobs, probes), not to people:

    Terminal window
    openssl rand -base64 32 | tr -d '\n' | gcloud secrets create dgem-server-key --project=<PROJECT> --data-file=-

    Vertex AI model environment variables are plain values, so anyone who can read the model resource (aiplatform.models.get) can read the key: keep that permission to the operators who deploy the endpoint.

Terminal window
GCP_PROJECT=<PROJECT> GCP_PROJECT_NUMBER=<PROJECT_NUMBER> GCP_REGION=<REGION> \
IMAGE_URI=<REGION>-docker.pkg.dev/<PROJECT>/dgem/dgemma:<tag> \
./scripts/deploy_vertex_endpoint.sh

The default profile (VERTEX_PROFILE=g4-rtxpro6000) configures:

SettingValue
Machineg4-standard-48 + 1× NVIDIA_RTX_PRO_6000
Replicasmin 1 (VERTEX_MIN_REPLICAS), max 2 (VERTEX_MAX_REPLICAS), autoscale at 70% GPU duty cycle
EngineKV_CACHE_GB=12, MAX_SEQS=32, MAX_INFLIGHT=8, DEFAULT_SAMPLES=1, vision on
Endpointdedicated DNS, display name dgemma-dedicated-g4 (VERTEX_ENDPOINT_NAME; reused if it exists)
StartuphealthRoute: "/ready" (503 until vLLM serves, so crash-looping engines fail deploy instead of taking traffic)

The script prints the endpoint ID and the invoke URLs:

https://<ENDPOINT_ID>.<REGION>-<PROJECT_NUMBER>.prediction.vertexai.goog/v1/projects/<PROJECT>/locations/<REGION>/endpoints/<ENDPOINT_ID>/invoke/...
POST /v1/chat/completions structured decisions (what dgem uses)
POST /v1/systemone /v1/systemone decision protocol
POST /ready readiness (200 when vllm_ready, 503 otherwise; Vertex healthRoute polls GET /ready internally)
POST /health cold-start telemetry (always 200)
Terminal window
INVOKE=https://<ENDPOINT_ID>.<REGION>-<PROJECT_NUMBER>.prediction.vertexai.goog/v1/projects/<PROJECT>/locations/<REGION>/endpoints/<ENDPOINT_ID>/invoke
# Without the service key, POST requests (decisions and external /invoke/* probes) are refused
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" -d '{}' $INVOKE/v1/chat/completions
# -> 401
# As an operator, with the key from Secret Manager (Vertex dedicated endpoints accept POST on /invoke/*)
export GCP_PROJECT=<PROJECT> GCP_PROJECT_NUMBER=<PROJECT_NUMBER>
export DGEM_SERVER_KEY="$(gcloud secrets versions access latest --secret=dgem-server-key --project=<PROJECT>)"
curl -sf -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "X-DGem-Key: ${DGEM_SERVER_KEY%%,*}" -H "Content-Type: application/json" -d '{}' $INVOKE/ready
# -> HTTP 200: {"status": "ok", "vllm_ready": true, "phase": "ready", "warmed": true, ...}
# (POST $INVOKE/health always returns 200 once the container starts; check /ready or "vllm_ready": true)
./bin/dgem decide --vertex-url <ENDPOINT_ID> --gcp-auth -t templates/support_triage.json.tmpl \
-v ticket="Charged twice" --stats
./bin/dgem decide --vertex-url <ENDPOINT_ID> --gcp-auth -t templates/multimodal/bbox_localization.json.tmpl \
-I fixtures/bbox/bbox-t1-03-offgrid-card.png -v target=checkout_summary_card
  • Measure one replica’s throughput for your policies with a concurrency sweep; past the point where requests start to queue (MAX_INFLIGHT), latency rises while throughput plateaus. Set VERTEX_MAX_REPLICAS for your peak; the endpoint adds replicas on GPU duty cycle. Details: Latency and capacity.
  • Keep minimum replicas ≥ 1 and a periodic POST /invoke/predict {"health": true} probe (the gateway runs one every 4 minutes). If an endpoint reports 0 available replicas it returns 429 until it scales back up.

Deploy the new image next to the current one at 0% traffic, verify it on the same endpoint, then move traffic:

Terminal window
# 1. New model on the same endpoint, 0% traffic (the current split is kept)
VERTEX_NEW_TRAFFIC=0 GCP_PROJECT=<PROJECT> GCP_PROJECT_NUMBER=<PROJECT_NUMBER> \
IMAGE_URI=<NEW_IMAGE> VERTEX_MODEL_NAME=dgemma-g4-<TAG> ./scripts/deploy_vertex_endpoint.sh
# 2. Move traffic (deployed model IDs from `gcloud ai endpoints describe <ENDPOINT_ID> --region=<REGION>`)
curl -X PATCH -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" \
"https://<REGION>-aiplatform.googleapis.com/v1/projects/<PROJECT>/locations/<REGION>/endpoints/<ENDPOINT_ID>?updateMask=trafficSplit" \
-d '{"trafficSplit": {"<NEW_DEPLOYED_MODEL_ID>": 100, "<OLD_DEPLOYED_MODEL_ID>": 0}}'
# 3. After a soak period, undeploy the old model (its model resource stays uploaded for rollback)
gcloud ai endpoints undeploy-model <ENDPOINT_ID> --region=<REGION> --deployed-model-id=<OLD_DEPLOYED_MODEL_ID>

Rollback while both are deployed is instant (move the split back); after undeploying, redeploy the old model resource (~15 minutes). Before step 2 on a new serving image, run the comparison in Promote a new serving image.

Private endpoint (Private Service Connect)

Section titled “Private endpoint (Private Service Connect)”

A dedicated private (PSC) endpoint has no invoke routes, so /invoke/* is unavailable there. Upload the model with a single route instead and call it with :rawPredict:

Terminal window
ROUTE_MODE=predict ./scripts/deploy_vertex_endpoint.sh # predictRoute /predict, healthRoute /ready
  • Callers inside the VPC use the private host: http://<ENDPOINT_ID>-<REGION>-<PROJECT_NUMBER>.prediction.p.vertexai.goog/v1/projects/<PROJECT_NUMBER>/locations/<REGION>/endpoints/<ENDPOINT_ID>:rawPredict (resolved through your private DNS zone). Use plain http on port 80: port 443 serves a self-signed certificate, and the traffic stays inside the VPC. Vertex still enforces IAM there, so send an OAuth access token (dgem mints one for these hosts, as for dedicated endpoints) plus X-DGem-Key if the server uses one.
  • dgem (-u, --vertex-url, DGEM_VERTEX_URL) keeps the :rawPredict URL as given and the gateway sends both /v1/chat/completions and /v1/systemone traffic to it. http is accepted only for *.prediction.p.vertexai.goog hosts with a :rawPredict path; every other Vertex URL must be https.
  • The serving image’s single route dispatches on the body: chat messages → decisions, a questions map or a multipart upload → /v1/systemone, {"ready": true} → readiness (200 when vllm_ready, 503 otherwise), {"health": true} → the cold-start health report (send X-DGem-Key if the server uses one).
  • The gateway reads the endpoint’s state from the Vertex control plane (no data-plane health route through PSC).
  • Creating the PSC endpoint itself (allowlisted projects, forwarding rule, private DNS, Direct VPC egress for the gateway) follows the Private Service Connect guide.
Terminal window
GCP_PROJECT=<PROJECT> make vertex-teardown # undeploys models and deletes the dgemma-dedicated-g4 endpoint

A deployed replica bills per hour whether or not it serves traffic; tear down endpoints you don’t need, and keep a scale-to-zero Cloud Run service for occasional use.

Gateway and routing: one URL in front of Vertex and Cloud Run, with automatic failover.