v1.6.1 on GKE: ADR-019 requalification legs, with the Dynamo pairing
Date: 2026-10-21, from 16:20 UTC. Cluster: GKE gke-agent-authority-test (project scc-manual-testing-7, zone us-central1-a), two e2-medium nodes for leg 3; a temporary e2-standard-4 pair (adr019-tmp) for the Dynamo legs; no GPUs. Found with no Helm releases and only a KServe CRD installed. Artifacts: the published v1.6.1 release — image sha256:eca8ce634fda750b7d8f82ecf1ae4e350ea59c19a9ead07022f05ab5531eec0d, chart k8s-aibom 1.6.1 (sha256:e56839c1097bf20176d9a4d8484991acbb0b23b01de8e622d5006c843cc02a77), chart k8s-aibom-crds 1.6.1 (sha256:03b1f41b9dda1540bef96fa266941e1047e4f51247d7ebf7acd988dbbec86278). Values: AICR’s recipes/components/k8s-aibom/values.yaml as shipped (digest pin moved to the version under test; readiness.strictConfig=true, discovery: {}, no sinks), plus metrics.enabled=true where counters are read. Method as in the v1.5.1 and v1.6.0-rc.1 evidence; the cluster is restored to as-found at the end.
Scope asked for by NVIDIA AICR (Mark Chmarny, 2026-10-21): (1) dynamo-platform 1.4.2 + grove with one multinode DynamoGraphDeployment, DGD storage still v1alpha1 behind the conversion webhook; (2) the same with the Dynamo operator scaled to zero; (3) 1.5.1 → 1.6.x on stale CRDs with default values, then a rollback with the new CRDs left in place; (4) cost with the DGD and DCD watches on. Legs are numbered as asked; run order was 3, 1, 2, 4.
Release artifacts (verified before touching the cluster)
| Check | Result |
|---|---|
| GitHub release assets | 14, including five *.intoto.jsonl bundles |
gh attestation verify kubectl-aibom_v1.6.1_linux_amd64.tar.gz --owner GoogleCloudPlatform --bundle kubectl-aibom_v1.6.1.provenance.intoto.jsonl | succeeded (offline) |
gh attestation verify oci://…k8s-aibom@sha256:eca8ce63… (SLSA provenance, and the CycloneDX SBOM predicate) | both succeeded |
gh attestation verify on both chart digests | succeeded |
Plugin archive checksums vs kubectl-aibom_checksums.txt | match |
| Plugin behaviour fixed in 1.6.1 | help exits 0 with no kubeconfig; find --digest sha256: is rejected before any cluster access |
Leg 3 — 1.5.1 → 1.6.1 on stale CRDs, then rollback with the 1.6.1 CRDs in place
| UTC | Step | Result |
|---|---|---|
| 16:20 | helm install k8s-aibom --version 1.5.1 with AICR’s values (image sha256:7b027315…) | deployed; AIBOMControllerConfig/default Ready=True, Degraded=False |
| 16:21 | Opt in adr019-leg3; vLLM Deployment (vllm/vllm-openai:v0.6.3, --model Qwen/Qwen3-0.6B) and an nginx control, both pinned to a non-existent node | one AIBOM (runtime vllm, category inference, Ready); none for the control. inputHash 30de39cd…, bomHash 6a72da0f… |
| 16:21 | 1.5.1 CRD inspected | v1alpha1 served, v1beta1 served+storage; v1beta1 spec fields bomGeneration, discovery, logging, sinks, verification; discovery has inferenceRuntimeImagePatterns, namespaceSelector only — no workloadKinds |
| 16:31 | helm upgrade --version 1.6.1, same values with the 1.6.1 image digest, 1.5.1 CRDs left in place | deployed in 24 s; controller Ready on sha256:eca8ce63…; config Ready=True, Degraded=False; no SchemaPredatesController condition or event (expected: the skew check compares top-level spec fields and 1.6.x adds none; the one new field, discovery.workloadKinds, is nested and therefore not reported — Mark’s finding, tracked for 1.6.2) |
| 16:31 | AIBOM after the upgrade | still exactly one, Ready; regenerated once: inputHash e2b9589a… (the 1.6.x input hash includes the owned-workload list, so the composition differs from 1.5.1 by design), bomHash e9f8ce63… |
| 16:31 | Apply the 1.6.1 CRDs with the #209 remedy: helm show crds … 1.6.1 \| kubectl apply --server-side --force-conflicts --field-manager=k8s-aibom-crds -f - | both CRDs server-side applied; managers now helm (Update), k8s-aibom-crds (Apply); workloadKinds declared; config conditions unchanged; the AIBOM untouched (same hashes) |
| 16:32 | helm rollback k8s-aibom 1 (to 1.5.1) with the 1.6.1 CRDs left in place | success in 18 s; controller Ready on sha256:7b027315…; config Ready=True, Degraded=False |
| 16:33 | AIBOM after the rollback | one, Ready; inputHash back to the 1.5.1 value 30de39cd… (1.5.1 recomputes the same inputs deterministically); bomHash be36dd7f… (new generation timestamp) — one regeneration per direction, one status write per AIBOM, as AICR’s upgrade record states |
| 16:34 | helm upgrade --version 1.6.1 again (+ metrics.enabled=true) for the Dynamo legs | see leg 1 |
Reading for AICR’s record: with their values the 1.5.1 → 1.6.1 chart bump on stale CRDs upgrades cleanly on every deployer, the forced CRD apply is a no-op for the running controller, and rollback with the newer CRDs in place is clean. What is NOT covered by the 1.6.x guard: setting discovery.workloadKinds against 1.5.1 CRDs (nested field; #105-class failure; fix targeted at 1.6.2).
Leg 1 — dynamo-platform 1.4.2 + grove, one multinode DGD, v1alpha1 storage behind the conversion webhook
Install: helm install dynamo-platform nvidia-dynamo/dynamo-platform --version 1.4.2 -n dynamo-system with global.grove.install=true, global.grove.enabled=true, global.etcd.install=true, global.nats.install=true (operator, etcd, NATS and grove-operator Running on the adr019-tmp nodes in 72 s). CRDs as AICR described: dynamographdeployments and dynamocomponentdeployments v1alpha1(served, storage) + v1beta1(served), conversion: Webhook to dynamo-platform-dynamo-operator-webhook-service, failurePolicy: Fail; grove’s podcliquesets, podcliques, podcliquescalinggroups, podgangs. k8s-aibom 1.6.1 restarted once so the new CRDs are watched: supervisor watch healthy for DynamoGraphDeployment and DynamoComponentDeployment (reads nvidia.com/v1beta1).
Fixture: Dynamo v1.4.2 examples/backends/vllm/deploy/disagg-multinode.yaml (nvidia.com/v1alpha1; Frontend + prefill and decode workers, multinode.nodeCount: 2, nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.2, --model Qwen/Qwen3-0.6B, 1 GPU each), applied unchanged into the opted-in namespace adr019-dynamo.
| UTC | Step | Result |
|---|---|---|
| 16:39 | Apply the DGD | API server stores v1alpha1, serves v1beta1 (the deprecation warning fires). The operator + grove materialize one PodCliqueSet, five PodCliques, two PodCliqueScalingGroups and five pods (no DynamoComponentDeployments on this path). Pods Pending: no GPUs |
| 16:40 | AIBOMs in the namespace | exactly one, nvidia-com-dynamographdeployment-vllm-disagg, owner reference DynamoGraphDeployment/vllm-disagg (nvidia.com/v1beta1), Ready. Nothing for the PodCliqueSet, cliques or pods: the Grove chain is rolled up |
| 16:40 | The document (via kubectl aibom view, 1.6.1 plugin) | root application vllm-disagg; 8 components: three vllm 1.4.2 runtime components (inferred, image_pattern, locators spec.components[0..2].podTemplate.spec.containers[0].image, roles frontend/worker/worker), three container components (nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.2, digest unresolved — pods never ran), two Qwen/Qwen3-0.6B model components (declared, container_arg, locators …containers[0].args[0 1](--model) on the two workers). backendFramework is unset in this v1alpha1 example, so the runtime is inferred from the image rather than declared |
| 16:42 | Frontend replicas 1→2 (patched through v1beta1) | still one AIBOM; observedGeneration 1→2; inputHash unchanged (replica count is not a document input); six pods |
| 16:50 | Frontend image tag 1.4.2→1.4.1 (a content change) | still one AIBOM; inputHash 3077296f… → e8580643…, observedGeneration 3: one regeneration |
| 16:49 | A second DGD (vllm-disagg-2) applied while the operator is up | its own AIBOM within 40 s |
Leg 2 — the same with the Dynamo operator scaled to zero
| UTC | Step | Result |
|---|---|---|
| 16:51 | scale deploy/dynamo-platform-dynamo-operator-controller-manager --replicas=0; webhook Service has no endpoints | — |
| 16:51–16:59 | Polled AIBOMControllerConfig/default every 30 s for 7 min 15 s | Degraded=False throughout; aibom_watch_healthy{kind="DynamoGraphDeployment"} and …DynamoComponentDeployment stayed 1; no WatchUnhealthy event; no watch degraded log line |
| 16:52 | Existing DGD AIBOMs | both still present and Ready, hashes unchanged |
| 16:56 | New apps/v1 Deployment (vLLM) in the same namespace | AIBOM within 20 s; controller 0 restarts |
| 16:52 | kubectl apply of a third DGD while the webhook is down | rejected by the API server: conversion webhook for nvidia.com/v1alpha1, Kind=DynamoGraphDeployment failed: Post "https://dynamo-platform-dynamo-operator-webhook-service…": connection refused |
| 17:01 | Direct reads while the webhook is down | kubectl get dynamographdeployments.v1beta1.nvidia.com -A --chunk-size=1 succeeds (both DGDs listed); GET /apis/nvidia.com/v1beta1/…/dynamographdeployments?limit=1 succeeds with and without resourceVersion=0; kubectl get …v1beta1… vllm-disagg (single-object GET) fails with the same conversion error |
| 16:59 | Operator scaled back to 1 | nothing to recover; watches never left healthy; the third DGD applied afterwards gets its AIBOM in 40 s |
What this means, stated plainly:
- The API server keeps serving v1beta1 lists and watches from its per-version watch cache while the conversion webhook is down; only single-object GETs and all writes reach the webhook and fail. So k8s-aibom’s informer stays established and its steady-state probe (a
Limit=1LIST through the uncached reader, Design 004) succeeds, which is why noDegraded/WatchUnhealthysignal appeared. During the outage the Dynamo API is effectively read-only: no DGD can be created, changed or deleted, so there is no change for the inventory to miss, and the last-known inventory remains correct. - The gap is observability, not correctness: an operator reading the
AIBOMControllerConfigconditions does not learn that the Dynamo kinds are running on cached reads. Filed as a follow-up: make the steady-state probe a single-object GET (take one name from the cached list, read it uncached), which this run shows does fail through the webhook. Targeted at 1.6.2. - Watch-isolation itself held: the apps/v1 path, the other third-party kinds and the controller process were unaffected (0 restarts), which is what #127 set out to guarantee. The crash-loop and “zero AIBOMs” failure modes recorded in the #127 investigation did not occur.
Leg 4 — steady-state cost with the DynamoGraphDeployment and DynamoComponentDeployment watches on
Window 17:02:28–17:17:37 UTC (900 s), operator up, present in the opted-in namespaces: two DGDs (five + six Pending pods under grove), three vLLM Deployments and one nginx control, all pinned to a non-existent node. Third-party kinds watched: DynamoGraphDeployment, DynamoComponentDeployment, InferenceService (pre-existing KServe CRD); LWS, DisaggregatedSet and NIMService CRDs absent. Counters read from the controller’s own /metrics over kubectl port-forward, kubectl top for the working set.
| Measure | Value |
|---|---|
rest_client_requests_total delta | 59 in 900 s = 3.9 requests/min (48 GET, 11 PUT) |
| Reconciles in the window | 11, all fast-path (inputs unchanged) status touches, which are the 11 PUTs; the Dynamo operator and grove update DGD status while pods sit Pending, and each update re-enqueues the DGD |
aibom_watch_errors_total delta | 0 |
| Process CPU over the window | 2.0 s = 2.2 mCPU average (kubectl top: 3m throughout) |
| Memory | 28–34 MiB working set (kubectl top); 61 MiB RSS |
aibom_watch_healthy at the end | 1 for all three watched kinds |
Against the v1.6.0-rc.1 walk (3.26 requests/min with three present kinds and no DGD) and the 1.5.1 baseline (0.96/min, zero writes at 1,002 workloads), the difference is the 11 fast-path status writes driven by DGD status churn from a graph whose pods never schedule; with running pods that churn stops. A generation-change filter on third-party kinds would remove it entirely and is noted for 1.7.
Teardown (17:18–17:20 UTC)
Both DGDs deleted: their AIBOMs went with them by owner reference while the two Deployments’ AIBOMs stayed until their namespace was deleted. helm uninstall of k8s-aibom left no Deployment, RBAC, ServiceAccount or AIBOMControllerConfig; the two k8s-aibom CRDs were deleted by hand (the chart keeps them by design). helm uninstall dynamo-platform, then its 14 nvidia.com/grove.io CRDs and the namespaces. End state: zero Helm releases, zero PVCs, no run namespaces; CRDs present are the pre-existing inferenceservices.serving.kserve.io and rayclusters.ray.io, neither installed by this run. The adr019-tmp node pool was deleted afterwards.
Follow-ups filed
- #216 — the watch-supervisor probe cannot see a conversion-webhook outage because the API server serves lists from its per-version cache; probe with a single-object GET (1.6.2).
- Nested-field schema skew (
discovery.workloadKindson 1.5.1 CRDs) is not reported by the guard or the skew check; Mark Chmarny is filing the kind repro (1.6.2). - Generation-change filtering for third-party kinds, to drop the fast-path status writes driven by DGD status churn (1.7).