Skip to content

Decision Models Practitioner Guide

Audience: Software Engineers, ML Platform Architects, and Applied AI Practitioners Runtime: dgem (DiffusionGemma 26B-A4B IT) across Apple Silicon (diffgemma), Cloud Run GPU, and Vertex AI Dedicated Endpoints


Most production AI workloads are not open-ended conversations. They are structured decisions: routing support tickets, triaging security alerts, screening pull requests, verifying policy compliance, classifying user intent across large catalogs, and gating agent tool executions.

When teams use autoregressive reasoning models for every high-volume classification or routing turn, they pay a steep tax in latency (seconds of sequential token generation), compute cost, and brittle post-hoc confidence heuristics. Conversely, traditional task-specific classifiers require curated training pipelines and cannot adapt zero-shot when a policy changes.

dgem bridges these two worlds by operating DiffusionGemma (26B-A4B Mixture-of-Experts, ~3.8B active parameters per forward pass) as a Zero-Shot System-1 Decision Model:

  1. Declarative Policies (.json.tmpl): You express business logic as version-controlled Go templates (templates/*.json.tmpl) with explicit choice, boolean (noul), or ordinal score slots.
  2. Single-Pass Parallel Readout: Instead of generating a chain-of-thought token by token, dgem resolves up to 8 decision slots simultaneously in a single discrete diffusion denoising pass (~135–265 ms warm on GPU).
  3. Intrinsic Epistemic Hesitation (H~m\tilde{H}_m): Because every slot is scored via restricted softmax over its exact allowed vocabulary Vm\mathcal{V}_m, dgem emits an exact probability distribution and Cardinality-Normalized Shannon Entropy (H~m∈[0,1]\tilde{H}_m \in [0, 1]) on every read.
  4. Selective System-2 Escalation: Easy, low-entropy decisions (H~m<0.16\tilde{H}_m < 0.16) finish immediately in Stage 1. Ambiguous or out-of-distribution inputs (H~mβ‰₯0.16\tilde{H}_m \ge 0.16) automatically escalate to a Stage-2 Gemini model (gemini-3.8-flash or gemini-3.5-flash-lite) or a human review queue.

In a two-stage decision architecture, dgem serves as the System-1 reflex layer in front of heavier System-2 reasoning models or human-in-the-loop workflows:

flowchart LR
    In["Incoming Document / State / Image"] --> Tmpl["Policy-as-Template (.json.tmpl)<br/>document_first Layout"]
    Tmpl --> Stage1["Stage 1: dgem Readout<br/>Single-Pass Restricted Softmax"]
    Stage1 --> Gate{"Normalized Hesitation<br/>H_norm >= tau (0.16)?"}
    Gate -- "No (Clear: < 16%)" --> Fast["Return Stage 1 Decision<br/>(~135-265 ms on GPU)"]
    Gate -- "Yes (Uncertain: >= 16%)" --> Stage2["Stage 2: Gemini Cascade<br/>(gemini-3.8-flash / gemini-3.5-flash-lite)<br/>or Human Review Interrupt"]
    Stage2 --> Out["Audited Final Decision<br/>+ Full Probability & Trace Receipt"]
Architectural PropertyAutoregressive LLM PromptingTraditional Fine-Tuned Classifierdgem Decision Model (Stage 1 + Cascade)
Policy AdaptabilityZero-shot (prompt edit)Requires retraining & redeployingZero-shot (.json.tmpl edit)
Multi-Question LatencyScales with output length & reasoning traceFast, single-headSingle forward pass for up to 8 slots
Output ContractCan fail JSON formatting or hallucinate keysFixed class indexRestricted softmax over exact schema options
Uncertainty SignalVerbalized confidence is poorly calibratedNeeds per-head temperature fittingNormalized entropy H~m\tilde{H}_m + optional calibration
Stage-2 EscalationRuns expensive model on 100% of trafficAd-hoc threshold per headPrincipled hesitation gate (H~mβ‰₯Ο„\tilde{H}_m \ge \tau)

Every dgem decision follows a deterministic five-step pipeline across the CLI (dgem decide), HTTP Gateway (POST /api/decide, POST /v1/systemone), MCP Server (dgem mcp), and Web Studio:

  1. Template Compilation (pkg/template): The .json.tmpl policy renders caller variables into a JSON payload containing the input state and an array of questions.
  2. Prompt Layout Construction (document_first): By default (DEFAULT_LAYOUT=document_first), the serving image places the input document/state first and the question schema second inside the user turn so every question attends to the document from the start of the prompt (Prompt layout).
  3. Single-Pass Restricted Softmax (deploy/cloudrun/server/structured_server.py): For each question mm with option vocabulary Vm\mathcal{V}_m (mapped to single-token option labels A–Z, or up to 52 labels with opt-in az_aa / az_lower), the model extracts the slot logits zm,vz_{m,v} and computes restricted-softmax probabilities with optional temperature scaling Tmβˆ—T_m^*: pm(v∣x)=exp⁑(zm,v/Tmβˆ—)βˆ‘u∈Vmexp⁑(zm,u/Tmβˆ—)p_m(v \mid x) = \frac{\exp(z_{m,v} / T_m^*)}{\sum_{u \in \mathcal{V}_m} \exp(z_{m,u} / T_m^*)}
  4. Cardinality-Normalized Hesitation (H~m\tilde{H}_m): Raw Shannon entropy Hm=βˆ’βˆ‘v∈Vmpm(v)ln⁑pm(v)H_m = -\sum_{v \in \mathcal{V}_m} p_m(v)\ln p_m(v) grows with the number of options ∣Vm∣|\mathcal{V}_m|. Dividing by ln⁑∣Vm∣\ln |\mathcal{V}_m| yields a scale-free hesitation score in [0,1][0, 1]: H~m=Hmln⁑∣Vm∣∈[0,1]\tilde{H}_m = \frac{H_m}{\ln |\mathcal{V}_m|} \in [0, 1] In the Web Studio and CLI (studio/src/hesitation.ts), H~m<0.16\tilde{H}_m < 0.16 is reported as Clear, 0.16≀H~m≀0.500.16 \le \tilde{H}_m \le 0.50 as Somewhat Unsure, and H~m>0.50\tilde{H}_m > 0.50 as Very Unsure (Confidence beyond Shannon).
  5. Post-Processing & Escalation (cmd/serve.go, pkg/decisionindex): When cascade_mode: "entropy" is enabled and a slot exceeds cascade_threshold, dgem opens an OpenTelemetry child span (dgem.cascade.gemini) and escalates that question to gemini-3.8-flash or gemini-3.5-flash-lite (Authoring and Stage 2 cascades).

Each feature below is validated against dgem’s regression matrix (Regression matrix) and exposed with consistent semantics across the CLI, HTTP Gateway, MCP Server, and Studio.

3.1 Hesitation-Gated Cascades (H~mβ‰₯0.16\tilde{H}_m \ge 0.16) and Temperature Scaling (Tβˆ—T^*)

Section titled β€œ3.1 Hesitation-Gated Cascades (H~mβ‰₯0.16\tilde{H}_m \ge 0.16H~m​β‰₯0.16) and Temperature Scaling (Tβˆ—T^*Tβˆ—)”

Raw restricted-softmax probabilities provide a strong zero-shot ranking signal for separating confident answers from errors.

  • Selective Escalation: Gate Stage-2 calls (gemini-3.8-flash or gemini-3.5-flash-lite) or human-approval interrupts (examples/genkit-tool-approval/) on normalized entropy H~mβ‰₯0.16\tilde{H}_m \ge 0.16 (or raw entropy Hmβ‰₯0.35Β natsH_m \ge 0.35\text{ nats}).
  • Per-Policy Calibration: When downstream consumers need probabilities that match empirical accuracy (low ECE / Brier score), fit a post-hoc temperature Tβˆ—T^* (temperature or temperature_by_type in your .json.tmpl) on a held-out labelled split using scripts/policy_calibration.py (Calibrate your policy).

Placing the document/state before the question definitions (layout: "document_first", the default in the serving image and dgem systemone serve) improves accuracy on long-context Grounded NLI, hallucination detection, and catch-all intent suites without adding latency (Prompt layout).

3.3 Multi-Slot Canvas Batching (≀8\le 8 Questions) and Conditional DAGs

Section titled β€œ3.3 Multi-Slot Canvas Batching (≀8\le 8≀8 Questions) and Conditional DAGs”
  • Independent Questions: Group up to 8 related questions (MaxSlotsPerPass = 8) into a single .json.tmpl policy (for example, templates/support_triage.json.tmpl or templates/github_spam_triage.json.tmpl). All slots are answered in one forward pass. Beyond 8 questions, pkg/decisionindex automatically partitions questions into multi-slot batches.
  • Sequential Dependencies (depends_on / ask_if): When a follow-up question only makes sense given an earlier answer (for example, asking for severity only when is_incident == "yes" in templates/secops_conditional_dag.json.tmpl), declare depends_on and ask_if: ["yes"]. Leave them out when a single pass suffices.

The serving image maps choice options to single letters A–Z by default (MaxOptionsPerSlot = 26). When a taxonomy has more than 26 options (such as 77 banking intents or 151 cross-domain intents), dgem offers three graduated strategies:

StrategyFlag / OptionPassesBest For
1. Two-Stage Bracket Tournament (Default)--bracket-size 20 (dgem systemone serve)2+2+Maximum zero-shot accuracy up to K=255+K = 255+ options with no external retriever required.
2. Extended 52-Option Label Sets (Experimental)--labels az_aa or az_lower11 (up to K=52K=52)Medium-wide taxonomies (27≀K≀5227 \le K \le 52) when running against a serving image with label-set support.
3. Pre-Filtered Shortlist Routing (Opt-in)--shortlist-topk M (e.g. M=15) + prior_logits11Latency-critical wide choices (K>26K > 26, measured up to K=151K = 151) when an upstream bi-encoder or lexical ranker supplies prior_logits.
  • Catch-All Handling (--catch-all): On wide taxonomies that include an "out_of_scope" or "none of the listed" option, --catch-all final holds the catch-all out of Round-1 brackets and injects it into the final round, while ShortlistFunc automatically preserves catch-all keys alongside the top-MM shortlist.
  • Cosine Prior Normalization (NormalizeCosinePriorLogits): Raw bi-encoder cosine similarities in [βˆ’1,1][-1, 1] occupy a narrow numeric range. Before passing embedding scores as prior_logits, convert them to temperature-scaled log-softmax logits with decisionindex.NormalizeCosinePriorLogits(cosines, 0.07) so they live on a proper log-probability simplex while preserving exact rank order.

3.5 Opt-In Pre-Gated Prior Fusion (--prior-weight, --prior-gate-mode)

Section titled β€œ3.5 Opt-In Pre-Gated Prior Fusion (--prior-weight, --prior-gate-mode)”

When callers supply per-option log-priors (prior_logits on a /v1/systemone question, such as historical class base rates or retrieval log-probabilities), dgem can combine them with the model’s log-probabilities via log-linear Product-of-Experts fusion (pkg/decisionindex/engine.go):

  1. Pre-Fusion Hesitation Invariant (raw_normalized_entropy): FusePriorWithGateMode always computes and records raw_entropy and raw_normalized_entropy (H~raw\tilde{H}_{\text{raw}}) before applying prior_logits. Downstream Stage-2 cascades must gate on raw_normalized_entropy so a sharp external prior never masks model uncertainty.
  2. Choosing a Gate Mode (--prior-gate-mode):
    • hesitant (Recommended for tie-breaking): Fuses prior_logits only when the model is uncertain (H~rawβ‰₯Ο„\tilde{H}_{\text{raw}} \ge \tau, default Ο„=0.16\tau = 0.16), leaving confident model predictions untouched even if the prior disagrees.
    • proportional: Scales the effective prior weight by Ξ±β‹…H~raw\alpha \cdot \tilde{H}_{\text{raw}}, smoothly increasing prior influence as model hesitation rises.
    • confident: Fuses only when H~raw<Ο„\tilde{H}_{\text{raw}} < \tau.

3.6 Co-Denoised Evidence Slots (evidence_prompt + evidence_criteria)

Section titled β€œ3.6 Co-Denoised Evidence Slots (evidence_prompt + evidence_criteria)”

For fine-grained visual or textual verification on standard choice questions (K≀26K \le 26), a question on /v1/systemone can include evidence_prompt and caller-supplied evidence_criteria (β‰₯2\ge 2 options).

  • The adapter places a companion <question_id>_evidence choice slot immediately ahead of <question_id> on the same diffusion canvas (consuming 2 of the 8 slots in that pass) and returns the selected cue in answers[q].evidence.
  • For multi-image visual inspection (templates/multimodal/contact_sheet_verify.json.tmpl), tiling candidate crops into a numbered contact sheet and asking an evidence slot to identify the matching tile before the final decision slot resolves the comparison in a single forward pass.

dgem is a discrete choice and scoring engine, not a continuous coordinate regressor. For image localization (dgem locate, POST /api/locate, MCP locate_object; see What dgem can do with images):

  1. Stage 1 (dgem): Answers target presence and coarse 3Γ—3 grid cell in one pass. With --skip-h 0.16, images where dgem is confidently "absent" return immediately (path: "skipped_absent"), saving the Stage-2 call.
  2. Stage 2 (gemini-3.8-flash / gemini-3.5-flash-lite): Predicts continuous box_1000 / box_pct coordinates at low thinking.
  3. Optional Stage 3 (DGEM_SAM_URL): Refines the bounding box into a pixel mask and polygon.

Empirical testing across the dgem evaluation suites ruled out several tempting heuristics. Avoid these patterns in production:

Anti-PatternWhat HappensWhat to Do Instead
Direct continuous [0..1000] coordinate regression in dgem choice slotsQuantizing image coordinates into 16 or 26 discrete bins yields coarse boxes and quantization rounding error.Use dgem locate (dgem presence + 3Γ—3 cell gate β†’\to gemini-3.8-flash bounding box β†’\to optional SAM mask).
Gating Stage-2 cascades on post-fusion normalized_entropy when prior_logits are fusedA strong but wrong external prior depresses post-fusion entropy on errors, collapsing error-detection AUROC.Always gate cascades on pre-fusion raw_normalized_entropy and prefer --prior-gate-mode hesitant or proportional.
Overpacking >8>8 independent questions into one raw passExceeding the 256-token output canvas degrades slot formatting and per-question accuracy.Keep policies to ≀8\le 8 questions per pass, or route through POST /v1/systemone (MaxSlotsPerPass = 8 auto-batching).
Enabling --shortlist-contrastive by default on compact snake_case intent keysAppending [distinguishing: ...] sibling token tags helps verbose overlapping product descriptions, but hurts compact intent names (e.g. banking77).Keep --shortlist-contrastive=false (the default) unless A/B tested on verbose overlapping catalog descriptions.
Replacing balanced bracket tournaments with 2-level coarse-to-fine taxonomy slot triesA Pass-1 coarse super-category routing error is unrecoverable in Pass 2 (because Pass 2 only sees the chosen child branch), and broad super-categories flatten out-of-scope entropy.Use dgem systemone serve balanced 20-way bracket tournaments (--bracket-size 20 --catch-all final) or --shortlist-topk M with normalized embedding prior_logits.
Enabling --null-prior-debias or --dual-mirror in productionContent-free positional de-biasing helps one suite while hurting another; dual-mirror doubles slot consumption with little gain over H~m\tilde{H}_m.Keep both flags off in production; use neutral question IDs and document_first layout (Confidence and calibration).
Using bias-inducing question IDs (mirror, reverse, opposite, alt)Question IDs are part of the prompt seen by the model; suggestive names shift the answer distribution.Use neutral, descriptive snake_case IDs (intent, urgency, policy_violation).

dgem supports four execution topologies behind a single CLI, Gateway, and MCP interface (From laptop to production):

flowchart TD
    Start{"What is your workload?"}
    Start -- "Local Dev / Policy Authoring" --> Metal["1. Laptop (diffgemma on Apple Silicon)<br/>dgem serve --local"]
    Start -- "Bursty / Scale-to-Zero / Batch Jobs" --> CR["2. Serverless Cloud Run GPU<br/>(RTX Pro 6000 / L4)<br/>deploy/batch/ for Cloud Run Jobs & Dataflow"]
    Start -- "Low-Latency 24/7 Production Traffic" --> VAI["3. Vertex AI Dedicated Endpoint (G4)<br/>Public /invoke/v1/* or Private PSC :rawPredict"]
    VAI --> GW["4. High-Availability Gateway (dgem serve)<br/>--default-backend vertex_first<br/>Warm Vertex primary + Cloud Run GPU failover"]
    CR --> GW
TopologyTypical Warm LatencyCold StartScaling & Cost ProfileGuide
1. Apple Silicon (diffgemma)~250–600 ms (M-series Max/Ultra)InstantFree local iteration (32 GB+ unified memory)Run on your laptop
2. Cloud Run GPU (dgemma)~185–265 ms (1–3 questions)~2–3 min from zeroScales 0..N per-second GPU billing; ideal for bursty traffic & deploy/batch/Deploy on Cloud Run Β· Batch guide
3. Vertex AI Dedicated (G4)~135–185 ms (1–3 questions)0 ms (always warm)Dedicated 24/7 GPU node(s); supports public /invoke/v1/* and private PSC :rawPredictProduction on Vertex AI Β· Vertex vs. Cloud Run
4. Gateway (vertex_first)~135–185 msMasked by failoverRoutes to warm Vertex AI primary with automatic Cloud Run GPU failover & Stage-2 Gemini cascadeGateway and routing

Before promoting a new .json.tmpl policy or serving configuration to production:

  1. Schema & Layout Hygiene:
    • Keep choice questions to ≀26\le 26 options per pass (or configure /v1/systemone tournament brackets / --shortlist-topk / --labels az_aa).
    • Use neutral question IDs, samples: {{ default 1 .samples | toJson }}, and document_first layout.
  2. Calibration & Escalation:
    • Evaluate your policy on a labelled dataset (Run a dataset) and inspect hesitation H~mβ†’\tilde{H}_m \to error AUROC.
    • Configure Stage-2 escalation (--cascade-model gemini-3.8-flash or gemini-3.5-flash-lite) or Genkit tool-approval interrupts (examples/genkit-tool-approval/) on H~mβ‰₯0.16\tilde{H}_m \ge 0.16 (or raw_normalized_entropy when prior_logits fusion is enabled).
  3. Security & Credentials:
    • Protect the serving image with DGEM_SERVER_KEY (X-DGem-Key) and the HTTP gateway with Cloud Run IAP (--iap-audience) or --api-keys (X-DGem-Api-Key) (Security and access).
  4. Regression Gate: