Decision Models Practitioner Guide
Audience: Software Engineers, ML Platform Architects, and Applied AI Practitioners Runtime:
dgem(DiffusionGemma 26B-A4B IT) across Apple Silicon (diffgemma), Cloud Run GPU, and Vertex AI Dedicated Endpoints
Executive Summary
Section titled βExecutive SummaryβMost production AI workloads are not open-ended conversations. They are structured decisions: routing support tickets, triaging security alerts, screening pull requests, verifying policy compliance, classifying user intent across large catalogs, and gating agent tool executions.
When teams use autoregressive reasoning models for every high-volume classification or routing turn, they pay a steep tax in latency (seconds of sequential token generation), compute cost, and brittle post-hoc confidence heuristics. Conversely, traditional task-specific classifiers require curated training pipelines and cannot adapt zero-shot when a policy changes.
dgem bridges these two worlds by operating DiffusionGemma (26B-A4B Mixture-of-Experts, ~3.8B active parameters per forward pass) as a Zero-Shot System-1 Decision Model:
- Declarative Policies (
.json.tmpl): You express business logic as version-controlled Go templates (templates/*.json.tmpl) with explicitchoice,boolean(noul), or ordinalscoreslots. - Single-Pass Parallel Readout: Instead of generating a chain-of-thought token by token,
dgemresolves up to 8 decision slots simultaneously in a single discrete diffusion denoising pass (~135β265 mswarm on GPU). - Intrinsic Epistemic Hesitation (): Because every slot is scored via restricted softmax over its exact allowed vocabulary ,
dgememits an exact probability distribution and Cardinality-Normalized Shannon Entropy () on every read. - Selective System-2 Escalation: Easy, low-entropy decisions () finish immediately in Stage 1. Ambiguous or out-of-distribution inputs () automatically escalate to a Stage-2 Gemini model (
gemini-3.8-flashorgemini-3.5-flash-lite) or a human review queue.
1. Where dgem Fits in Your Stack
Section titled β1. Where dgem Fits in Your StackβIn a two-stage decision architecture, dgem serves as the System-1 reflex layer in front of heavier System-2 reasoning models or human-in-the-loop workflows:
flowchart LR
In["Incoming Document / State / Image"] --> Tmpl["Policy-as-Template (.json.tmpl)<br/>document_first Layout"]
Tmpl --> Stage1["Stage 1: dgem Readout<br/>Single-Pass Restricted Softmax"]
Stage1 --> Gate{"Normalized Hesitation<br/>H_norm >= tau (0.16)?"}
Gate -- "No (Clear: < 16%)" --> Fast["Return Stage 1 Decision<br/>(~135-265 ms on GPU)"]
Gate -- "Yes (Uncertain: >= 16%)" --> Stage2["Stage 2: Gemini Cascade<br/>(gemini-3.8-flash / gemini-3.5-flash-lite)<br/>or Human Review Interrupt"]
Stage2 --> Out["Audited Final Decision<br/>+ Full Probability & Trace Receipt"]| Architectural Property | Autoregressive LLM Prompting | Traditional Fine-Tuned Classifier | dgem Decision Model (Stage 1 + Cascade) |
|---|---|---|---|
| Policy Adaptability | Zero-shot (prompt edit) | Requires retraining & redeploying | Zero-shot (.json.tmpl edit) |
| Multi-Question Latency | Scales with output length & reasoning trace | Fast, single-head | Single forward pass for up to 8 slots |
| Output Contract | Can fail JSON formatting or hallucinate keys | Fixed class index | Restricted softmax over exact schema options |
| Uncertainty Signal | Verbalized confidence is poorly calibrated | Needs per-head temperature fitting | Normalized entropy + optional calibration |
| Stage-2 Escalation | Runs expensive model on 100% of traffic | Ad-hoc threshold per head | Principled hesitation gate () |
2. End-to-End Request Lifecycle
Section titled β2. End-to-End Request LifecycleβEvery dgem decision follows a deterministic five-step pipeline across the CLI (dgem decide), HTTP Gateway (POST /api/decide, POST /v1/systemone), MCP Server (dgem mcp), and Web Studio:
- Template Compilation (
pkg/template): The.json.tmplpolicy renders caller variables into a JSON payload containing the inputstateand an array ofquestions. - Prompt Layout Construction (
document_first): By default (DEFAULT_LAYOUT=document_first), the serving image places the input document/state first and the question schema second inside the user turn so every question attends to the document from the start of the prompt (Prompt layout). - Single-Pass Restricted Softmax (
deploy/cloudrun/server/structured_server.py): For each question with option vocabulary (mapped to single-token option labelsAβZ, or up to 52 labels with opt-inaz_aa/az_lower), the model extracts the slot logits and computes restricted-softmax probabilities with optional temperature scaling : - Cardinality-Normalized Hesitation (): Raw Shannon entropy grows with the number of options . Dividing by yields a scale-free hesitation score in :
In the Web Studio and CLI (
studio/src/hesitation.ts), is reported as Clear, as Somewhat Unsure, and as Very Unsure (Confidence beyond Shannon). - Post-Processing & Escalation (
cmd/serve.go,pkg/decisionindex): Whencascade_mode: "entropy"is enabled and a slot exceedscascade_threshold,dgemopens an OpenTelemetry child span (dgem.cascade.gemini) and escalates that question togemini-3.8-flashorgemini-3.5-flash-lite(Authoring and Stage 2 cascades).
3. What Works in Production (Graduated Patterns)
Section titled β3. What Works in Production (Graduated Patterns)βEach feature below is validated against dgemβs regression matrix (Regression matrix) and exposed with consistent semantics across the CLI, HTTP Gateway, MCP Server, and Studio.
3.1 Hesitation-Gated Cascades () and Temperature Scaling ()
Section titled β3.1 Hesitation-Gated Cascades (H~mβ₯0.16\tilde{H}_m \ge 0.16H~mββ₯0.16) and Temperature Scaling (TβT^*Tβ)βRaw restricted-softmax probabilities provide a strong zero-shot ranking signal for separating confident answers from errors.
- Selective Escalation: Gate Stage-2 calls (
gemini-3.8-flashorgemini-3.5-flash-lite) or human-approval interrupts (examples/genkit-tool-approval/) on normalized entropy (or raw entropy ). - Per-Policy Calibration: When downstream consumers need probabilities that match empirical accuracy (low ECE / Brier score), fit a post-hoc temperature (
temperatureortemperature_by_typein your.json.tmpl) on a held-out labelled split usingscripts/policy_calibration.py(Calibrate your policy).
3.2 Document-First Prompt Layout (document_first)
Section titled β3.2 Document-First Prompt Layout (document_first)βPlacing the document/state before the question definitions (layout: "document_first", the default in the serving image and dgem systemone serve) improves accuracy on long-context Grounded NLI, hallucination detection, and catch-all intent suites without adding latency (Prompt layout).
3.3 Multi-Slot Canvas Batching ( Questions) and Conditional DAGs
Section titled β3.3 Multi-Slot Canvas Batching (β€8\le 8β€8 Questions) and Conditional DAGsβ- Independent Questions: Group up to 8 related questions (
MaxSlotsPerPass = 8) into a single.json.tmplpolicy (for example,templates/support_triage.json.tmplortemplates/github_spam_triage.json.tmpl). All slots are answered in one forward pass. Beyond 8 questions,pkg/decisionindexautomatically partitions questions into multi-slot batches. - Sequential Dependencies (
depends_on/ask_if): When a follow-up question only makes sense given an earlier answer (for example, asking for severity only whenis_incident == "yes"intemplates/secops_conditional_dag.json.tmpl), declaredepends_onandask_if: ["yes"]. Leave them out when a single pass suffices.
3.4 Wide-Option Taxonomies ( Options)
Section titled β3.4 Wide-Option Taxonomies (K>26K > 26K>26 Options)βThe serving image maps choice options to single letters AβZ by default (MaxOptionsPerSlot = 26). When a taxonomy has more than 26 options (such as 77 banking intents or 151 cross-domain intents), dgem offers three graduated strategies:
| Strategy | Flag / Option | Passes | Best For |
|---|---|---|---|
| 1. Two-Stage Bracket Tournament (Default) | --bracket-size 20 (dgem systemone serve) | Maximum zero-shot accuracy up to options with no external retriever required. | |
| 2. Extended 52-Option Label Sets (Experimental) | --labels az_aa or az_lower | (up to ) | Medium-wide taxonomies () when running against a serving image with label-set support. |
| 3. Pre-Filtered Shortlist Routing (Opt-in) | --shortlist-topk M (e.g. M=15) + prior_logits | Latency-critical wide choices (, measured up to ) when an upstream bi-encoder or lexical ranker supplies prior_logits. |
- Catch-All Handling (
--catch-all): On wide taxonomies that include an"out_of_scope"or"none of the listed"option,--catch-all finalholds the catch-all out of Round-1 brackets and injects it into the final round, whileShortlistFuncautomatically preserves catch-all keys alongside the top- shortlist. - Cosine Prior Normalization (
NormalizeCosinePriorLogits): Raw bi-encoder cosine similarities in occupy a narrow numeric range. Before passing embedding scores asprior_logits, convert them to temperature-scaled log-softmax logits withdecisionindex.NormalizeCosinePriorLogits(cosines, 0.07)so they live on a proper log-probability simplex while preserving exact rank order.
3.5 Opt-In Pre-Gated Prior Fusion (--prior-weight, --prior-gate-mode)
Section titled β3.5 Opt-In Pre-Gated Prior Fusion (--prior-weight, --prior-gate-mode)βWhen callers supply per-option log-priors (prior_logits on a /v1/systemone question, such as historical class base rates or retrieval log-probabilities), dgem can combine them with the modelβs log-probabilities via log-linear Product-of-Experts fusion (pkg/decisionindex/engine.go):
- Pre-Fusion Hesitation Invariant (
raw_normalized_entropy):FusePriorWithGateModealways computes and recordsraw_entropyandraw_normalized_entropy() before applyingprior_logits. Downstream Stage-2 cascades must gate onraw_normalized_entropyso a sharp external prior never masks model uncertainty. - Choosing a Gate Mode (
--prior-gate-mode):hesitant(Recommended for tie-breaking): Fusesprior_logitsonly when the model is uncertain (, default ), leaving confident model predictions untouched even if the prior disagrees.proportional: Scales the effective prior weight by , smoothly increasing prior influence as model hesitation rises.confident: Fuses only when .
3.6 Co-Denoised Evidence Slots (evidence_prompt + evidence_criteria)
Section titled β3.6 Co-Denoised Evidence Slots (evidence_prompt + evidence_criteria)βFor fine-grained visual or textual verification on standard choice questions (), a question on /v1/systemone can include evidence_prompt and caller-supplied evidence_criteria ( options).
- The adapter places a companion
<question_id>_evidencechoice slot immediately ahead of<question_id>on the same diffusion canvas (consuming 2 of the 8 slots in that pass) and returns the selected cue inanswers[q].evidence. - For multi-image visual inspection (
templates/multimodal/contact_sheet_verify.json.tmpl), tiling candidate crops into a numbered contact sheet and asking an evidence slot to identify the matching tile before the final decision slot resolves the comparison in a single forward pass.
3.7 Guided Object Localization (dgem locate)
Section titled β3.7 Guided Object Localization (dgem locate)βdgem is a discrete choice and scoring engine, not a continuous coordinate regressor. For image localization (dgem locate, POST /api/locate, MCP locate_object; see What dgem can do with images):
- Stage 1 (
dgem): Answers target presence and coarse 3Γ3 grid cell in one pass. With--skip-h 0.16, images wheredgemis confidently"absent"return immediately (path: "skipped_absent"), saving the Stage-2 call. - Stage 2 (
gemini-3.8-flash/gemini-3.5-flash-lite): Predicts continuousbox_1000/box_pctcoordinates atlowthinking. - Optional Stage 3 (
DGEM_SAM_URL): Refines the bounding box into a pixel mask and polygon.
4. What Does Not Work (Anti-Patterns to Avoid)
Section titled β4. What Does Not Work (Anti-Patterns to Avoid)βEmpirical testing across the dgem evaluation suites ruled out several tempting heuristics. Avoid these patterns in production:
| Anti-Pattern | What Happens | What to Do Instead |
|---|---|---|
Direct continuous [0..1000] coordinate regression in dgem choice slots | Quantizing image coordinates into 16 or 26 discrete bins yields coarse boxes and quantization rounding error. | Use dgem locate (dgem presence + 3Γ3 cell gate gemini-3.8-flash bounding box optional SAM mask). |
Gating Stage-2 cascades on post-fusion normalized_entropy when prior_logits are fused | A strong but wrong external prior depresses post-fusion entropy on errors, collapsing error-detection AUROC. | Always gate cascades on pre-fusion raw_normalized_entropy and prefer --prior-gate-mode hesitant or proportional. |
| Overpacking independent questions into one raw pass | Exceeding the 256-token output canvas degrades slot formatting and per-question accuracy. | Keep policies to questions per pass, or route through POST /v1/systemone (MaxSlotsPerPass = 8 auto-batching). |
Enabling --shortlist-contrastive by default on compact snake_case intent keys | Appending [distinguishing: ...] sibling token tags helps verbose overlapping product descriptions, but hurts compact intent names (e.g. banking77). | Keep --shortlist-contrastive=false (the default) unless A/B tested on verbose overlapping catalog descriptions. |
| Replacing balanced bracket tournaments with 2-level coarse-to-fine taxonomy slot tries | A Pass-1 coarse super-category routing error is unrecoverable in Pass 2 (because Pass 2 only sees the chosen child branch), and broad super-categories flatten out-of-scope entropy. | Use dgem systemone serve balanced 20-way bracket tournaments (--bracket-size 20 --catch-all final) or --shortlist-topk M with normalized embedding prior_logits. |
Enabling --null-prior-debias or --dual-mirror in production | Content-free positional de-biasing helps one suite while hurting another; dual-mirror doubles slot consumption with little gain over . | Keep both flags off in production; use neutral question IDs and document_first layout (Confidence and calibration). |
Using bias-inducing question IDs (mirror, reverse, opposite, alt) | Question IDs are part of the prompt seen by the model; suggestive names shift the answer distribution. | Use neutral, descriptive snake_case IDs (intent, urgency, policy_violation). |
5. Choosing Your Deployment Topology
Section titled β5. Choosing Your Deployment Topologyβdgem supports four execution topologies behind a single CLI, Gateway, and MCP interface (From laptop to production):
flowchart TD
Start{"What is your workload?"}
Start -- "Local Dev / Policy Authoring" --> Metal["1. Laptop (diffgemma on Apple Silicon)<br/>dgem serve --local"]
Start -- "Bursty / Scale-to-Zero / Batch Jobs" --> CR["2. Serverless Cloud Run GPU<br/>(RTX Pro 6000 / L4)<br/>deploy/batch/ for Cloud Run Jobs & Dataflow"]
Start -- "Low-Latency 24/7 Production Traffic" --> VAI["3. Vertex AI Dedicated Endpoint (G4)<br/>Public /invoke/v1/* or Private PSC :rawPredict"]
VAI --> GW["4. High-Availability Gateway (dgem serve)<br/>--default-backend vertex_first<br/>Warm Vertex primary + Cloud Run GPU failover"]
CR --> GW| Topology | Typical Warm Latency | Cold Start | Scaling & Cost Profile | Guide |
|---|---|---|---|---|
1. Apple Silicon (diffgemma) | ~250β600 ms (M-series Max/Ultra) | Instant | Free local iteration (32 GB+ unified memory) | Run on your laptop |
2. Cloud Run GPU (dgemma) | ~185β265 ms (1β3 questions) | ~2β3 min from zero | Scales 0..N per-second GPU billing; ideal for bursty traffic & deploy/batch/ | Deploy on Cloud Run Β· Batch guide |
3. Vertex AI Dedicated (G4) | ~135β185 ms (1β3 questions) | 0 ms (always warm) | Dedicated 24/7 GPU node(s); supports public /invoke/v1/* and private PSC :rawPredict | Production on Vertex AI Β· Vertex vs. Cloud Run |
4. Gateway (vertex_first) | ~135β185 ms | Masked by failover | Routes to warm Vertex AI primary with automatic Cloud Run GPU failover & Stage-2 Gemini cascade | Gateway and routing |
6. Production Launch Checklist
Section titled β6. Production Launch ChecklistβBefore promoting a new .json.tmpl policy or serving configuration to production:
- Schema & Layout Hygiene:
- Keep
choicequestions to options per pass (or configure/v1/systemonetournament brackets /--shortlist-topk/--labels az_aa). - Use neutral question IDs,
samples: {{ default 1 .samples | toJson }}, anddocument_firstlayout.
- Keep
- Calibration & Escalation:
- Evaluate your policy on a labelled dataset (Run a dataset) and inspect hesitation error AUROC.
- Configure Stage-2 escalation (
--cascade-model gemini-3.8-flashorgemini-3.5-flash-lite) or Genkit tool-approval interrupts (examples/genkit-tool-approval/) on (orraw_normalized_entropywhenprior_logitsfusion is enabled).
- Security & Credentials:
- Protect the serving image with
DGEM_SERVER_KEY(X-DGem-Key) and the HTTP gateway with Cloud Run IAP (--iap-audience) or--api-keys(X-DGem-Api-Key) (Security and access).
- Protect the serving image with
- Regression Gate:
- Run T0/T1 verification via
scripts/bench_matrix.pybefore switching traffic (Operations runbook Β· Regression matrix).
- Run T0/T1 verification via