Local Simulations¶

Local Simulations take a different approach to testing than Platform Goldens. Instead of scripting exact conversations and expected responses, you describe a goal and let an AI-powered user simulator (Gemini) try to achieve it. At the end, Gemini judges whether the agent met the goal and any additional expectations you specified.

This is valuable when you want to test that an agent can complete a task, without caring about the exact phrasing of each response — which is especially important for voice agents where natural language variation is expected.

How simulations work¶

1. SCRAPI starts a real session with your agent using the Sessions API
2. Gemini plays the role of a human user, sending messages to the agent to try to achieve the goal
3. The conversation continues until the goal is met, the max number of turns is reached, or the agent ends the session
4. Gemini evaluates whether each step's success_criteria was met and whether any expectations were satisfied
5. SCRAPI produces a report with pass/fail status for each step and expectation

Because the user is simulated by a language model, the conversation is non-deterministic — each run may produce slightly different messages. This mirrors how real users behave.

YAML format¶

Simulation files use the evals: key at the top level:

evals:
  - name: "successful_order_lookup"
    tags: ["P0", "order_management"]
    session_parameters:
      order_12345_status: "shipped"
      order_12345_eta: "2026-04-18"
    steps:
      - goal: "Ask about the status of order ORD-12345"
        success_criteria: "The user has provided order ID ORD-12345 and the agent has acknowledged it"
        response_guide: "The user is a customer checking on a recent purchase. They are polite but want a quick answer."
        max_turns: 3

      - goal: "Get the order status and delivery date"
        success_criteria: "The agent has provided the shipping status and the estimated delivery date"
        max_turns: 2

    expectations:
      - "The agent correctly identified the order as shipped"
      - "The agent mentioned the estimated delivery date"
      - "The agent maintained a friendly, helpful tone throughout"

Top-level fields¶

Step fields¶

Expectations¶

Expectations are evaluated by Gemini after the full conversation completes, looking at the entire transcript. They're natural language assertions:

expectations:
  - "The agent never made up information that wasn't in the tool response"
  - "The agent asked for the order ID before looking it up"
  - "The agent offered to help with anything else before ending"

Each expectation is judged as Met or Not Met, with a justification from Gemini.

The SimulationEvals class¶

For programmatic use, import SimulationEvals:

from cxas_scrapi.evals.simulation_evals import SimulationEvals

sim_evals = SimulationEvals(
    app_name="projects/my-project/locations/us/apps/my-app",
)

Running a single evaluation programmatically¶

The simulate_conversation method takes a test_case dict defining the steps and expectations:

from cxas_scrapi.evals.simulation_evals import SimulationEvals

sim_evals = SimulationEvals(app_name="projects/my-project/locations/us/apps/my-app")

test_case = {
    "steps": [
        {
            "goal": "Ask about order ORD-12345",
            "success_criteria": "User provided order ID and agent acknowledged",
            "max_turns": 3,
        },
        {
            "goal": "Get delivery date",
            "success_criteria": "Agent provided estimated delivery date",
            "max_turns": 2,
        },
    ],
    "expectations": [
        "Agent maintained professional tone",
        "Agent never hallucinated data",
    ],
}

eval_conv = sim_evals.simulate_conversation(test_case=test_case)
report = eval_conv.generate_report()

# Goals report (one row per step)
print(report.goals_df)

# Expectations report (one row per expectation)
if report.expectations_df is not None:
    print(report.expectations_df)

Running in parallel¶

Simulations can be slow because they involve multiple real API calls. Run them in parallel to speed things up:

import concurrent.futures

test_cases = [...]  # list of test_case dicts

def run_single(tc):
    eval_conv = sim_evals.simulate_conversation(test_case=tc, console_logging=False)
    return eval_conv.generate_report()

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(run_single, tc) for tc in test_cases]
    reports = [f.result() for f in concurrent.futures.as_completed(futures)]

Audio modality¶

If your agent handles voice conversations, you can run simulations in audio mode:

sim_evals = SimulationEvals(app_name="projects/my-project/locations/us/apps/my-app")

eval_conv = sim_evals.simulate_conversation(
    test_case=test_case,
    modality="audio",  # default is "text"
    # voice_config is optional (defaults to US English voice)
    voice_config={
        "language_code": "en-US",
        "voice_name": "en-US-Standard-A"
    }
)

In audio mode, SCRAPI uses the Sessions API's audio streaming endpoint. The simulated user's messages are still text internally, but they're processed by the agent's audio pipeline, which exercises TTS/STT and any audio-specific callbacks.

Naturalness Metric¶

Goals and expectations judge what the agent did. The Naturalness Metric judges how it said it: Gemini grades every agent turn on how closely it resembles a warm, competent human contact-centre agent rather than a script-reading bot.

Use it for voice and GECX / Gemini Composite agents, where emotive expression, pacing, disfluency, emotive tags ([warm], [short pause], [sigh]), and spoken-form numbers matter as much as task completion. A turn can complete every goal and still sound robotic — this metric is what catches that.

[!NOTE] The metric is entirely opt-in. A test case that does not declare a naturalness_metric key behaves exactly as it did before: no extra Gemini call is made, and no extra keys appear in the results. Existing simulation YAML needs no changes, and a YAML file written against a newer version still loads (unknown keys inside the block are tolerated, not rejected).

Minimal example¶

The shorthand true enables the metric with all defaults:

evals:
  - name: upset_caller_billing_dispute
    tags: [P0, voice, naturalness]
    naturalness_metric: true
    steps:
      - goal: Complain about an incorrect charge and get it refunded
        success_criteria: Agent apologizes, looks up the account, and issues a refund
        max_turns: 10

Use naturalness_metric: false to explicitly disable it. The keys naturalness and naturalness_config are accepted as aliases for naturalness_metric.

Fully-configured example¶

evals:
  - name: upset_caller_billing_dispute
    tags: [P0, voice, naturalness]
    naturalness_metric:
      enabled: true
      model: "gemini-3.1-pro-preview"
      turn_qualities: [emotion, pacing, grammarStyle, disfluency, spokenNumbers]
      conversation_qualities: [personaConsistency, emotionalArcTracking]
      extra_guidance: >
        This is a US English voice agent. Currency must be spoken as
        "a hundred and fifty dollars", never "USD 150". Keep the calm
        register for the whole call once the caller has been upset.
      turn_weight: 0.8
      bot_like_below: 2.5
      human_like_at_or_above: 4.0
      pass_threshold: 3.5
      use_audio: true
      audio_source: auto
      latency_bands_ms: [1000, 2000, 3000, 4000]
      latency_weight: 3.0
      latency_failure_threshold_ms: 4000
      include_tool_calls: true
    steps:
      - goal: Complain about an incorrect charge and get it refunded
        success_criteria: Agent apologizes, looks up the account, and issues a refund
        max_turns: 10

Configuration fields¶

Default turn-level qualities¶

Every agent turn is scored 1–5 on each of these:

Default conversation-level qualities¶

These are scored once across the whole call:

Perceived latency¶

Dead air is the fastest way for an agent to stop sounding human, and it is the one dimension you cannot hear in a transcript. perceivedLatency is therefore measured, not judged: SCRAPI reads the platform's reported perceived latency for each turn from the conversation trace after the run and scores it against latency_bands_ms. No extra model call is involved, and any perceivedLatency factor the grader tries to invent is discarded.

Because a turn carries roughly ten judged factors, an unweighted latency factor would move the turn score by about 0.2 no matter how slow the agent was. latency_weight (default 3.0) makes latency count for three factors instead of one. Set it to 0 to keep latency in the report without letting it affect any score.

Turns with no reported latency get no factor at all, rather than a neutral 3, so a missing measurement cannot quietly drag a turn up or down. The platform publishes a conversation some time after the call ends — usually under a minute — so each simulation waits for it, polling until it appears or latency_fetch_timeout_s is spent. If it never arrives, the dimension is skipped and the rest of the metric still runs.

[!TIP] That wait is real wall-clock time per simulation. If you are running a large suite and only want the judged dimensions, set latency_fetch_timeout_s: 0.

[!WARNING] latency_failure_threshold_ms defaults to 4000, so a single turn at 4s or slower fails the simulation outright, independently of pass_threshold and of whether every goal was met. Set it to null if you only want latency reported.

Judging the audio, not the transcript¶

With use_audio, the agent's recorded audio for each turn is attached to the grading call so pacing, emotion, and disfluency are judged from what the caller actually heard. An inline tag such as [warm] or [short pause] is then only credited if it is audible — which is what catches tags that the agent read aloud verbatim or silently dropped.

This is enabled automatically when the simulation runs in audio modality; you do not need to set it. Audio is read from the platform's recording bucket by default and never downloaded. audio_source controls where it comes from:

• auto (default) — the recording bucket, falling back to WAVs captured locally during the run
• gcs — the recording bucket only
• local — locally captured WAVs only, which requires capture_agent_audio=True
• none — disable audio even when use_audio is set

If no audio can be attached, grading degrades to the transcript rather than failing.

[!NOTE] Audio grading needs the app's evaluation audio recording bucket configured. Recordings are named per platform turn (agent-turn-1.wav is simulation turn 0), which is how each clip is matched to the turn it belongs to.

How labels and scores are computed¶

Each factor score is an integer clamped to 1–5. Turn and overall scores are then derived by SCRAPI (not taken from the model's own holistic number), which keeps the scores, averages, and labels mutually consistent:

1. Turn score = weighted mean of that turn's factor scores, rounded to two decimals. Every judged factor has weight 1; perceivedLatency has weight latency_weight. (If the model returned no factors for a turn, its own score is used instead.)
2. Turn label comes from the score bands:
3. score < bot_like_below → Bot-like (default: below 2.5)
4. score >= human_like_at_or_above → Human-like (default: 3.75 and up)
5. otherwise → Transitional
6. Overall score:

overall_score = turn_weight * mean(turn scores)
              + (1 - turn_weight) * mean(conversation factor scores)

If the grader returned no conversation-level factors, overall_score is simply the mean of the turn scores. The result is clamped to 1–5 and rounded to two decimals, where 5 means fully human-like.

1. Overall label is derived from overall_score using the same bands.

Pass/fail behaviour¶

By default the score is purely informational — it reports a score and a label but never changes whether the simulation passed.

Setting pass_threshold opts the test case into enforcement: the simulation fails if overall_score is below the threshold, even when every goal and expectation was met.

naturalness_metric:
  pass_threshold: 3.5  # fail the sim if the agent scores below 3.5/5

Separately, a turn at or above latency_failure_threshold_ms fails the run on its own. This is deliberately independent of the score: an agent can be charming and still be unusable if it leaves the caller in silence. The reason appears in the console line and the HTML report, and in naturalness_details.failure_reasons.

naturalness_metric:
  latency_failure_threshold_ms: null  # report slow turns, never fail on them

Enabling it for a whole run¶

You can turn the metric on without editing any YAML, using the naturalness argument on the constructor, run_simulations(), or simulate_conversation():

# Enable with defaults for every test case that doesn't declare its own block
sim_evals = SimulationEvals(app_name=app_name, naturalness=True)

# Same, at call time
results = sim_evals.run_simulations(test_cases=test_cases, naturalness=True)

# A dict is merged OVER whatever the test case declared
eval_conv = sim_evals.simulate_conversation(
    test_case=test_case,
    naturalness={"pass_threshold": 4.0, "model": "gemini-3.1-pro-preview"},
)

• None (the default) defers entirely to each test case.
• True enables the metric with defaults for test cases that don't declare the key. A test case with an explicit naturalness_metric: false stays off.
• False force-disables the metric everywhere, even where a test case asked for it.
• A dict is merged over the test case's own block, so run-level keys win.

The CLI exposes the same switch on evals report:

# Grade every simulation, even ones that don't declare the metric
cxas evals report --run --include sims --naturalness

# Skip grading entirely, even for test cases that do declare it
cxas evals report --run --include sims --no-naturalness

# Omit the flag to let each test case decide
cxas evals report --run --include sims

The flag is tri-state, mirroring the naturalness argument: --naturalness is True, --no-naturalness is False, and omitting it is None.

[!TIP] --no-naturalness is the hill-climbing switch. Iterate on correctness first with grading off — it skips the extra Gemini call per run and stops any pass_threshold from failing a simulation that is functionally correct — then drop the flag to evaluate naturalness once the task success rate is where you want it.

What you get in the results¶

When the metric ran, each entry in sim_results.json gains three keys (they are absent when it did not run, so existing consumers see an unchanged payload):

The console progress line also gains a | naturalness: 4.1/5 (Human-like) suffix, plus | FAILED: ... when a turn broke the perceived-latency limit.

Programmatically, the result object lives on the conversation and feeds the report:

eval_conv = sim_evals.simulate_conversation(test_case=test_case, naturalness=True)

result = eval_conv.naturalness_result  # None if the metric didn't run
if result:
    print(result.overall_score, result.overall_label.value)
    print(result.factor_averages)  # mean score per quality across turns
    print(result.summary)

report = eval_conv.generate_report()
if report.naturalness_df is not None:
    print(report.naturalness_headline)  # e.g. "Overall: 4.1/5 (Human-like)"
    print(report.naturalness_df)

Cost¶

Grading adds one extra Gemini call per simulation run (per test case, per repeat), sending the whole transcript. It's cheap relative to the simulation itself, but it is not free — reserve it for the evals where sounding human actually matters. In audio modality the agent's recordings are attached to that call as well, which makes it larger; set audio_source: none to grade the transcript only. perceivedLatency costs nothing extra: it is read from the conversation trace, not judged.

Interpreting results¶

The SimulationReport object has two DataFrames, plus an optional third when the Naturalness Metric ran:

goals_df¶

expectations_df¶

naturalness_df¶

Present only when the Naturalness Metric ran — otherwise report.naturalness_df is None. One row per graded agent turn:

report.naturalness_headline carries the summary line, e.g. Overall: 4.1/5 (Human-like), with | threshold 3.5 -> PASS appended when pass_threshold is set and | FAILED: ... when a turn broke the perceived-latency limit.

Reading the output¶

# Overall pass rate
total = len(report.goals_df)
passed = (report.goals_df["status"] == "Completed").sum()
print(f"Steps completed: {passed}/{total} ({passed/total*100:.0f}%)")

# Failed steps
failed = report.goals_df[report.goals_df["status"] != "Completed"]
for _, row in failed.iterrows():
    print(f"FAILED: {row['goal']}")
    print(f"  Reason: {row['justification']}")

Using Tool Fakes¶

By default, simulations execute real tool calls (including webhooks or database queries) when the agent invokes a tool. During testing, you can enable Tool Fakes to tell the platform to return pre-defined mock/fake responses instead of running the actual tool backend.

[!NOTE] Tool fakes (mock tool responses) are defined per-tool within the Agent Studio Console. See the Mock Tool Responses guide for how to configure these in the console and write the mock handler scripts.

To enable tool fakes in your simulations:

Programmatically¶

Pass use_tool_fakes=True when calling simulate_conversation or run_simulations:

eval_conv = sim_evals.simulate_conversation(
    test_case=test_case,
    use_tool_fakes=True,  # Bypasses real tool backends
)

Via CLI¶

Run your simulations in bulk using the cxas evals report command with the --run and --include sims flags, specifying the agent app name, output directory, parallel workers, and the --use-tool-fakes flag:

cxas evals report \
    --run \
    --include sims \
    --app-name "projects/my-project/locations/us/apps/my-app" \
    --output-dir "eval-reports" \
    --sim-parallel 5 \
    --use-tool-fakes

Tips for writing good simulations¶

Keep steps focused

Each step should test one thing. Broad goals like "complete the full conversation" are hard to debug when they fail.

Write meaningful success criteria

"The agent helped the user" is too vague. "The agent provided the order status and delivery date" is testable.

Use response_guide to set tone

If your agent needs to handle impatient users or edge cases, use response_guide to set that context for the simulator.

Use static_utterance for exact inputs

When you want to test how the agent handles a specific phrasing (e.g., "what's my ETA?"), use static_utterance to send that exact text.

Use session parameters for mocking

Just like goldens, use session_parameters to inject mock tool responses so your simulations are deterministic and fast.