Shadow Evals¶

Shadow Evals replay a past conversation against the current version of your agent. The simulated caller streams the original recorded caller audio turn by turn, so the agent hears the same audio (accents, pauses, background noise) as it did the first time. That lets you check a new agent version against real calls, not just scripted or synthesized ones.

When the new agent goes off script (asks something new, or asks in a different order), a Gemini-driven simulated caller decides each turn whether to:

• replay a recorded caller turn (use_past_audio), which is the default, or
• synthesize a new reply with TTS (generate_tts), only when no recording contains the needed information.

After the replay, Gemini judges the new conversation against your natural-language expectations.

Prerequisites¶

Check these before writing a Shadow Eval. ShadowEvals.preflight (run automatically, see Preflight checks) checks most of them for you.

1. The original call was recorded. The source app must have had audio recording (loggingSettings.audioRecordingConfig) enabled when the call happened. Recordings are stored as gs://<bucket>/<project>/<location>/<app>/<date>/<conversation_id>/user-turn-N.wav.
2. You can read the recording bucket. Your credentials need storage.objects.get / list on it.
3. You know the session state the original call ran with. If the original session relied on seeded session variables or tool fakes (for example a mocked caller or account lookup keyed on a session variable), set the same session_parameters / use_tool_fakes. If you don't, the agent can take a different path from turn one, and the replay can't follow the original call.

YAML format¶

Put Shadow Eval files under evals/shadows/ (subdirectories are fine; .yaml and .yml are both picked up):

config:                      # defaults inherited by every case below
  project_id: my-project     # where the *past* conversation lives
  location: us
  app_id: my-app
  gcs_bucket: gs://my-recordings-bucket
  use_tool_fakes: true       # run with the same tool fakes as the original
  session_parameters:
    customer_tier: gold      # same seeded state as the original session
  max_turns: 15

shadow_evals:
  - name: order_never_arrived
    conversation_id: 7f3c9a52-...        # the past conversation to replay
    replay_mode: hybrid                  # or "exact"
    tags: [P0]
    expectations:
      - "The agent recognizes the returning customer without asking."
      - "The agent finds the order and explains the delivery status."
      - "The agent offers a replacement or refund."

Case fields¶

Everything in config: except session_parameters is a default that a case can override; session_parameters are merged, with case values winning.

Replay modes¶

Running Shadow Evals¶

CLI¶

cxas evals report --run \
  --app-name projects/my-project/locations/us/apps/my-app \
  --include shadows --modality audio \
  --output-dir eval_results/

Shadow Evals always run in audio modality. With --output-dir, each run's audio and a side-by-side HTML report are written under eval_results/shadow_artifacts/<case>_run<N>/ (with a timestamp suffix on shadow_artifacts when timestamped output is enabled).

Python¶

from cxas_scrapi.evals import ShadowEvals

shadow = ShadowEvals(app_name="projects/my-project/locations/us/apps/my-app")
cases = shadow.load_shadow_test_cases_from_file("evals/shadows/orders.yaml")

results = shadow.run_shadow_evals(
    cases,
    modality="audio",
    artifacts_dir="shadow_artifacts/",  # keep audio + HTML report per run
)
for r in results:
    print(r["name"], r["passed"], r["expectations"],
          "past audio:", r.get("past_audio_turns_used"),
          "tts:", r.get("tts_turns_generated"),
          r.get("report_path"))

For a single case, run_shadow_conversation(test_case, artifacts_dir=...) returns the ShadowUserConversation; conversation.generate_report() gives per-turn decision, goal and expectation DataFrames, and conversation.report_path points to the HTML report.

Preflight checks¶

run_shadow_evals calls ShadowEvals.preflight for every case before opening any session (preflight=False turns this off). You can also call it on its own:

check = shadow.preflight(cases[0])
print(check.ok, check.past_user_turns, check.past_user_turns_with_audio)
for issue in check.issues:
    print(issue.level, issue.message)

Cases with an error are reported as failed with error: "Preflight failed: ..." and are not run. Other issues are printed and attached to the result as preflight_issues.

Side-by-side report and audio artifacts¶

Downloaded and captured audio normally lives in a temporary directory that is deleted after each run. Pass artifacts_dir (or --output-dir on the CLI) to keep it, which makes it possible to audit the replay:

<artifacts_dir>/<case>_run1/
  shadow_report.html          # past call vs. new call, turn by turn, with audio
  shadow_result.json          # the same data, machine-readable
  audio/past_user_turn_N.wav  # recorded caller audio (exactly what was streamed)
  audio/past_agent_turn_N.wav # the original agent reply
  audio/new_agent_turn_N.wav  # the new agent reply

Each row of the report pairs a new turn with the past turn it replayed. You can play the recorded caller audio, the original agent reply and the new agent reply next to each other, alongside the simulated caller's decision and reasoning. Recorded turns that were never replayed are listed at the end. Agent audio is captured automatically when artifacts_dir is set. The directory uses relative paths, so it can be zipped or served statically as is.

Troubleshooting¶