Skip to content

cxas test-callbacks

cxas test-callbacks discovers and runs pytest-based unit tests for every callback in your app directory, so you can catch bugs in your callback logic before deploying anything.

Usage

cxas test-callbacks --app-dir DIR
                    [--agent-name NAME]
                    [--callback-type TYPE]
                    [--callback-name NAME]
                    [--log-file PATH]
                    [--pytest-args ARGS]

Options

Option Required Default Description
--app-dir DIR Yes — Path to the app directory that contains the agents/ subdirectory. The CLI searches this directory for callbacks and their accompanying test files.
--agent-name NAME No — Narrow the test run to callbacks belonging to a specific agent (e.g., pilot). Skips all other agents.
--callback-type TYPE No — Only run callbacks of this type (e.g., before_call, after_call, webhook).
--callback-name NAME No — Run tests for a single callback by name.
--log-file PATH No — Path to a file where pytest's full output will be written. Useful for saving logs in CI.
--pytest-args ARGS No — Comma-separated list of extra arguments to pass directly to pytest (e.g., "-v,-s" for verbose output with live stdout).

How Discovery Works

The CLI delegates discovery to CallbackEvals.test_all_callbacks_in_app_dir. It walks the app directory, finds callback Python files, and looks for adjacent test files following your project's naming conventions. All discovered tests are executed with pytest.

Exit Codes

Code Meaning
0 All callback tests passed.
1 One or more callback tests failed, or no test files were found.

Examples

Run all callback tests in the app directory:

cxas test-callbacks --app-dir ./my-agent

Run only callbacks for the pilot agent:

cxas test-callbacks \
  --app-dir ./my-agent \
  --agent-name pilot

Run only before_call callbacks, with verbose output, and save logs:

cxas test-callbacks \
  --app-dir ./my-agent \
  --callback-type before_call \
  --log-file /tmp/callback-test.log \
  --pytest-args "-v,-s"

Narrow to a single callback by name:

cxas test-callbacks \
  --app-dir ./my-agent \
  --agent-name pilot \
  --callback-name validate_session