vertex-ai-samples
Multi-Turn Reinforcement Learning for τ2-bench
Authors: Fei Xia, Genquan Duan, Youbao Tang, Jingya Liu, Jiuqiang Tang, Xuehan Xiong
Table of Contents
Intro
This blog is the third installment of our blog series dedicated to model training best practices for Managed Training Cluster (MTC) customers. Building on the off-policy distillation methodology covered in the first installment, this article explores how reinforcement learning (RL) can further improve tool-calling agent capabilities through direct environment interaction and reward optimization.
Training tool-calling agents with RL on multi-turn tasks is heavily constrained by sparse outcome rewards and complex credit assignment across extended dialogues. In this blog, we leverage τ2-bench to evaluate agent capabilities across realistic retail, airline, and telecom customer service domains. Our training architecture employs the NeMo RL framework paired with the Group Relative Policy Optimization (GRPO) algorithm. In this setup, the policy model (agent) learns optimal dialogue and tool-utilization strategies by interacting with a dedicated user simulator model powered by separate LLM endpoints, while an automated verifier evaluates final task completion. To establish a strong baseline, we synthesized data using open-source models (GLM-4.7) to boost our Supervised Fine-Tuning (SFT) checkpoints from 65.5% to 70.2% on the τ2-bench evaluation dataset.
To support our MTC community in accelerating their own development, we release our complete synthetic datasets, codebase, and training recipes to enable reproducible RL pipelines.
Background
Multi-Turn Tool-Calling Agents
Multi-Turn Tool-Calling Agents are autonomous architectures that interact with external functions or APIs over extended, iterative dialogues to solve complex, multi-step tasks. Instead of generating a final answer in a single pass, these agents alternate between reasoning, executing a tool, processing the tool’s output, and planning their next move over several sequential rounds. At each turn \(t\), the agent maintains an internal state consisting of the initial user query \(q\), the hidden text history \(h_t\), and a list of all prior tool executions and results \(z_0, \dots, z_{t-1}\):
\[s_t = (q, h_t, z_0, \dots, z_{t-1})\]Using this state, the agent’s policy executes a classic Observation → Planning → Action loop:
- Planning: The agent decides whether it has enough information to answer the user or if it needs to invoke an external tool.
- Action (Tool Invocation): It generates a structured API call (e.g., JSON parameters) targeting a specific tool.
- Observation (Execution): The environment runs the API, captures the output, and appends it back into the agent’s context window as a new message turn.
- Iterate or Terminate: The loop repeats until the agent determines it has solved the problem and yields a final answer.
GRPO
Group Relative Policy Optimization (GRPO) normalizes rewards within groups of \(G\) rollouts per prompt. The Group Relative Advantage is calculated as \(A_i = \frac{R_i - \bar{R}}{\sigma_R}\), where:
- \(A_i\): The relative advantage of the \(i\)-th output in the group.
- \(R_i\): The absolute reward score given to the \(i\)-th output.
- \(\bar{R}\): The mean reward across all outputs in the sampled group (\(G\)): \(\bar{R} = \frac{1}{G} \sum_{j=1}^G R_j\)
- \(\sigma_R\): The standard deviation of the rewards within the group: \(\sigma_R = \sqrt{\frac{1}{G} \sum_{j=1}^G (R_j - \bar{R})^2}\)
We apply the decoupled clipped objective:
\[L^{\text{CLIP}}_{\text{decoupled}}(\theta) := \hat{\mathbb{E}}_t \left[ \frac{\pi_{\theta_{\text{prox}}}(a_t \mid s_t)}{\pi_{\theta_{\text{behav}}}(a_t \mid s_t)} \min \left( r_t(\theta)\hat{A}_t, \text{clip}\left(r_t(\theta), 1-\epsilon, 1+\epsilon\right)\hat{A}_t \right) \right]\]where \(\hat{A}_t\) is an estimator of the advantage at timestep \(t\), \(\hat{\mathbb{E}}_t[\dots]\) indicates the empirical average over a finite batch of timesteps \(t\), and the probability ratio \(r_t(\theta)\) is defined as \(r_t(\theta) := \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{prox}}}(a_t \mid s_t)}\).
τ2-bench
τ2-bench, developed by Sierra Research, is an open-source evaluation framework designed to test LLM-based autonomous agents in realistic customer service environments. While the original benchmark focused on agents working entirely on their own, τ2-bench introduces a shared action space where the AI agent and a simulated user must collaborate to solve problems. It tests agents across complex, multi-step tasks in industries like retail, airlines, telecom, and banking knowledge.
Reward
For any given task scenario, the overall reward for a completed interaction sequence is binary, \(R_{\text{episode}} \in \{0, 1\}\). To achieve a perfect reward of 1, the agent must simultaneously clear two distinct evaluation layers: State-Based Verification and Action-Based Verification:
\[R_{\text{episode}}=\mathbf{1}(\text{State Verified}) \times \mathbf{1}(\text{Actions Verified})\]State-Based Verification: The state of the environment is represented as a database state, \(S_{\text{db}}\). At the beginning of a task, the database is initialized to a specific state, \(S_{\text{db}}^{\text{init}}\). The user simulator interacts with the agent to achieve an underlying goal state. At the end of the conversation, the evaluation engine extracts the final database state, \(S_{\text{db}}^{\text{final}}\), and compares it against the pre-annotated ground-truth expected state, \(S_{\text{db}}^{\text{target}}\).
\[\mathbf{1}(\text{State Verified}) = \begin{cases} 1 & \text{if } S_{\text{db}}^{\text{final}} = S_{\text{db}}^{\text{target}} \\ 0 & \text{otherwise} \end{cases}\]This ensures that regardless of the exact phrasing or natural language drift during the conversation, the structural side-effects of the agent’s tool executions match the exact user intent.
Action-Based Verification: Even if the final database matches the target state, the agent must not violate organizational logic or safety guidelines along the way. The evaluation engine validates the trajectory’s sequence of actions against a set of constraints:
- Policy Adherence: The agent must respect conditional boundaries (e.g., checking user ID before pulling records or refusing to apply a discount if the user is ineligible).
- Structural Correctness: The agent cannot execute invalid combinations of tools, such as firing multiple database mutations in parallel when the system guidelines demand single, sequential turn boundaries.
Where \(\tau\) is the trajectory history and \(\mathcal{C}_{\text{policy}}\) maps an action to its validity given the policy document.
Metric
Because LLM-based agents are inherently stochastic, evaluating a task a single time can lead to misleading variance in performance numbers. The fundamental metric reported on the benchmark leaderboards is Pass1. It represents the expected success rate across the evaluation dataset when running exactly one trial per task scenario. Given a dataset of \(N\) unique task descriptions, Pass1 is computed as:
\[\text{Pass}^1 = \frac{1}{N} \sum_{i=1}^{N} R_{\text{episode}}^{(i)}\]We report Pass1 with 4 trials in the evaluation below.
Training Pipeline
Training Framework
Training Framework and System Architecture
We utilize NVIDIA NeMo RL as the primary training framework. We implement the τ2-bench sandbox environment inside NVIDIA NeMo Gym, which provides a unified interface for building and scaling reinforcement learning environments and is seamlessly integrated with the NeMo RL library for RL training runs.
 |
We train on τ2-bench, a customer-service simulation benchmark spanning the airline, retail, and telecom domains. Each task instantiates a tool-augmented dialogue between a policy agent (the model under training) and an LLM-driven user simulator, grounded in a domain policy document and a per-domain tool/API suite. An episode is a multi-turn loop; at each turn the agent either replies to the user in natural language or issues a tool call against the domain backend, and the environment advances the user-simulator state, returns tool results and the user’s next message. Rewards are produced by τ2’s built-in verifier against each task’s expected outcome, yielding the per-episode scalar that drives GRPO.
The system architecture deliberately partitions the workload across three execution domains—a CPU VM, a CPU cluster for environment execution, and a GPU cluster for the trainer/sampler—so that each scales independently and the GPUs stay saturated on the only work that needs them: training and generation. As shown in Figure 1, a single Driver Program on the CPU VM owns the training loop and hosts two cooperating components.
The first is the Training Service Client, which talks to the MTC Training Service on the GPU cluster and provisions two modules—a policy Trainer and a rollout Sampler—colocated to share GPUs or disaggregated for async workload. The client issues train / compute_logprobs calls to the Trainer and pulls generations from the Sampler, and after each update synchronizes policy weights Trainer→Sampler over a dedicated weights group so the next round of rollouts is on-policy.
The second component is the Rollout Proxy & Trajectory Manager. Rather than letting environment code call the Sampler directly, all generation is funneled through an OpenAI-compatible /chat/completions proxy that fronts the Sampler endpoint. This buys three things at once: (i) environment code stays a stock LLM client—the Episode Worker on the CPU cluster runs an unmodified τ2 AgentGymEnv and reaches the model through a standard LiteLLM/OpenAI client pointed at the proxy URL; and (ii) because every agent turn transits the proxy, the Trajectory Manager records token-faithful prompt/completion segments and logprobs as they are generated, so trajectories are reconstructed exactly for the GRPO update instead of being re-tokenized after the fact.
This separation is what lets the environment tier scale horizontally and independently of the GPUs. Environment execution runs as a fleet of Ray actors on the CPU cluster, fanned out by the EnvRolloutDispatcher across two pools—a train pool and an eval pool—pinned to their respective Ray workergroups with the τ2 data corpus baked into the worker image. Each step dispatches num_prompts × repeat_n episodes onto the train pool, all of them generating concurrently against the shared Sampler through the rollout proxy; the driver then filters failed and length-truncated trajectories, computes leave-one-out GRPO advantages within each prompt group, applies a clipped policy-gradient update on the Trainer, and syncs weights back to the Sampler before the next step. Evaluation runs periodically on the eval pool, and best-N checkpoint retention is keyed on the eval reward. The net effect is that slow, CPU-bound, highly parallel environment simulation is kept off the GPU critical path, while the GPU cluster does nothing but generate and train.
User Simulator
Unlike passive benchmarks where the user is merely a text prompt, τ2-bench introduces a dual-control architecture. The User Simulator functions as an active environment entity. To eliminate the chaotic hallucinations common in pure LLM simulations, τ2-bench tightly couples the user’s behavior to the actual underlying state machine. The user cannot magically fix a setting or misrepresent device states; they must be accurately guided by the RL agent’s communication policy, making coordination and explicit user-modeling a strict requirement for policy success. The user simulator endpoints use vLLM or SGLang with OpenAI-compatible formats.
Training Data Synthesis
To train our RL agent within τ2-bench’s dual-control environment, we developed an efficient data synthesis pipeline to produce high-quality training data for three customer-service domains: Telecom, Retail, and Airline. The pipeline uses an LLM to generate tasks, then iteratively refines and verifies them through multiple stages to ensure solvability and correctness, and finally converts the verified rollout results into training data.
 |
The pipeline (Figure 2) comprises the following stages:
- Task Generation: The process begins by prompting a large language model to generate a self-contained “Task Bundle”. Each bundle contains a simulated database state, a concrete user scenario, and a list of machine-verifiable evaluation criteria. To prevent the LLM from generating repetitive tasks, a unique diversity seed is constructed for each call by randomly sampling:
- User Profiles: Names, addresses, and contact info.
- Difficulty Levels: Controlling the expected length and complexity (Easy, Medium, Hard).
- Scenarios: Specific problems mapped from domain pools (e.g., billing disputes, cancellations, or connectivity issues).
- Task Refinement:
- Rollout Refinement (Crash Fixing): Every task runs once in a live simulator. Tasks that crash are captured, and their stack tracebacks are sent back to the LLM for automated repair up to 3 rounds.
- Ground-Truth (GT) Refinement (Solvability): A specialized “Golden Agent” with perfect knowledge of the correct resolution path attempts each task. If this expert agent cannot achieve a perfect reward (reward=1.0), the task’s database state or evaluation criteria are fundamentally misaligned and are sent back to the LLM to be repaired. If the expert fails to solve the task after 2 rounds, then the task is marked as failed to check ground truth.
-
Task Verification: The pipeline verifies each task across 16 independent, stochastic rollouts with standard agents. This stage calculates a statistical Pass Rate for each task to evaluate solvability: \(\text{Pass Rate} = \frac{\text{num pass}}{\text{num trials}}\). If a task is unsolvable by standard agents and has a 0% pass rate, then the task is marked as failed to check ground truth.
- Failure Refinement and Re-verify: Rather than discarding failed tasks entirely, the pipeline takes a “fix the test, not the code” approach. The LLM reviews the best recorded trajectory and only modifies evaluation criteria to make them solvable but still meaningful. Refined tasks are verified again and merged with previously verified results.
- Task Export: Generated tasks are categorized into difficulty buckets based on their statistical pass rates: easy (9–12 correct rollouts), medium (5–8 correct rollouts), and hard (1–4 correct rollouts). Tasks with 13–16 correct rollouts are excluded because they are already well-solved and provide limited training signal.
We used GLM-4.7-FP8 and achieved the following synthesized data distribution:
| Difficulty | Airline | Retail | Telecom | Total Tasks |
|---|---|---|---|---|
| Easy | 170 (36.9%) | 255 (55.3%) | 36 (7.8%) | 461 |
| Medium | 190 (54.6%) | 143 (41.1%) | 15 (4.3%) | 348 |
| Hard | 346 (46.1%) | 388 (51.7%) | 16 (2.1%) | 750 |
Experiments
Setup
User Simulator
The selected user simulator model for training and evaluation is GLM-5-FP8. The user simulator endpoints can be deployed locally or in Vertex AI Model Garden. For easy reproduction, we provide sample scripts to deploy GLM-5-FP8 locally in clusters as well.
While our offline task generation pipeline utilized GLM-4.7 to efficiently scale the synthesis and verification of thousands of scenarios, utilizing a more powerful model as the live user simulator is essential to mitigate negative impacts on RL training stability. Specifically, GLM-5 outperforms GLM-4.7 in this role, providing a more robust and strictly compliant simulation environment. Furthermore, this decoupling mitigates self-reinforcing biases by ensuring the policy agent does not merely overfit to the linguistic quirks of the model used to generate its training data.
Training Configuration
- Checkpoint: Our SFT checkpoints were fine-tuned from Qwen3-8B, as described in the Model Distillation Best Practices blog.
- Training Data: Our synthesized data described above.
- Hyperparameters:
| Parameter | Value |
|---|---|
| Prompts per step | 64 |
| Generations per prompt | 16 |
| Global batch size | 1024 |
| Max turns | 40 |
| Optimizer | Adam |
| Max num steps | 150 |
| Temperature | 1.0 |
Evaluation
We use τ2-bench (v2) as our evaluation dataset. The τ2-bench community mainly reports Pass1 with 4 trials and averages across three different domains. The same models may produce different results across runs—this variance is by design in τ2-bench. Due to limited resources, we report the mean and standard deviation for the main results from 5 runs, and only report results from one run in ablation studies. Please refer to the Background section for a description of the evaluation metrics, and to the original paper for more details.
Main Results
We compare our SFT and RL models against state-of-the-art models:
| Model | Setup | Stage | Retail | Airline | Telecom | Avg |
|---|---|---|---|---|---|---|
| Qwen3-8B-Base | Qwen3 official pre-trained checkpoint | Pre-trained | 6.1 | 39.0 | 15.4 | 20.2 |
| Qwen3-8B | Qwen3 official post-trained checkpoint | Post-trained | 50.7 | 30.0 | 45.8 | 42.2 |
| Qwen3-235B-A22B-Thinking-2507 | Qwen3 official flagship post-trained model | Post-trained | 72.1 | 56.5 | 73.2 | 67.3 |
| Cirrus-Agent-SFT 8B [Ours] | Cirrus-0.5 8B, SFT with tool use data and rejection sampling | SFT | 67.4 ± 3.0 | 55.5 ± 3.3 | 73.5 ± 1.3 | 65.5 ± 1.5 |
| Cirrus-Agent-RL 8B [Ours] | RL based on Cirrus-Agent-SFT 8B | RL | 68.1 ± 0.8 | 56.8 ± 3.2 | 85.9 ± 2.2 | 70.2 ± 1.4 |
Key observations from our main results:
- RL models improve Pass1 from 65.5 to 70.2 (+4.7) overall and achieve a massive improvement on telecom tasks from 73.5 to 85.9 (+12.4), confidently demonstrating that RL helps improve model performance.
- On retail tasks, RL models improve Pass1 slightly (+0.7, within noise), but variance collapses from ±3.0 to ±0.8 (~73% reduction). This dramatic variance reduction means that while RL did not make the model more accurate on average, it made it far more consistent and predictable.
- On airline tasks, both the variances of SFT and RL models are large (~3) and the improvements of RL models are minor (+1.3, within noise).
Evaluation Details for SFT and RL Models
For better reproduction and understanding of evaluation results, here are detailed per-run results and a suggested interpretation guide. The evaluated RL model was trained with all synthetic data.
| Model | #Run | Retail | Airline | Telecom | Avg |
|---|---|---|---|---|---|
| Cirrus-Agent-SFT 8B | Run 1 | 71.3 | 51.0 | 73.0 | 65.1 |
| Run 2 | 69.7 | 59.0 | 75.0 | 67.9 | |
| Run 3 | 66.2 | 58.5 | 72.1 | 65.6 | |
| Run 4 | 63.8 | 55.0 | 72.5 | 63.8 | |
| Run 5 | 66.2 | 54.0 | 74.8 | 65.0 | |
| x̄ | 67.4 | 55.5 | 73.5 | 65.5 | |
| σSFT | 3.0 | 3.3 | 1.3 | 1.5 | |
| Cirrus-Agent-RL 8B | Run 1 | 67.5 | 62.5 | 87.1 | 72.4 |
| Run 2 | 68.6 | 55.0 | 82.7 | 68.8 | |
| Run 3 | 69.1 | 55.0 | 84.6 | 69.6 | |
| Run 4 | 67.8 | 56.0 | 88.2 | 70.7 | |
| Run 5 | 67.3 | 55.5 | 86.8 | 69.9 | |
| x̄ | 68.1 | 56.8 | 85.9 | 70.2 | |
| σRL | 0.8 | 3.2 | 2.2 | 1.4 | |
| Δx̄ | 0.7 | 1.3 | 12.4 | 4.7 | |
| σcombined | 3.1 | 4.6 | 2.6 | 2.0 | |
| Significance | 0.2 | 0.3 | 4.8 | 2.3 | |
Suggested Interpretation Guide:
The significance of overall (2.3×) and telecom (4.8×) results confidently demonstrates that RL improves performance.
| Significance Level | Sigma | Interpretation |
|---|---|---|
| Very High | >3σ | Definitive effect |
| High | >2σ | Statistically significant |
| Moderate | 1σ–2σ | Suggestive but inconclusive |
| Low | <1σ | Within random variation |
Training Curves
 |
Ablation Studies
We performed ablation studies on different learning rates, KL penalties, and data combinations. Due to limited resources, we only report Pass1 with 4 trials from a single run.
| Data | Step | LR | KL | Retail | Airline | Telecom | Avg |
|---|---|---|---|---|---|---|---|
| Easy | 70 | 1.0E-6 | n/a | 70.6 | 58.5 | 84.9 | 71.3 |
| Easy | 70 | 5.0E-7 | n/a | 68.2 | 57.5 | 79.2 | 68.3 |
| Easy | 70 | 1.5E-6 | n/a | 67.3 | 58.0 | 88.2 | 71.1 |
| Easy | 75 | 2.0E-6 | n/a | 72.4 | 56.0 | 84.9 | 71.1 |
| Easy | 135 | 1.0E-6 | 0.01 | 69.1 | 58.0 | 85.0 | 70.7 |
| Easy | 140 | 1.0E-6 | 0.02 | 67.3 | 58.0 | 81.4 | 68.9 |
| Easy | 115 | 1.0E-6 | 0.05 | 71.7 | 58.0 | 83.3 | 71.0 |
| Easy | 105 | 1.0E-6 | 0.1 | 69.5 | 58.5 | 82.9 | 70.3 |
| Easy+Medium | 45 | 2.0E-6 | n/a | 69.3 | 59.0 | 84.2 | 70.9 |
| Easy+Medium+Hard | 50 | 2.0E-6 | n/a | 67.5 | 62.5 | 87.1 | 72.4 |
Key observations from the ablation studies:
- Using the easy data, models trained with learning rates 1.0E-6, 1.5E-6, and 2.0E-6, or KL penalty 0.05, achieved similar results and outperformed other configurations.
- Mixing easy and medium data produced similar results to using only easy data.
- Mixing easy, medium, and hard data yielded the best results, outperforming both easy-only and easy+medium configurations.
- The best results occurred after training 45–75 steps (approximately 2–10 epochs) for training without KL. Training may overfit to the training data when running for additional steps.
More Analysis
Failure Patterns. In the evaluation dataset, there are tasks with simple tool-call sequences—simple state toggles and straightforward procedures—such as all telecom tasks and partial airline/retail tasks. Other tasks require correct multi-step tool-call chains with multi-entity reasoning and constraints, such as the majority of airline/retail tasks. SFT models generally understand what to do and maintain strong user communication, but sometimes struggle to execute the correct tool-call sequences. RL models directly optimize tool-calling behavior through reward signals, improving performance overall, but exhibit some common failure patterns:
- Skipped tool calls: The model converses correctly but omits necessary actions (e.g.,
modify_pending_order_items,get_reservation_details), resulting in the database not being updated correctly. - Incorrect tool parameters: The model calls the correct tools but with wrong arguments (e.g., wrong item IDs, order IDs), leaving the database in the wrong state.
- Over-action: Instead of refusing disallowed operations or escalating to a human agent (
transfer_to_human_agents), the model proceeds with actions that should be declined, becoming more “action-biased.”
Data Paradox. Telecom has 10× less training data than airline and retail, but achieves significantly better performance:
| Domain | % of Training Data | Pass1 |
|---|---|---|
| Retail | 50.4% | 68.1 |
| Airline | 45.3% | 56.8 |
| Telecom | 4.3% | 85.9 |
This telecom performance advantage is likely driven by a more deterministic tool graph, structured slot-filling parameters, and lower linguistic variance from the simulator compared to the other more open-ended domains. We analyze airline and retail failures further:
- There are many airline failures for complex tasks (3+ actions), indicating that trained models should improve their ability to chain multi-step workflows.
- The retail failures are more long-tail in nature—various small failures where trained models make occasional mistakes on many different actions.
Known Issues for Airline and Retail Evaluations. The community has been invaluable in identifying issues—from annotation errors to underspecified tasks—in the original airline and retail domains. 50+ tasks were fixed in τ3-bench releases.
Top Directions for Addressing Remaining Error Patterns:
- Add action-sequence SFT pre-training before RL to learn tool-calling patterns, which may accelerate RL convergence.
- Enable light reward shaping (e.g., 0.15 format weight) to provide learning signal on total failures instead of pure 0 reward.
- Use τ3-bench as evaluations.
Key Takeaways
Thanks for reading. We hope this RL training framework and these insights help you build better tool-calling agents on Managed Training Clusters.
- Performance Gains from RL: RL training increases the overall Pass1 success rate from 65.5 to 70.2 (+4.7), highlighted by a massive +12.4 performance boost on telecom tasks.
- Variance Reduction in Retail: While average performance gains on retail tasks are minor, RL reduces variance by roughly 73% (from ±3.0 to ±0.8), ensuring much more consistent and predictable agent behavior.
- The Data Paradox: Despite having 10× less training data than other domains, telecom achieves the highest performance (85.9 Pass1), demonstrating that domain clarity and data quality are far more critical than raw quantity.
- Actionable Future Directions: To address complex workflow failures and long-tail action errors, future iterations should incorporate action-sequence SFT pre-training to accelerate RL convergence and implement light reward shaping to provide a stronger learning signal.
Acknowledgements
We would like to express our sincere gratitude to the NVIDIA NeMo RL team for their invaluable support throughout this project.
We would also like to express our gratitude to our MTC teammates: Mohammadreza Mohseni, Weiran Zhao, and Bo Wu for their infrastructure support, feedback, and insightful discussions throughout the project. We also thank Ting Yu, Shengyang Dai, Peng Xu, and Aparna Ramani for their leadership and support.