fix(tau2): exclude review_model from snapshot comparison - #2292
Merged
Conversation
review_model is a tau2-bench config field whose default value varies across tau2-bench versions (added in v1.0.1 with default 'claude-opus-4-5', but not always present depending on the installed commit). _clean() already strips other volatile config fields (max_agent_steps, turns_remaining_interval); review_model follows the same pattern. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
cmunley1
approved these changes
Aug 3, 2026
anwithk
approved these changes
Aug 3, 2026
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
…#2292) ## Summary \`test_sanity_query_input\` was failing in CI because \`review_model\` is present in \`test_data.json\` (the snapshot) but absent from the actual server response in some tau2-bench versions. \`review_model\` was added to \`TextRunConfig\` in tau2-bench v1.0.1 with a default of \`'claude-opus-4-5'\`. Its presence in the serialized config depends on which tau2-bench commit is installed. The test's \`_clean()\` function already strips other volatile config fields (\`max_agent_steps\`, \`turns_remaining_interval\`) — adding \`review_model\` follows the same pattern and makes the comparison stable across tau2-bench versions. ## Change One line added to \`_clean()\` in \`responses_api_agents/tau2/tests/test_app.py\`: \`\`\`python d["config"].pop("review_model", None) \`\`\` ## Verified 6/6 tau2 tests pass locally. Failure first surfaced in the full suite run off NVIDIA-NeMo#2290: https://github.com/NVIDIA-NeMo/Gym/actions/runs/30840804435 Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
`test_sanity_query_input` was failing in CI because `review_model` is present in `test_data.json` (the snapshot) but absent from the actual server response in some tau2-bench versions.
`review_model` was added to `TextRunConfig` in tau2-bench v1.0.1 with a default of `'claude-opus-4-5'`. Its presence in the serialized config depends on which tau2-bench commit is installed. The test's `_clean()` function already strips other volatile config fields (`max_agent_steps`, `turns_remaining_interval`) — adding `review_model` follows the same pattern and makes the comparison stable across tau2-bench versions.
Change
One line added to `_clean()` in `responses_api_agents/tau2/tests/test_app.py`:
```python
d["config"].pop("review_model", None)
```
Verified
6/6 tau2 tests pass locally.
Failure first surfaced in the full suite run off #2290:
https://github.com/NVIDIA-NeMo/Gym/actions/runs/30840804435