feat(experimentalist): add the tau3-bench banking benchmark suite - #935
Merged
Conversation
cursor
Bot
changed the title
cursor
Bot
force-pushed
the
cursor/cloud-agent-1785190461860-3q8ge
branch
from
July 27, 2026 23:09
84d41ba to
db3f82b
Compare
Contributor
|
The runner hardcoded Terminal Bench's workspace and framework skill and assumed a package held exactly one domain, so a second suite could not be expressed as a manifest. A manifest now carries its own workspace and framework_skills, and may set task_id_prefix to scope a multi-domain package to one domain; coverage is then enforced against the matching subset. Skill directories resolve before the baseline evaluation, because otherwise a typo'd skill name surfaces hours of image builds later. A config may also name a user_simulator, which exports the OpenAI-style credentials that tau-style task environments and their NL-assertion judge read. Those are assigned rather than defaulted: a developer's ambient OPENAI_API_KEY paired with this gateway's base URL would 401 on every call. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
Scopes sierra-research/tau3-bench@1 to its 97 banking_knowledge tasks. The quality partition reuses the 41/28/28 split from optimization-datasets feat/tau2-other-domains, whose tau2-banking-knowledge-NNN names map onto the canonical Hub IDs; only task IDs are recorded here, never task content. The 6/3/3 fast partition draws solely from the 87 tasks whose reward_basis is pure database state, so smoke runs score without an LLM judge. Both partitions were validated against live Harbor Hub. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
A NOOA CodeAct agent that reaches the tau2 domain tools over MCP from the task's tau3-runtime sidecar. AGENT-SPEC.md stays domain-generic because the policy arrives at runtime in the task instruction, and it is what the Experimentalist mutates. Three things the task images required: Installing from the committed lock uses the pip that python:3.12-slim already ships, since tau3 images carry neither uv nor the curl its installer wants. Terminal Bench's uv-upload machinery is unnecessary here because every tau3 task shares that one base image. Model credentials are handed over at exec time. A task's [environment].env only interpolates its docker-compose file, and upstream wires those variables into the sidecar; the main service declares no environment, so the agent had none. mcp_timeout.py replaces nooa 0.0.6's MCP transport, which is built without an httpx timeout and so inherits httpx's 5 second default. The sidecar runs the user simulator inside start_conversation and send_message_to_user, which take longer; the server completed the work and advanced its state, but the reply landed on an abandoned stream and nooa surfaced no timeout, so the agent hung indefinitely. Verified end to end on the fast test split: all three tasks complete with no harness error, one solved, for a 1/3 baseline the optimizer can improve on. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
Turns the single-suite README into one that covers both suites: how a manifest carries its own workspace, framework skills, and optional domain prefix, the banking provenance and partition derivation, and the two-container topology with its memory and build-timeout consequences. Also records that upstream's tau3 Dockerfiles clone tau2-bench without a ref, so a cold layer cache can pick up newer commits than an earlier run saw. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
callingmedic911
force-pushed
the
cursor/cloud-agent-1785190461860-3q8ge
branch
from
July 28, 2026 20:18
fd292b4 to
ca4ff47
Compare
…sk-id prefix Every ID in a domain-scoped package repeats the domain prefix, so listing 109 of them buried the part that actually differs and made the banking manifest 143 lines of near-identical strings. Partition entries are now names inside the suite's task_id_prefix, joined back on at load so the rest of the runner keeps working in canonical IDs. The banking manifest drops to 50 lines and its splits are legible at a glance. Suites without a prefix, meaning Terminal Bench, are unaffected. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
A config carries the models and optimizer depth for one suite, but the original pair was named only for its partition. Once tau3-smoke.yaml and tau3-quality.yaml sat beside them, smoke.yaml and quality.yaml read as the generic pair rather than the Terminal Bench one. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
Drops the container topology, the memory and build-timeout guidance, and the note about upstream's unpinned tau2-bench clone. What remains mirrors the Terminal Bench section: package, hash, Hub record, partition derivation, and the one config field the suite introduces. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
callingmedic911
changed the title
nicot
reviewed
Jul 28, 2026
nicot
approved these changes
Jul 29, 2026
nooa built its streamable-http transport with an httpx client that had no timeout, so httpx's 5 second default governed the MCP read stream and any tool call taking longer than that hung forever. The tau3 agent hit this on every turn, because the tau3-runtime sidecar runs a user-simulator LLM inside start_conversation and send_message_to_user. That is fixed upstream in NVIDIA-NeMo/labs-OO-Agents#46, so this drops the local mcp_timeout.py shim and goes back to MCPManager.create_from_server, which now takes tool_call_timeout directly. The agent still asks for 300 seconds: nooa's 60 second default would do today but leaves no room for a slower model. The pin is a commit rather than a tag because no release carries the fix yet. The lock's extra wheel rows are generated: no package version moves except nooa. Verified against a live sidecar that start_conversation now returns in 7.3s instead of hanging, and that a full Harbor trial completes with no harness error. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
…85190461860-3q8ge Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> # Conflicts: # uv.lock
callingmedic911
marked this pull request as ready for review
July 29, 2026 20:39
callingmedic911
enabled auto-merge
July 29, 2026 20:39
Contributor
📝 WalkthroughWalkthroughChangesExperimentalist benchmark execution
Sequence Diagram(s)sequenceDiagram
participant Harness as Harbor harness
participant Wrapper as WrappedAgent
participant Agent as Codeact
participant Runtime as tau3-runtime
Harness->>Wrapper: install and run agent
Wrapper->>Agent: execute instruction
Agent->>Runtime: call MCP tools
Agent-->>Wrapper: write traces
Wrapper-->>Harness: return metrics and artifacts
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/benchmarks/README.md`:
- Around line 4-15: Restructure the benchmark README into a single how-to page:
place prerequisites first, add Python SDK and CLI tabbed examples, provide
substitutions for product names, and finish with Next Steps. Move benchmark
provenance and artifact details, including the suite metadata currently in the
overview table, into linked reference and explanation pages. Apply the same
structure to the content covering the referenced suite sections.
In `@plugins/nemo-experimentalist/examples/tau3-nooa-agent/harbor_wrapper.py`:
- Line 75: Update the agent setup command in the harbor wrapper to remove
pip-based installation of uv. Use a pinned uv binary already provided in the
image or bootstrap uv through a non-pip mechanism, then retain the existing
UV_HTTP_TIMEOUT and run uv sync --frozen.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: be40c140-2513-4b7d-81e6-1d099dc2ba89
⛔ Files ignored due to path filters (2)
plugins/nemo-experimentalist/examples/tau3-nooa-agent/uv.lockis excluded by!**/*.lockuv.lockis excluded by!**/*.lock
📒 Files selected for processing (19)
plugins/nemo-experimentalist/AGENTS.mdplugins/nemo-experimentalist/README.mdplugins/nemo-experimentalist/benchmarks/README.mdplugins/nemo-experimentalist/benchmarks/configs/tau3-quality.yamlplugins/nemo-experimentalist/benchmarks/configs/tau3-smoke.yamlplugins/nemo-experimentalist/benchmarks/configs/terminal-bench-quality.yamlplugins/nemo-experimentalist/benchmarks/configs/terminal-bench-smoke.yamlplugins/nemo-experimentalist/benchmarks/run.pyplugins/nemo-experimentalist/benchmarks/suites/tau3-banking.yamlplugins/nemo-experimentalist/benchmarks/suites/terminal-bench-2.1.yamlplugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.mdplugins/nemo-experimentalist/examples/tau3-nooa-agent/README.mdplugins/nemo-experimentalist/examples/tau3-nooa-agent/agent.pyplugins/nemo-experimentalist/examples/tau3-nooa-agent/harbor_wrapper.pyplugins/nemo-experimentalist/examples/tau3-nooa-agent/main.pyplugins/nemo-experimentalist/examples/tau3-nooa-agent/pyproject.tomlplugins/nemo-experimentalist/framework-skills/nooa/SKILL.mdplugins/nemo-experimentalist/tests/test_experimentalist_benchmark.pypyproject.toml
This was referenced Jul 30, 2026
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the banking domain of
sierra-research/tau3-benchas a second Experimentalist benchmark suite, next to Terminal Bench.Suite.
sierra-research/tau3-bench@1, pinned by content hash. We pull and use that directly.Agent.
examples/tau3-nooa-agentdrives the task'stau3-runtimesidecar over MCP. Three things had to be fixed before it ran at all:uvmcp_timeout.pyworks around it. fix: honor tool_call_timeout so slow MCP tool calls stop hanging labs-OO-Agents#46 once this lands, I'll clean that up.Summary by CodeRabbit
New Features
Documentation
Bug Fixes