feat(experimentalist): Remove tau2 from getting started guide and add experimentalist instructions - #1010
Merged
Merged
Conversation
Consolidate the Airline datasets and provide a bounded runner so users can ingest ground-truth-free Tau3 traces before analyzing or optimizing the agent. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
Contributor
|
Name the standalone command after its observable recording behavior rather than its downstream consumer. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
Keep the standalone example focused on its validated end-to-end workflow without a dedicated Insights runner test module. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
Document the consolidated dataset, trace ingestion, Insights, and Experimentalist flow with a sourceable environment template. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
nicot
marked this pull request as ready for review
July 31, 2026 17:44
nicot
changed the title
Contributor
Set the Tau3 agent and simulator to Luna while assigning Sol, Terra, and Luna to the Experimentalist smart, mid, and fast roles. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
Contributor
📝 WalkthroughWalkthroughThe Tau3 no-OOA airline example now configures services, prepares validated datasets, records Harbor evaluation traces, uploads them to Intake, and documents Insights and Experimentalist workflows. ChangesTau3 airline workflow
Sequence Diagram(s)sequenceDiagram
participant User
participant DatasetScript
participant Harbor
participant Intake
participant Insights
participant Experimentalist
User->>DatasetScript: Prepare Tau3 dataset splits
DatasetScript-->>User: Return Insights and Experimentalist paths
User->>Harbor: Run agent trials
Harbor-->>User: Return trial traces
User->>Intake: Upload traces with experiment metadata
Intake-->>User: Confirm trace availability
User->>Insights: Analyze uploaded traces
User->>Experimentalist: Run optimization with train and validation splits
Experimentalist-->>User: Write experiment results
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
plugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py (1)
152-189: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winNo safeguard against duplicate uploads on workspace reuse.
RUNBOOK.md warns to use a fresh
--workspacename because "Intake appends traces," but nothing here enforces it. Reusing a workspace by mistake silently duplicates the Insights corpus. Consider checking for existing traces in the target workspace before running and warning or requiring a--forceflag to proceed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py` around lines 152 - 189, Update the workspace setup flow before running HarborEvaluator in the main try block to detect whether the target workspace already contains traces. Require an explicit force/override option before continuing with a reused workspace, otherwise fail or warn and skip uploads; preserve the current behavior for fresh workspaces and avoid executing the upload path without this safeguard.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/examples/tau3-nooa-agent/agent.py`:
- Around line 36-72: Update the NOOA code-execution path used by
StopAwareMCPManager so ConversationEnded is not caught and converted into
RuntimeError by RuntimeServices.execute_code(); propagate it unchanged to
main.py while retaining existing handling for other BaseException values. Add a
smoke test covering a Tau3 stop token raised through tool execution and reaching
the main flow.
In
`@plugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py`:
- Around line 234-235: The recorder’s --model and --user-model arguments ignore
the configured environment variables. In record_tau_airline_traces.py, update
these defaults to read AUT_MODEL_NAME and TAU2_USER_MODEL with DEFAULT_MODEL as
fallback; make no changes to .env.example or RUNBOOK.md, which are corrected by
this root-cause fix.
---
Nitpick comments:
In
`@plugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py`:
- Around line 152-189: Update the workspace setup flow before running
HarborEvaluator in the main try block to detect whether the target workspace
already contains traces. Require an explicit force/override option before
continuing with a reused workspace, otherwise fail or warn and skip uploads;
preserve the current behavior for fresh workspaces and avoid executing the
upload path without this safeguard.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 018a6547-baa0-4c18-9ad9-8f1f4ffcf7fc
📒 Files selected for processing (8)
docs/get-started/example-agent.mdxplugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.exampleplugins/nemo-experimentalist/examples/tau3-nooa-agent/RUNBOOK.mdplugins/nemo-experimentalist/examples/tau3-nooa-agent/agent.pyplugins/nemo-experimentalist/examples/tau3-nooa-agent/main.pyplugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-datasets.shplugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-smoke.shplugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py
💤 Files with no reviewable changes (1)
- plugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-smoke.sh
Restore the example agent's original MCP behavior until conversation termination can be handled with a more thoroughly validated design. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
Use the configured agent and user-simulator models as recorder defaults so the documented environment controls execution. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
aleckhoury
approved these changes
Jul 31, 2026
Keep the Tau3 workflow documented only in the getting-started guide. Signed-off-by: Nico Tonozzi <ntonozzi@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates on Tau3, removes Tau2 data from getting started guide.
Uses tau3 tasks for insights generation
Adds instructions for running experimentalist on tau3
Summary by CodeRabbit
New Features
Documentation
Chores