fix(experimentalist): improve getting started guide and testbed

docs/get-started/example-agent.mdx

@@ -3,7 +3,10 @@ title: "Get started with an example agent"
 description: "Run the optimization loop from end to end on τ-Bench"
 ---
 
-This provides a guide for how you can quickly load agent traces into your NeMo Platform, run the analyst agent to discover issues with that agent, and launch an experimentalist to fix those issues. This is meant as an example to show how the whole platform works. You can also jump straight to setting up your agent with NeMo Platform.
+This guide runs one Tau3 Airline agent through the full loop: Harbor evaluates
+the checked-in example agent, the Analyst discovers issues from its traces, and
+the Experimentalist uses one discovered Insight to improve the same agent.
+You can also jump straight to setting up your agent with NeMo Platform.
 
 ## Prerequisites
 
@@ -32,82 +35,128 @@ services/intake/scripts/spans/run_clickhouse.sh
 uv run nemo services start --config packages/nmp_platform/config/local.yaml
 ```
 
+In order to visualize experiment results, you need to enable the `experiment` feature flag. You can do this by editing the `packages/nmp_platform/config/local.yaml` file and setting the `feature_flags.experiment` to `true`.
+
+```yaml
+feature_flags:
+  experiment: true
+```
+
 You should now be able to navigate to `http://localhost:8080` and see the NeMo Platform web UI.
 
-## 3. Prepare Tau2 and run the airline agent
+## 3. Run the Tau3 Airline agent with Harbor
 
-Now that we have a running NeMo Platform, it's time to load up some data! This example uses the τ-Bench from Sierra. It's a great representation of a simplified agent that performs a business-critical task. Our first step will be to download the tau2 repository and install the dependencies to run it.
+Now that NeMo Platform is running, evaluate the
+[checked-in Tau3 NOOA example agent](https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/plugins/nemo-experimentalist/examples/tau3-nooa-agent)
+against the
+[`sierra-research/tau3-bench`](https://hub.harborframework.com/datasets/sierra-research/tau3-bench/latest)
+dataset on Harbor Hub. The example is a customer-service agent built with
+[NVIDIA-labs OO Agents (NOOA)](https://github.com/NVIDIA-NeMo/labs-OO-Agents)
+and a CodeAct strategy. It receives the Airline
+policy at runtime and uses MCP tools to look up records, talk to the simulated
+customer, and complete permitted actions. Its `AGENT-SPEC.md` defines the
+behavior that the Experimentalist later optimizes.
 
-Then, we'll run a script to go through 30 of the tasks from the Tau 2 Airline dataset. In order to make our example more realistic and representative of the kind of agent data we typically see in production, we won't record whether the tasks passed verification – we want to see if we can gain useful insights even without ground truth data. This could take up to 20 minutes, and will cost about a dollar.
+The Insights testbed selects six Airline tasks, runs the agent through Harbor,
+and uploads the resulting traces and verifier rewards into NeMo Platform.
 
-```bash
-git init tmp/tau2-bench
-git -C tmp/tau2-bench remote add origin https://github.com/sierra-research/tau2-bench.git
-git -C tmp/tau2-bench fetch --depth 1 origin 8ebb7499622fc2be9b9d510d6f7a7653461f4f29
-git -C tmp/tau2-bench checkout --detach FETCH_HEAD
-uv --directory tmp/tau2-bench sync --frozen
-uv --directory tmp/tau2-bench run --frozen tau2 check-data
+This can take 20 minutes or longer because Harbor builds task containers and
+runs model-backed user simulation.
 
+```bash
 export NMP_BASE_URL=http://localhost:8080
 export INFERENCE_API_KEY=sk-...
 export OPENAI_API_KEY="$INFERENCE_API_KEY"
-export OPENAI_API_BASE=https://inference-api.nvidia.com/v1
+export OPENAI_BASE_URL=https://inference-api.nvidia.com/v1
 
 uv run --directory plugins/nemo-insights --frozen \
-  python -m testbed run tau2-airline \
-  --base "$NMP_BASE_URL" \
-  --set include_rewards=false \
-  --set tau2_repo="$PWD/tmp/tau2-bench"
+  python -m testbed run tau3-airline-harbor \
+  --base "$NMP_BASE_URL"
 ```
 
-The testbed runs the 30-task airline train split and ingests realistic traces into the `tau2-airline` workspace. At this point you can navigate to the traces view and see the traces from the agent.
+In Studio, open the `canonical-tau3-airline` workspace's Experiments area to
+inspect the task results, rewards, and traces.
 
 ## 4. Run the Analyst
 
 Now that we have data in our system, we can run the analyst agent to understand the ways that our agent is performing well, and where it is falling down. The goal of the analyst agent is to crawl production data, and determine where your agent is falling down or disappointing your customers.
 
 If the analyst discovers issues in your application, it will create what we call an 'insight'. An insight is a human-readable description of a problem in your agentic system. The closest analogy is a bug report. They don't try to describe why an issue happened or the code you should change to fix it, and they should be understandable to a user of your agent.
 
-Let's run it:
+Run the Analyst against the agent's traces in the dedicated workspace:
 
 ```bash
 uv run --frozen nemo insights analyze \
-  --agent tau2-airline \
-  --workspace tau2-airline \
+  --agent nemo-experimentalist-tau3-nooa \
+  --workspace canonical-tau3-airline \
   --base-url "$NMP_BASE_URL"
 ```
 
-This will take a few minutes to run. Once it's done, you can navigate to the insights page to see the issues the analyst discovered in the τ-Bench Airline Agent.
+The Insights are published to NeMo Platform. In Studio, open the
+`canonical-tau3-airline` workspace's Insights area to review them. Select an
+Insight and copy its ID for the Experimentalist command below.
 
-## 5. Prepare a Tau3 Airline smoke dataset
+## 5. Prepare the Tau3 Airline quality dataset
 
-The next step is to improve the τ-Bench agent using the experimentalist. The
-experimentalist will run your evals, debug failures using trace data, understand
-the root cause of the failure and attempt to fix it. After it makes the change,
-it will run the evals again to validate whether the change improved the
-performance of your agent on the evalaution.
+The Experimentalist now improves the same checked-in Tau3 agent. It uses the
+Insight selected above as its failure lens, generates candidate changes,
+and evaluates them on explicit train and validation splits from the same Harbor
+dataset.
 
-This is a shortened example that only uses a few tasks, but it can still take up to an hour to finish. First, set your environment variables and set up the task dataset:
+The partition contains 20 training tasks and 10 validation tasks. The smoke
+configuration used below runs one optimization round and samples at most four
+training tasks, keeping this getting-started run bounded. First, set your
+environment variables and prepare the dataset:
 
 ```bash
 export OPENAI_API_KEY="$INFERENCE_API_KEY"
 export OPENAI_BASE_URL=https://inference-api.nvidia.com/v1
 export TAU2_USER_MODEL=openai/openai/openai/gpt-5-mini
 export TAU2_NL_ASSERTIONS_MODEL=openai/openai/openai/gpt-5-mini
 export AUT_MODEL_NAME=openai/openai/openai/gpt-5-mini
-export EXPERIMENTALIST_SMART_MODEL_NAME=openai/openai/openai/gpt-5-mini
-export EXPERIMENTALIST_MID_MODEL_NAME=openai/openai/openai/gpt-5-mini
+export EXPERIMENTALIST_SMART_MODEL_NAME=openai/openai/openai/gpt-5.3-codex
+export EXPERIMENTALIST_MID_MODEL_NAME=openai/gcp/google/gemini-3.5-flash
 export EXPERIMENTALIST_FAST_MODEL_NAME=openai/openai/openai/gpt-5-mini
 
 uv run --frozen nemo workspaces create canonical-tau3-airline \
   --description "Tau3 Airline Experimentalist runs" \
   --exist-ok
 
-plugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-smoke.sh
+plugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline.sh
 ```
 
 Now we're ready to run the experimentalist!
 
+```bash
+export PLATFORM_INSIGHT_ID=insight-id-from-studio
+
+uv run --frozen nemo experimentalist run \
+  --insight "$PLATFORM_INSIGHT_ID" \
+  --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \
+  --agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \
+  --train-dataset plugins/nemo-experimentalist/tmp/tau3-airline-smoke/train \
+  --validation-dataset plugins/nemo-experimentalist/tmp/tau3-airline-smoke/validation \
+  --task-template plugins/nemo-experimentalist/tmp/tau3-airline-smoke/source/tau3-bench/tau3-bench__tau3-airline-0 \
+  --workspace canonical-tau3-airline \
+  --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \
+  --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \
+  --base-url "$NMP_BASE_URL"
+```
+
+After the run completes, open the workspace's Experiments area in Studio to
+review its evaluations, compare candidates, and inspect the selected winner.
+
+## 6. (Optional) Optimize without insights
+
+If your dataset already includes objective evaluation metrics, you can run the
+Experimentalist without an Analyst-generated Insight. In this mode,
+the Harbor verifier rewards from the training dataset drive candidate
+selection, while the validation dataset measures whether the selected changes
+generalize.
+
+Pass `--no-insight` explicitly. This prevents the Experimentalist from loading
+an Insight supplied by a profile or left over from an earlier Analyst run:
+
 ```bash
 uv run --frozen nemo experimentalist run \
   --no-insight \
@@ -118,11 +167,11 @@ uv run --frozen nemo experimentalist run \
   --workspace canonical-tau3-airline \
   --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \
   --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \
-  --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \
   --base-url "$NMP_BASE_URL"
 ```
 
-The trace records include the Experimentalist evaluation ID and Tau3 task ID.
-After the run completes, inspect `eval-and-optimize/run.json` for the selected
-winner and compare the `agent-0` and `agent-1` directories to review the code
-change that was evaluated.
+The Experimentalist skips the production-issue lens provided by an Insight; it
+does not skip evaluation or failure analysis inside the optimization loop. The
+Experimentalist still runs Harbor tasks, examines failing trajectories,
+proposes agent changes, and compares candidates using the dataset's verifier
+metrics.

plugins/nemo-experimentalist/benchmarks/configs/tau3-quality.yaml

@@ -21,9 +21,6 @@ optimizer:
   disable_convergence_check: false
   evaluator:
     n_attempts: 2
-    # Keep at 3 or lower: each task requests 8192 MB and parallel image builds
-    # have triggered Docker Hub rate limiting and DNS failures.
-    n_concurrent_trials: 3
     quiet: true
     agent_setup_timeout_multiplier: 2.0
     # tau3 builds two images per task, and the first build clones and installs

plugins/nemo-experimentalist/benchmarks/configs/tau3-smoke.yaml

@@ -21,7 +21,6 @@ optimizer:
   disable_convergence_check: true
   evaluator:
     n_attempts: 1
-    n_concurrent_trials: 1
     quiet: true
     agent_setup_timeout_multiplier: 2.0
     # tau3 builds two images per task, and the first build clones and installs

plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-quality.yaml

@@ -0,0 +1,22 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Agent-level equivalent of benchmarks/configs/tau3-quality.yaml's optimizer
+# section, for direct `nemo experimentalist run --config ...` usage.
+max_rounds: 3
+min_rounds_before_stopping: 2
+max_survivors: 2
+max_candidates: 3
+max_trajectory_tasks: 8
+max_train_batch_tasks: 16
+train_batch_seed: 20260727
+disable_trajectory_scoring: false
+disable_convergence_check: false
+evaluator:
+  n_attempts: 2
+  quiet: true
+  agent_setup_timeout_multiplier: 2.0
+  environment_build_timeout_multiplier: 3.0
+eval_author:
+  max_traces: 10
+  max_validation_repair_attempts: 5

plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml

@@ -12,7 +12,6 @@ disable_trajectory_scoring: true
 disable_convergence_check: true
 evaluator:
   n_attempts: 1
-  n_concurrent_trials: 1
   quiet: true
   agent_setup_timeout_multiplier: 2.0
   environment_build_timeout_multiplier: 3.0