feat(experimentalist): Remove tau2 from getting started guide and add experimentalist instructions

docs/get-started/example-agent.mdx

@@ -34,33 +34,34 @@ uv run nemo services start --config packages/nmp_platform/config/local.yaml
 
 You should now be able to navigate to `http://localhost:8080` and see the NeMo Platform web UI.
 
-## 3. Prepare Tau2 and run the airline agent
+## 3. Prepare τ-Bench and run the airline agent
 
-Now that we have a running NeMo Platform, it's time to load up some data! This example uses the τ-Bench from Sierra. It's a great representation of a simplified agent that performs a business-critical task. Our first step will be to download the tau2 repository and install the dependencies to run it.
+Now that we have a running NeMo Platform, it's time to load up some data! This example uses the τ-Bench from Sierra. It's a great representation of a simplified agent that performs a business-critical task. Our first step will be to set up our environment variables so that we can run models properly.
 
-Then, we'll run a script to go through 30 of the tasks from the Tau 2 Airline dataset. In order to make our example more realistic and representative of the kind of agent data we typically see in production, we won't record whether the tasks passed verification – we want to see if we can gain useful insights even without ground truth data. This could take up to 20 minutes, and will cost about a dollar.
+```bash
+cp plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.example plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env
+```
+
+Then, update the `.env` file to have an appropriate API key for your inference endpoint, and source the `.env` file.
+
+```bash
+source plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env
+```
+
+Then, we'll run a script to download the benchmark datasets and put it into the appropriate directories:
 
 ```bash
-git init tmp/tau2-bench
-git -C tmp/tau2-bench remote add origin https://github.com/sierra-research/tau2-bench.git
-git -C tmp/tau2-bench fetch --depth 1 origin 8ebb7499622fc2be9b9d510d6f7a7653461f4f29
-git -C tmp/tau2-bench checkout --detach FETCH_HEAD
-uv --directory tmp/tau2-bench sync --frozen
-uv --directory tmp/tau2-bench run --frozen tau2 check-data
-
-export NMP_BASE_URL=http://localhost:8080
-export INFERENCE_API_KEY=sk-...
-export OPENAI_API_KEY="$INFERENCE_API_KEY"
-export OPENAI_API_BASE=https://inference-api.nvidia.com/v1
-
-uv run --directory plugins/nemo-insights --frozen \
-  python -m testbed run tau2-airline \
-  --base "$NMP_BASE_URL" \
-  --set include_rewards=false \
-  --set tau2_repo="$PWD/tmp/tau2-bench"
+plugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-datasets.sh
 ```
 
-The testbed runs the 30-task airline train split and ingests realistic traces into the `tau2-airline` workspace. At this point you can navigate to the traces view and see the traces from the agent.
+Finally, we'll run our agent using the dataset tasks and record the data to NeMo Platform:
+
+```bash
+uv run --frozen plugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py
+```
+
+At this point you can navigate to the [traces tab](http://localhost:8080/studio/workspaces/tau3-airline/intake/traces) and see the traces from the agent.
+
 
 ## 4. Run the Analyst
 
@@ -72,38 +73,28 @@ Let's run it:
 
 ```bash
 uv run --frozen nemo insights analyze \
-  --agent tau2-airline \
-  --workspace tau2-airline \
+  --agent nemo-experimentalist-tau3-nooa \
+  --workspace tau3-airline \
   --base-url "$NMP_BASE_URL"
 ```
 
-This will take a few minutes to run. Once it's done, you can navigate to the insights page to see the issues the analyst discovered in the τ-Bench Airline Agent.
+This will take a few minutes to run. Once it's done, you can navigate to the [insights tab](http://localhost:8080/studio/workspaces/tau3-airline/optimizer) to see the issues the analyst discovered in the τ-Bench Airline Agent.
 
-## 5. Prepare a Tau3 Airline smoke dataset
+## 5. Optimize performance with the experimentalist
 
 The next step is to improve the τ-Bench agent using the experimentalist. The
 experimentalist will run your evals, debug failures using trace data, understand
 the root cause of the failure and attempt to fix it. After it makes the change,
 it will run the evals again to validate whether the change improved the
-performance of your agent on the evalaution.
+performance of your agent on the evaluation.
 
-This is a shortened example that only uses a few tasks, but it can still take up to an hour to finish. First, set your environment variables and set up the task dataset:
+This is a shortened example that only uses a few tasks, but it can still take up
+to an hour to finish. First we will set up a new workspace for the optimization process:
 
 ```bash
-export OPENAI_API_KEY="$INFERENCE_API_KEY"
-export OPENAI_BASE_URL=https://inference-api.nvidia.com/v1
-export TAU2_USER_MODEL=openai/openai/openai/gpt-5-mini
-export TAU2_NL_ASSERTIONS_MODEL=openai/openai/openai/gpt-5-mini
-export AUT_MODEL_NAME=openai/openai/openai/gpt-5-mini
-export EXPERIMENTALIST_SMART_MODEL_NAME=openai/openai/openai/gpt-5-mini
-export EXPERIMENTALIST_MID_MODEL_NAME=openai/openai/openai/gpt-5-mini
-export EXPERIMENTALIST_FAST_MODEL_NAME=openai/openai/openai/gpt-5-mini
-
 uv run --frozen nemo workspaces create canonical-tau3-airline \
   --description "Tau3 Airline Experimentalist runs" \
   --exist-ok
-
-plugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-smoke.sh
 ```
 
 Now we're ready to run the experimentalist!
@@ -113,8 +104,8 @@ uv run --frozen nemo experimentalist run \
   --no-insight \
   --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \
   --agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \
-  --train-dataset plugins/nemo-experimentalist/tmp/tau3-airline-smoke/train \
-  --validation-dataset plugins/nemo-experimentalist/tmp/tau3-airline-smoke/validation \
+  --train-dataset plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \
+  --validation-dataset plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \
   --workspace canonical-tau3-airline \
   --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \
   --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \
@@ -123,6 +114,7 @@ uv run --frozen nemo experimentalist run \
 ```
 
 The trace records include the Experimentalist evaluation ID and Tau3 task ID.
-After the run completes, inspect `eval-and-optimize/run.json` for the selected
-winner and compare the `agent-0` and `agent-1` directories to review the code
-change that was evaluated.
+After the run completes, inspect
+`plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist/eval-and-optimize/run.json`
+for the selected winner and compare the `agent-0` and `agent-1` directories to
+review the code change that was evaluated.

plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.example

@@ -0,0 +1,20 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+export NMP_BASE_URL=http://localhost:8080
+export INFERENCE_API_BASE=https://inference-api.nvidia.com/v1
+export INFERENCE_API_KEY=sk-...
+
+export OPENAI_API_KEY="$INFERENCE_API_KEY"
+export OPENAI_BASE_URL="$INFERENCE_API_BASE"
+export EXPERIMENTALIST_API_KEY="$INFERENCE_API_KEY"
+export EXPERIMENTALIST_API_BASE="$INFERENCE_API_BASE"
+
+# Upstream Tau3 currently reads these compatibility variable names.
+export TAU2_USER_MODEL=openai/openai/openai/gpt-5.6-luna
+export TAU2_NL_ASSERTIONS_MODEL=openai/openai/openai/gpt-5.6-luna
+export AUT_MODEL_NAME=openai/openai/openai/gpt-5.6-luna
+
+export EXPERIMENTALIST_SMART_MODEL_NAME=openai/openai/openai/gpt-5.6-sol
+export EXPERIMENTALIST_MID_MODEL_NAME=openai/openai/openai/gpt-5.6-terra
+export EXPERIMENTALIST_FAST_MODEL_NAME=openai/openai/openai/gpt-5.6-luna

plugins/nemo-experimentalist/examples/tau3-nooa-agent/prepare-airline-datasets.sh

@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+shopt -s nullglob
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+PLUGIN_ROOT="$(cd -- "${SCRIPT_DIR}/../.." && pwd)"
+OUTPUT_ROOT="${1:-${PLUGIN_ROOT}/tmp/tau3-airline}"
+SOURCE_ROOT="${OUTPUT_ROOT}/source"
+DATASET_ROOT="${SOURCE_ROOT}/tau3-bench"
+INSIGHTS_ROOT="${OUTPUT_ROOT}/insights"
+EXPERIMENTALIST_ROOT="${OUTPUT_ROOT}/experimentalist"
+TRAIN_ROOT="${EXPERIMENTALIST_ROOT}/train"
+VALIDATION_ROOT="${EXPERIMENTALIST_ROOT}/validation"
+DATASET_REF="sierra-research/tau3-bench@1"
+
+INSIGHTS_TASKS=(
+  "tau3-bench__tau3-airline-0"
+  "tau3-bench__tau3-airline-1"
+  "tau3-bench__tau3-airline-4"
+  "tau3-bench__tau3-airline-5"
+  "tau3-bench__tau3-airline-9"
+  "tau3-bench__tau3-airline-10"
+  "tau3-bench__tau3-airline-12"
+  "tau3-bench__tau3-airline-14"
+  "tau3-bench__tau3-airline-17"
+  "tau3-bench__tau3-airline-20"
+  "tau3-bench__tau3-airline-23"
+  "tau3-bench__tau3-airline-27"
+  "tau3-bench__tau3-airline-33"
+  "tau3-bench__tau3-airline-34"
+  "tau3-bench__tau3-airline-38"
+  "tau3-bench__tau3-airline-39"
+  "tau3-bench__tau3-airline-41"
+  "tau3-bench__tau3-airline-42"
+  "tau3-bench__tau3-airline-46"
+  "tau3-bench__tau3-airline-47"
+)
+EXPERIMENTALIST_TRAIN_TASKS=(
+  "tau3-bench__tau3-airline-0"
+  "tau3-bench__tau3-airline-20"
+  "tau3-bench__tau3-airline-39"
+)
+EXPERIMENTALIST_VALIDATION_TASKS=(
+  "tau3-bench__tau3-airline-3"
+  "tau3-bench__tau3-airline-36"
+)
+
+if [[ ! -f "${DATASET_ROOT}/${INSIGHTS_TASKS[0]}/task.toml" ]]; then
+  mkdir -p "${SOURCE_ROOT}"
+  (
+    cd "${PLUGIN_ROOT}"
+    uv run --frozen harbor download "${DATASET_REF}" \
+      --output-dir "${SOURCE_ROOT}" \
+      --export \
+      --overwrite
+  )
+fi
+
+validate_source() {
+  local task_names=("$@")
+  for task_name in "${task_names[@]}"; do
+    if [[ ! -f "${DATASET_ROOT}/${task_name}/task.toml" ]]; then
+      echo "Downloaded dataset is missing ${task_name}" >&2
+      exit 1
+    fi
+  done
+}
+
+prepare_split() {
+  local split_root="$1"
+  shift
+  local task_names=("$@")
+
+  if [[ -d "${split_root}" ]]; then
+    local existing_tasks=("${split_root}"/*/task.toml)
+    if [[ ${#existing_tasks[@]} -eq ${#task_names[@]} ]]; then
+      for task_name in "${task_names[@]}"; do
+        if [[ ! -f "${split_root}/${task_name}/task.toml" ]]; then
+          echo "Existing split is not the expected dataset: ${split_root}" >&2
+          exit 1
+        fi
+      done
+      echo "Reusing ${split_root}"
+      return
+    fi
+    echo "Existing split is incomplete: ${split_root}" >&2
+    exit 1
+  fi
+
+  mkdir -p "${split_root}"
+  for task_name in "${task_names[@]}"; do
+    local source_task="${DATASET_ROOT}/${task_name}"
+    cp -R "${source_task}" "${split_root}/${task_name}"
+  done
+}
+
+validate_source \
+  "${INSIGHTS_TASKS[@]}" \
+  "${EXPERIMENTALIST_TRAIN_TASKS[@]}" \
+  "${EXPERIMENTALIST_VALIDATION_TASKS[@]}"
+
+prepare_split "${INSIGHTS_ROOT}" "${INSIGHTS_TASKS[@]}"
+prepare_split "${TRAIN_ROOT}" "${EXPERIMENTALIST_TRAIN_TASKS[@]}"
+prepare_split "${VALIDATION_ROOT}" "${EXPERIMENTALIST_VALIDATION_TASKS[@]}"
+
+echo "Tau3 Airline datasets are ready:"
+echo "  insights:                   ${INSIGHTS_ROOT}"
+echo "  experimentalist train:      ${TRAIN_ROOT}"
+echo "  experimentalist validation: ${VALIDATION_ROOT}"