feat(experimentalist): add the tau3-bench banking benchmark suite

plugins/nemo-experimentalist/AGENTS.md

@@ -72,9 +72,11 @@ instead of restoring Curator imports, configuration, or aliases. The obsolete
   Configure Platform services, trace storage, and Insights analysis according
   to the Platform documentation; this repository does not own a service,
   scheduler, or testbed setup.
-- `nooa` is pinned to a tagged public GitHub release in `pyproject.toml`.
-  Update the tag and lock file together. Keep the Platform-supplied Insights
-  plugin separate.
+- `nooa` is pinned to an immutable public GitHub revision in `pyproject.toml`,
+  currently a commit rather than a tag because the MCP transport-timeout fix
+  landed after `v0.0.6`. Update the revision, the matching pin in
+  `examples/tau3-nooa-agent/pyproject.toml`, and both lock files together. Keep
+  the Platform-supplied Insights plugin separate.
 - This branch uses merged Platform PR 718 contracts only. After the Platform
   handoff lands, rebase and repin before adopting any new Platform testbed or
   installer interfaces.

plugins/nemo-experimentalist/README.md

@@ -28,8 +28,8 @@ export NEMO=/path/to/nemo-platform/.venv/bin/nemo
 ```
 
 The source dependencies are pinned to tagged or immutable revisions in
-`pyproject.toml`. NVIDIA-labs OO Agents (NOOA) is pinned to its public
-GitHub `v0.0.6` release.
+`pyproject.toml`. NVIDIA-labs OO Agents (NOOA) is pinned to a public GitHub
+commit, currently one past `v0.0.6` that carries an MCP transport-timeout fix.
 
 ## Insight-to-experiment flow
 

plugins/nemo-experimentalist/benchmarks/README.md

@@ -1,13 +1,20 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-# Canonical Experimentalist benchmark
+# Canonical Experimentalist benchmarks
 
-This benchmark measures M2 Experimentalist optimization on Harbor's unmodified
-Terminal-Bench 2.1 package. It does not cover M1 Insight → Eval Author behavior
-or Tau2's interactive domains.
+These benchmarks measure M2 Experimentalist optimization on unmodified Harbor Hub
+packages. They do not cover M1 Insight → Eval Author behavior.
 
-## Provenance
+Two suites ship today. Both store only task IDs and download task definitions into
+a local cache; no task content is vendored here.
+
+| Suite | Package | Tasks | Agent under test |
+| --- | --- | --- | --- |
+| `suites/terminal-bench-2.1.yaml` (default) | `terminal-bench/terminal-bench-2-1@6` | 89 | `examples/terminal-bench-agent` |
+| `suites/tau3-banking.yaml` | `sierra-research/tau3-bench@1`, banking scoped | 97 | `examples/tau3-nooa-agent` |
+
+## Terminal-Bench provenance
 
 The suite uses `terminal-bench/terminal-bench-2-1@6` from Harbor Hub:
 
@@ -17,21 +24,41 @@ The suite uses `terminal-bench/terminal-bench-2-1@6` from Harbor Hub:
 - Harbor Hub record:
   <https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1/6>
 
-The repository stores only task IDs. Task definitions are downloaded into a
-local cache and are never vendored here.
-
 The 38/25/26 quality partition and 35/12/12 fast partition came from Gaia's
 `optimization-datasets` commit
 `1b7688ad257dacd1a3267dddb88db6cefdc31376`. The manifest corrects
 `install-windows-3-11` to the canonical ID `install-windows-3.11`. At startup,
 the runner asks Harbor Hub for revision 6, verifies its content hash, and
 asserts that the quality partition covers all 89 canonical IDs exactly once.
 
+## tau3 banking provenance
+
+The suite scopes `sierra-research/tau3-bench@1` to its `banking_knowledge` domain:
+
+- 97 of the package's 375 tasks, selected by the
+  `tau3-bench__tau3-banking_knowledge-` task-ID prefix
+- dataset content hash
+  `sha256:a57304f682894ac061090769af771a3617664f3ff6e5417d4eadf8e30433e4d9`
+- Harbor Hub record:
+  <https://hub.harborframework.com/datasets/sierra-research/tau3-bench/1>
+
+The 41/28/28 quality partition came from `optimization-datasets`
+`feat/tau2-other-domains` commit `025ecd2ef2b518ad81f6b22d3f3937af8906fb01`,
+whose `tau2-banking-knowledge-NNN` names map onto the canonical
+`tau3-bench__tau3-banking_knowledge-task-NNN` IDs. The 6/3/3 fast partition draws
+only from the 87 tasks whose `reward_basis` is pure database state, so smoke runs
+score deterministically without an LLM judge.
+
+Each task also runs a `tau3-runtime` sidecar hosting the tau2 environment and user
+simulator. Tau-style suites set `models.user_simulator`, which makes the runner
+export `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `TAU2_USER_MODEL`, and
+`TAU2_NL_ASSERTIONS_MODEL` for the sidecar and the verifier.
+
 ## Held-out evaluation
 
 The runner:
 
-1. evaluates the unchanged LangChain baseline on the test split;
+1. evaluates the unchanged baseline agent on the test split;
 2. gives only train and validation IDs to `run_experimentalist`;
 3. resolves the validation-selected winner from the Experimentalist run;
 4. evaluates that winner on the same test IDs and number of attempts.
@@ -52,7 +79,7 @@ export INFERENCE_API_KEY=...
 ```
 
 `EXPERIMENTALIST_API_KEY` may be provided instead. The runner maps either credential
-to both the LangChain AUT and optimizer, using
+to both the AUT and optimizer, using
 `https://inference-api.nvidia.com/v1` unless an API base is explicitly set.
 
 Validate provenance and task IDs without Docker or model calls:
@@ -65,20 +92,28 @@ Run the bounded fast benchmark:
 
 ```bash
 uv run python benchmarks/run.py \
-  --config benchmarks/configs/smoke.yaml
+  --config benchmarks/configs/terminal-bench-smoke.yaml
 ```
 
 Run the reproducible quality benchmark:
 
 ```bash
 uv run python benchmarks/run.py \
-  --config benchmarks/configs/quality.yaml
+  --config benchmarks/configs/terminal-bench-quality.yaml
+```
+
+A non-default suite needs its own suite, config, and agent:
+
+```bash
+uv run python benchmarks/run.py \
+  --suite benchmarks/suites/tau3-banking.yaml \
+  --config benchmarks/configs/tau3-smoke.yaml \
+  --agent examples/tau3-nooa-agent
 ```
 
-Both setup and agent execution require network access. The AUT installs a
-checksum-pinned static `uv`, uv-managed Python 3.12, and dependencies from its
-committed `uv.lock` directly in each canonical task container. It does not need
-the image's system Python, package manager, a sidecar, or a Docker socket.
+Both setup and agent execution require network access. Each AUT installs its own
+dependencies from its committed `uv.lock` inside the task container and does not
+need a Docker socket.
 
 Pass the same `--output` directory to resume interrupted Harbor jobs. Use a new
 output directory for an intentionally fresh run.

plugins/nemo-experimentalist/benchmarks/configs/tau3-quality.yaml

@@ -0,0 +1,34 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+suite_partition: quality
+test_attempts: 3
+models:
+  aut: openai/openai/openai/gpt-5-mini
+  experimentalist_smart: openai/openai/openai/gpt-5-mini
+  experimentalist_mid: openai/openai/openai/gpt-5-mini
+  experimentalist_fast: openai/openai/openai/gpt-5-mini
+  user_simulator: openai/openai/openai/gpt-5-mini
+optimizer:
+  max_rounds: 3
+  min_rounds_before_stopping: 2
+  max_survivors: 2
+  max_candidates: 3
+  max_trajectory_tasks: 8
+  max_train_batch_tasks: 16
+  train_batch_seed: 20260727
+  disable_trajectory_scoring: false
+  disable_convergence_check: false
+  evaluator:
+    n_attempts: 2
+    # Keep at 3 or lower: each task requests 8192 MB and parallel image builds
+    # have triggered Docker Hub rate limiting and DNS failures.
+    n_concurrent_trials: 3
+    quiet: true
+    agent_setup_timeout_multiplier: 2.0
+    # tau3 builds two images per task, and the first build clones and installs
+    # tau2-bench (observed 2m36s cold, 26s warm).
+    environment_build_timeout_multiplier: 3.0
+  eval_author:
+    max_traces: 10
+    max_validation_repair_attempts: 5

plugins/nemo-experimentalist/benchmarks/configs/tau3-smoke.yaml

@@ -0,0 +1,32 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+suite_partition: fast
+test_attempts: 1
+models:
+  aut: openai/openai/openai/gpt-5-mini
+  experimentalist_smart: openai/openai/openai/gpt-5-mini
+  experimentalist_mid: openai/openai/openai/gpt-5-mini
+  experimentalist_fast: openai/openai/openai/gpt-5-mini
+  user_simulator: openai/openai/openai/gpt-5-mini
+optimizer:
+  max_rounds: 1
+  min_rounds_before_stopping: 1
+  max_survivors: 1
+  max_candidates: 1
+  max_trajectory_tasks: 2
+  max_train_batch_tasks: 4
+  train_batch_seed: 20260727
+  disable_trajectory_scoring: true
+  disable_convergence_check: true
+  evaluator:
+    n_attempts: 1
+    n_concurrent_trials: 1
+    quiet: true
+    agent_setup_timeout_multiplier: 2.0
+    # tau3 builds two images per task, and the first build clones and installs
+    # tau2-bench (observed 2m36s cold, 26s warm).
+    environment_build_timeout_multiplier: 3.0
+  eval_author:
+    max_traces: 3
+    max_validation_repair_attempts: 2

plugins/nemo-experimentalist/benchmarks/run.py

@@ -1,7 +1,7 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-"""Run the canonical Terminal-Bench Experimentalist benchmark."""
+"""Run a canonical Experimentalist benchmark suite."""
 
 import argparse
 import asyncio
@@ -32,7 +32,7 @@
 PLUGIN_ROOT = Path(__file__).resolve().parents[1]
 BENCHMARK_ROOT = Path(__file__).resolve().parent
 DEFAULT_SUITE = BENCHMARK_ROOT / "suites" / "terminal-bench-2.1.yaml"
-DEFAULT_CONFIG = BENCHMARK_ROOT / "configs" / "smoke.yaml"
+DEFAULT_CONFIG = BENCHMARK_ROOT / "configs" / "terminal-bench-smoke.yaml"
 DEFAULT_AGENT = PLUGIN_ROOT / "examples" / "terminal-bench-agent"
 DEFAULT_DATASET_CACHE = PLUGIN_ROOT / "tmp" / "benchmark-datasets"
 DEFAULT_RUNTIME_CACHE = PLUGIN_ROOT / "tmp" / "runtime-cache"
@@ -45,6 +45,7 @@ class CanonicalDatasetSpec(BaseModel):
     registry_url: str
     source_url: str
     expected_task_count: int = Field(gt=0)
+    task_id_prefix: str | None = None
 
     @property
     def requested_reference(self) -> str:
@@ -88,13 +89,44 @@ class SuiteSpec(BaseModel):
     schema_version: Literal[1]
     dataset: CanonicalDatasetSpec
     partitions: PartitionsSpec
+    workspace: str
+    framework_skills: list[str] = Field(min_length=1)
+
+    @model_validator(mode="after")
+    def expand_partition_task_ids(self) -> Self:
+        """Read partition entries as names inside the suite's ``task_id_prefix``.
+
+        Every ID in a domain-scoped package repeats that prefix, which buries the part
+        that differs. Manifests write the distinguishing name and the prefix is joined
+        back on here, so the rest of the runner still works in canonical IDs. A manifest
+        that writes full IDs anyway fails the quality-coverage check with both forms in
+        the message.
+        """
+        prefix = self.dataset.task_id_prefix
+        if prefix is None:
+            return self
+        for split in (self.partitions.quality, self.partitions.fast):
+            for role in ("train", "validation", "test"):
+                setattr(split, role, [f"{prefix}{name}" for name in getattr(split, role)])
+        return self
+
+    def framework_skills_dirs(self, plugin_root: Path) -> list[Path]:
+        """Resolve framework-skill names to directories, failing on unknown names."""
+        dirs: list[Path] = []
+        for name in self.framework_skills:
+            path = plugin_root / "framework-skills" / name
+            if not path.is_dir():
+                raise ValueError(f"Unknown framework skill {name!r}: {path} is not a directory")
+            dirs.append(path)
+        return dirs
 
 
 class ModelSpec(BaseModel):
     aut: str
     experimentalist_smart: str
     experimentalist_mid: str
     experimentalist_fast: str
+    user_simulator: str | None = None
 
 
 class BenchmarkConfig(BaseModel):
@@ -126,13 +158,23 @@ def validate_canonical_suite(
     *,
     canonical_task_ids: set[str],
     resolved_ref: str,
-) -> None:
-    """Verify immutable revision, quality coverage, and all partition IDs."""
+) -> set[str]:
+    """Verify immutable revision, quality coverage, and all partition IDs.
+
+    Returns the canonical task IDs scoped to this suite's domain. When the suite
+    sets ``task_id_prefix``, the package may hold other domains; coverage is then
+    enforced against the matching subset rather than the whole package.
+    """
     if resolved_ref != suite.dataset.resolved_ref:
         raise ValueError(
             f"Dataset {suite.dataset.requested_reference} resolved to {resolved_ref}, "
             f"expected {suite.dataset.resolved_ref}"
         )
+    prefix = suite.dataset.task_id_prefix
+    if prefix is not None:
+        canonical_task_ids = {task_id for task_id in canonical_task_ids if task_id.startswith(prefix)}
+        if not canonical_task_ids:
+            raise ValueError(f"No canonical tasks match task_id_prefix {prefix!r}")
     if len(canonical_task_ids) != suite.dataset.expected_task_count:
         raise ValueError(
             f"Canonical dataset has {len(canonical_task_ids)} tasks, expected {suite.dataset.expected_task_count}"
@@ -147,6 +189,7 @@ def validate_canonical_suite(
     unknown_fast = sorted(set(suite.partitions.fast.all_ids()) - canonical_task_ids)
     if unknown_fast:
         raise ValueError(f"Fast partition contains unknown canonical task IDs: {unknown_fast}")
+    return canonical_task_ids
 
 
 def _configure_models(models: ModelSpec) -> None:
@@ -164,6 +207,16 @@ def _configure_models(models: ModelSpec) -> None:
     os.environ["EXPERIMENTALIST_MID_MODEL_NAME"] = models.experimentalist_mid
     os.environ["EXPERIMENTALIST_FAST_MODEL_NAME"] = models.experimentalist_fast
     os.environ.setdefault("NEMO_EXPERIMENTALIST_RUNTIME_CACHE", str(DEFAULT_RUNTIME_CACHE))
+    if models.user_simulator is not None:
+        # tau-style tasks run a user simulator and NL-assertion judge inside the task
+        # environment, which read OpenAI-style variables and need the /v1 suffix.
+        # Assigned rather than setdefault: an ambient OPENAI_API_KEY from the developer's
+        # shell would otherwise be paired with this gateway base URL and always 401.
+        trimmed_base = api_base.rstrip("/")
+        os.environ["OPENAI_API_KEY"] = api_key
+        os.environ["OPENAI_BASE_URL"] = trimmed_base if trimmed_base.endswith("/v1") else f"{trimmed_base}/v1"
+        os.environ["TAU2_USER_MODEL"] = models.user_simulator
+        os.environ["TAU2_NL_ASSERTIONS_MODEL"] = models.user_simulator
 
 
 def _agent_digest(agent_dir: Path) -> str:
@@ -342,7 +395,12 @@ async def run_benchmark(args: argparse.Namespace) -> Path:
     package_client = PackageDatasetClient()
     metadata = await package_client.get_dataset_metadata(suite.dataset.requested_reference)
     canonical_task_ids = {task.name for task in metadata.task_ids}
-    validate_canonical_suite(suite, canonical_task_ids=canonical_task_ids, resolved_ref=metadata.version)
+    canonical_task_ids = validate_canonical_suite(
+        suite, canonical_task_ids=canonical_task_ids, resolved_ref=metadata.version
+    )
+    # Resolved up front: the optimizer only needs these after the baseline evaluation,
+    # which is hours of image builds to discover a typo'd skill name.
+    framework_skills_dirs = suite.framework_skills_dirs(PLUGIN_ROOT)
 
     if args.validate_only:
         print(
@@ -397,10 +455,10 @@ async def run_benchmark(args: argparse.Namespace) -> Path:
             metadata={"id": f"{benchmark_config.suite_partition}-validation", "task_ids": split.validation},
         ),
         experiment_dir=experimentalist_dir,
-        workspace="canonical-terminal-bench-2-1",
+        workspace=suite.workspace,
         client=None,
         config=benchmark_config.optimizer,
-        framework_skills_dirs=[PLUGIN_ROOT / "framework-skills" / "langchain-framework"],
+        framework_skills_dirs=framework_skills_dirs,
     )
     run_document = json.loads((experimentalist_dir / "eval-and-optimize" / "run.json").read_text(encoding="utf-8"))
     winner_label = run_document.get("winner_agent")