NVIDIA-NeMo · gaiadilorenzo · Aug 5, 2026 · Aug 4, 2026 · Aug 4, 2026 · Aug 4, 2026
@@ -64,6 +64,10 @@ Finally, we'll run our agent using the dataset tasks and record the data to NeMo
 uv run --frozen plugins/nemo-experimentalist/examples/tau3-nooa-agent/record_tau_airline_traces.py
 ```
 
+To run more tasks concurrently, add an explicit value such as `--concurrency 10`.
+Each task starts Docker containers, so choose a value appropriate for your
+available CPU and memory.
+
 At this point you can navigate to the [traces tab](http://localhost:8080/studio/workspaces/tau3-airline/intake/traces) and see the traces from the agent.
 
 
@@ -101,13 +105,102 @@ uv run --frozen nemo workspaces create canonical-tau3-airline \
   --exist-ok
 ```
 
-Run the experimentalist in a Docker Sandbox so that both the optimization
-process and Harbor's task containers use the sandbox's microVM and private
-Docker daemon. Clone mode gives the experimentalist a private writable clone
-instead of write access to the host checkout. The host checkout remains
-available read-only at `/run/sandbox/source`; the prepared evaluation dataset
-is read from there. The sandbox uses `host.docker.internal` to reach the NeMo
-Platform services running on the host:
+The Experimentalist generates a draft pull request (PR) or merge request (MR)
+when it receives a GitHub or GitLab agent source. It creates a branch only when
+a changed candidate wins validation.
+
+Git integration is optional. To run the optimization without creating a
+repository or PR/MR, [jump to the local-only alternative](#optional-run-locally-without-creating-a-pr-or-mr).
+
+To optimize the example agent, create a private GitHub repository with the
+[GitHub CLI](https://cli.github.com/) and populate it with the example source.
+Replace `your-github-user-or-org` with the GitHub account or organization that
+will own the repository. `rsync` excludes `.env`, so your API key is not copied
+into the repository.
+
+```bash
+gh auth login
+# Choose HTTPS when prompted for the preferred Git protocol.
+export GITHUB_OWNER="your-github-user-or-org"
+mkdir -p "$HOME/src"
+cd "$HOME/src"
+gh repo create "$GITHUB_OWNER/tau3-nooa-agent" --private --clone
+export EXAMPLE_AGENT_REPO="$PWD/tau3-nooa-agent"
+cd -
+rsync -a --exclude='.env' plugins/nemo-experimentalist/examples/tau3-nooa-agent/ "$EXAMPLE_AGENT_REPO/"
+git -C "$EXAMPLE_AGENT_REPO" checkout -b main
+git -C "$EXAMPLE_AGENT_REPO" add .
+git -C "$EXAMPLE_AGENT_REPO" commit -m "Add the tau3 NOOA example agent"
+git -C "$EXAMPLE_AGENT_REPO" push -u origin main
+export AGENT_REPO_URL="https://github.com/$GITHUB_OWNER/tau3-nooa-agent.git"
+```
+
+Run the Experimentalist against the Git repository inside a Docker Sandbox. The
+`@main` suffix selects the source ref to optimize. It clones that baseline,
+pushes a validated winner to a new branch, and opens a draft PR or MR against
+`main`. To use a different target branch, set `storage.pr_base_branch` in the
+configuration. Authenticate GitHub inside the sandbox so it can clone and push
+the private repository.
+
+```bash
+repo="$(git rev-parse --show-toplevel)"
+sbx create --clone --name nemo-experimentalist-git shell "$repo"
+gh auth token | sbx exec -i --workdir "$repo" nemo-experimentalist-git \
+  gh auth login --with-token
+sbx exec --workdir "$repo" nemo-experimentalist-git gh auth setup-git
+sbx exec --workdir "$repo" nemo-experimentalist-git \
+  git clone "$AGENT_REPO_URL" /tmp/tau3-nooa-agent
+sbx exec --workdir "$repo" \
+  --env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \
+  --env INFERENCE_API_KEY \
+  --env INFERENCE_API_BASE \
+  --env OPENAI_API_KEY \
+  --env OPENAI_BASE_URL \
+  --env NEMO_EXPERIMENTALIST_API_KEY \
+  --env NEMO_EXPERIMENTALIST_API_BASE \
+  --env NEMO_EXPERIMENTALIST_MODELS_SMART \
+  --env NEMO_EXPERIMENTALIST_MODELS_MID \
+  --env NEMO_EXPERIMENTALIST_MODELS_FAST \
+  --env TAU2_USER_MODEL \
+  --env TAU2_NL_ASSERTIONS_MODEL \
+  --env AUT_MODEL_NAME \
+  nemo-experimentalist-git \
+  uv run --frozen --python 3.13 --package nemo-experimentalist-plugin --with ./plugins/nemo-agents \
+  nemo agents experimentalist run \
+  --no-insight \
+  --agent "${AGENT_REPO_URL}@main" \
+  --agent-spec /tmp/tau3-nooa-agent/AGENT-SPEC.md \
+  --train-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \
+  --validation-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \
+  --workspace canonical-tau3-airline \
+  --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \
+  --config /tmp/tau3-nooa-agent/experimentalist-smoke.yaml \
+  --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \
+  --base-url http://host.docker.internal:8080
+```
+
+This setup enables `storage.archive_candidates`, which pushes every generated
+candidate branch; the default pushes only the winner. After the run, inspect a
+non-winning candidate with `git fetch origin`,
+`git -C "$EXAMPLE_AGENT_REPO" branch -r --list 'origin/optimizer/*'`, and
+`git diff main...origin/optimizer/<run-id>/<candidate>`.
+
+GitLab is supported too: use a GitLab repository URL and authenticate with
+`glab auth login`. The Experimentalist opens a draft PR/MR only when it finds a
+changed winning candidate.
+
+The trace records include the Experimentalist evaluation ID and Tau3 task ID.
+After the run completes, inspect the sandbox's
+`plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist/eval-and-optimize/run.json`
+for the selected winner and compare the `agent-0` and `agent-1` directories to
+review the code change that was evaluated.
+
+### Optional: run locally without creating a PR or MR
+
+Use this alternative if you do not want to create or push to a repository. It
+changes only a private sandbox clone and never creates a PR/MR. The sandbox
+uses `host.docker.internal` to reach the NeMo Platform services running on the
+host:
 
 ```bash
 repo="$(git rev-parse --show-toplevel)"
@@ -127,7 +220,7 @@ sbx exec --workdir "$repo" \
   --env TAU2_NL_ASSERTIONS_MODEL \
   --env AUT_MODEL_NAME \
   nemo-experimentalist \
-  uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \
+  uv run --frozen --python 3.13 --package nemo-experimentalist-plugin --with ./plugins/nemo-agents \
   nemo agents experimentalist run \
   --no-insight \
   --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \
@@ -140,7 +233,6 @@ sbx exec --workdir "$repo" \
   --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \
   --base-url http://host.docker.internal:8080
 ```
-
 The sandbox needs outbound access to the package, model, registry, Harbor
 dataset, and NeMo Platform endpoints used by the run. `host.docker.internal`
 is translated to the host's loopback interface so the sandbox can reach the

@@ -263,9 +263,9 @@ def result_manager_factory(
         job_name=job_name,
         workspace=workspace,
         attempt_id=attempt_id,
-        file_manager_cls=file_manager_cls,  # type: ignore
-        files_sdk=files_sdk,  # type: ignore
-        jobs_sdk=jobs_sdk,  # type: ignore
+        file_manager_cls=file_manager_cls,
+        files_sdk=files_sdk,
+        jobs_sdk=jobs_sdk,
     )
 
 

@@ -81,7 +81,7 @@ sbx exec --workdir "$repo" \
   --env NEMO_EXPERIMENTALIST_MODELS_MID \
   --env NEMO_EXPERIMENTALIST_MODELS_FAST \
   nemo-experimentalist \
-  uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \
+  uv run --frozen --python 3.13 --package nemo-experimentalist-plugin --with ./plugins/nemo-agents \
   nemo agents experimentalist run
 ```
 

@@ -0,0 +1,8 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+# Local inference credentials
+.env
+
+# Generated runtime artifacts
+metadata.json
+__pycache__/
@@ -10,9 +10,10 @@ max_train_batch_tasks: 4
 train_batch_seed: 20260727
 disable_trajectory_scoring: true
 disable_convergence_check: true
+storage:
+  archive_candidates: true
 evaluator:
   n_attempts: 1
-  n_concurrent_trials: 1
   quiet: true
   agent_setup_timeout_multiplier: 2.0
   environment_build_timeout_multiplier: 3.0

@@ -2,7 +2,11 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-"""Run the Tau3 Airline agent and upload only its execution traces to Intake."""
+"""Example setup helper: run Tau3 Airline and upload its traces to Intake.
+
+This script supplies trace data for the walkthrough; Experimentalist does not require
+agent repositories to include it.
+"""
 
 import argparse
 import asyncio
@@ -58,6 +62,72 @@ def _configure_models(*, model: str, user_model: str, api_base: str) -> None:
     os.environ["TAU2_NL_ASSERTIONS_MODEL"] = user_model
 
 
+def _resolve_harbor_output_dir(path: Path) -> tuple[Path, Path]:
+    """Return the Harbor job directory and its enclosing run directory."""
+    candidate = path.expanduser().resolve()
+    if not candidate.is_dir():
+        raise FileNotFoundError(f"Harbor output directory not found: {candidate}")
+
+    if any(child.is_dir() and (child / "result.json").is_file() for child in candidate.iterdir()):
+        job_dir = candidate
+        run_dir = candidate.parent.parent if candidate.parent.name == "results" else candidate
+    else:
+        jobs_dir = candidate / "results"
+        job_dirs = (
+            [
+                child
+                for child in jobs_dir.iterdir()
+                if child.is_dir()
+                and any(trial.is_dir() and (trial / "result.json").is_file() for trial in child.iterdir())
+            ]
+            if jobs_dir.is_dir()
+            else []
+        )
+        if len(job_dirs) != 1:
+            raise ValueError(f"{candidate} is not a Harbor output directory with exactly one job under results/")
+        job_dir = job_dirs[0]
+        run_dir = candidate
+
+    if not any(child.is_dir() and (child / "result.json").is_file() for child in job_dir.iterdir()):
+        raise ValueError(f"Harbor job directory contains no trial result files: {job_dir}")
+    return job_dir, run_dir
+
+
+def _write_upload_summary(
+    run_dir: Path,
+    *,
+    experiment_id: str,
+    workspace: str,
+    agent_name: str,
+    agent_version: str,
+    model: str,
+    trials: list[TrialResult],
+    trace_ids: dict[str, str],
+) -> Path:
+    summary_path = run_dir / "uploaded-traces.json"
+    summary_path.write_text(
+        json.dumps(
+            {
+                "experiment_id": experiment_id,
+                "workspace": workspace,
+                "agent_name": agent_name,
+                "agent_version": agent_version,
+                "model": model,
+                "trace_count": len(trace_ids),
+                "traces": [
+                    {"trial_id": trial.id, "task_id": trial.task_id, "trace_id": trace_ids[trial.id]}
+                    for trial in trials
+                ],
+            },
+            indent=2,
+            sort_keys=True,
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    return summary_path
+
+
 async def _upload_trials(
     client: AsyncNeMoPlatform,
     trials: list[TrialResult],
@@ -137,11 +207,54 @@ async def run(args: argparse.Namespace) -> Path:
     dataset = HarborDataset.from_path(dataset_path)
     if args.task_ids:
         dataset = dataset.subset(args.task_ids)
-    if len(dataset.tasks) != args.expected_task_count:
+    if args.upload_dir is None and len(dataset.tasks) != args.expected_task_count:
         raise RuntimeError(
             f"Expected {args.expected_task_count} Tau3 tasks in {dataset_path}, found {len(dataset.tasks)}"
         )
 
+    if args.upload_dir is not None:
+        job_dir, run_dir = _resolve_harbor_output_dir(args.upload_dir)
+        evaluator = HarborEvaluator(experiment_dir=run_dir)
+        trials = list(await evaluator._trials_from_dir(job_dir, dataset.tasks))
+        uploadable_trials = [trial for trial in trials if trial.status == "completed" and trial.trace is not None]
+        if not uploadable_trials:
+            raise RuntimeError(f"No completed Harbor trials with trace artifacts found in {job_dir}")
+
+        experiment_id = args.experiment_id or run_dir.name
+        client = make_client(args.base_url)
+        try:
+            await client.workspaces.create(
+                name=args.workspace,
+                description="Tau3 Airline agent traces for Insights",
+                exist_ok=True,
+            )
+            trace_ids = await _upload_trials(
+                client,
+                uploadable_trials,
+                workspace=args.workspace,
+                experiment_id=experiment_id,
+                agent_name=args.agent_name,
+                agent_version=args.agent_version,
+                model=args.model,
+            )
+            await _wait_for_traces(client, set(trace_ids.values()), workspace=args.workspace)
+        finally:
+            await client.close()
+
+        summary_path = _write_upload_summary(
+            run_dir,
+            experiment_id=experiment_id,
+            workspace=args.workspace,
+            agent_name=args.agent_name,
+            agent_version=args.agent_version,
+            model=args.model,
+            trials=uploadable_trials,
+            trace_ids=trace_ids,
+        )
+        print(f"Uploaded and verified {len(trace_ids)} agent traces in workspace {args.workspace!r}.")
+        print(summary_path)
+        return summary_path
+
     experiment_id = args.experiment_id or _experiment_id()
     run_dir = args.output.expanduser().resolve() / experiment_id
     if run_dir.exists():
@@ -190,30 +303,15 @@ async def run(args: argparse.Namespace) -> Path:
     finally:
         await client.close()
 
-    summary_path = run_dir / "uploaded-traces.json"
-    summary_path.write_text(
-        json.dumps(
-            {
-                "experiment_id": experiment_id,
-                "workspace": args.workspace,
-                "agent_name": args.agent_name,
-                "agent_version": args.agent_version,
-                "model": args.model,
-                "trace_count": len(trace_ids),
-                "traces": [
-                    {
-                        "trial_id": trial.id,
-                        "task_id": trial.task_id,
-                        "trace_id": trace_ids[trial.id],
-                    }
-                    for trial in trials
-                ],
-            },
-            indent=2,
-            sort_keys=True,
-        )
-        + "\n",
-        encoding="utf-8",
+    summary_path = _write_upload_summary(
+        run_dir,
+        experiment_id=experiment_id,
+        workspace=args.workspace,
+        agent_name=args.agent_name,
+        agent_version=args.agent_version,
+        model=args.model,
+        trials=trials,
+        trace_ids=trace_ids,
     )
     print(f"Uploaded and verified {len(trace_ids)} agent traces in workspace {args.workspace!r}.")
     print(summary_path)
@@ -226,6 +324,11 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--agent", type=Path, default=SCRIPT_DIR)
     parser.add_argument("--workspace", default=DEFAULT_WORKSPACE)
     parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
+    parser.add_argument(
+        "--upload-dir",
+        type=Path,
+        help="Existing Harbor run directory or Harbor job-results directory to upload without rerunning trials.",
+    )
     parser.add_argument("--base-url", default=os.environ.get("NMP_BASE_URL", "http://localhost:8080"))
     parser.add_argument(
         "--api-base",

@@ -274,7 +274,7 @@ async def _flow() -> str:
                         profile,
                         task_template=plan.task_template,
                         agent_source=plan.agent,
-                        storage=plan.config.storage.model_dump(),
+                        storage=plan.config.storage.model_dump(exclude_unset=True),
                         require_template=plan.insight is not None,
                         probes=_PREFLIGHT_PROBES,
                     )

@@ -43,7 +43,7 @@ class CandidateStorageConfig(BaseModel):
 
     archive_candidates: bool = False
     candidate_branch_prefix: str = "optimizer"
-    publish_winner: bool = False
+    publish_winner: bool = True
     pr_draft: bool = True
     pr_base_branch: str | None = None
     pr_title: str | None = None