refactor(evaluator)!: make persisting an agent-eval run an explicit call

docs/evaluator/agent-eval/harbor-runner.mdx

@@ -135,8 +135,8 @@ Swap `agent_name` (or `agent_import_path`) for your own agent to get a real scor
 2. The runner reads that reward onto each trial's metadata, and `HarborRewardMetric` scores it — the
    `reward` value, or `0.0` for a trial whose verifier emitted none.
 3. `result.summary` aggregates the reward across tasks (`harbor_reward.reward`), `result.trials` holds
-   each trial's status and evidence, and — if you pass a `config` with an `output_dir` to `run()` — the
-   run **bundle** (including `report.html`) is written like any other agent-eval run.
+   each trial's status and evidence, and — if you call `result.persist()` — the run **bundle**
+   (including `report.html`) is written like any other agent-eval run.
 
 ## Attempts, concurrency, and caching
 

docs/evaluator/agent-eval/quickstart.mdx

@@ -106,8 +106,8 @@ async def my_agent(task: AgentEvalTask) -> str:
 ## 4. Run the evaluation
 
 `AgentEvaluator.run()` sends the tasks to the runner, collects the trials, scores them, and returns
-an `AgentEvalResult`. Setting `output_dir` also writes a run bundle to disk. (`run()` is async, so it
-lives inside an `async` function — see the full script below.)
+an `AgentEvalResult`. It writes nothing on its own — call `result.persist()` to store a run bundle to
+disk. (`run()` is async, so it lives inside an `async` function — see the full script below.)
 
 ```python
 from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
@@ -117,8 +117,9 @@ from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
 result = await AgentEvaluator().run(
     tasks=make_tasks(),
     target=CallableAgentTaskRunner(my_agent),
-    config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
+    config=AgentEvalRunConfig(work_dir="./agent-eval-run", parallelism=2),
 )
+result.persist()
 
 for aggregate in result.summary.scores.scores:
     print(f"{aggregate.name}: {aggregate.mean}")
@@ -135,7 +136,8 @@ for aggregate in result.summary.scores.scores:
   state, logs), and its status (`completed` / `partial` / `failed`).
 - `result.run_id` — a stable identifier for this run.
 
-Because you set `output_dir`, the same data was also written to `./agent-eval-run/` as a run bundle:
+Because you called `persist()`, the same data was also written to `./agent-eval-run/` as a run bundle
+(it defaults to `work_dir`):
 
 | File | Contents |
 |---|---|
@@ -144,7 +146,7 @@ Because you set `output_dir`, the same data was also written to `./agent-eval-ru
 | `trials.jsonl` | one row per trial — the agent's output, its evidence, and status |
 | `tasks.jsonl` | the tasks that were evaluated |
 | `run.json` | the run manifest — the run id and a map of the artifact files |
-| `benchmark.json` | benchmark-grouping metadata for the run |
+| `metadata.json` | run provenance — labels, target identity, timings, SDK version |
 | `report.html` | a browsable dashboard of the run — open it in a browser |
 
 The in-memory result and the on-disk bundle hold the same information: use the result object for
@@ -209,8 +211,9 @@ async def main() -> None:
     result = await AgentEvaluator().run(
         tasks=make_tasks(),
         target=CallableAgentTaskRunner(my_agent),
-        config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
+        config=AgentEvalRunConfig(work_dir="./agent-eval-run", parallelism=2),
     )
+    result.persist()
     for aggregate in result.summary.scores.scores:
         print(f"{aggregate.name}: {aggregate.mean}")
 

docs/evaluator/agent-eval/reading-results.mdx

@@ -1,11 +1,11 @@
 ---
 title: "Reading Results"
-description: "Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle it writes when you set output_dir, including the browsable HTML report."
+description: "Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle you get by calling persist(), including the browsable HTML report."
 ---
 
-`AgentEvaluator().run(...)` returns an `AgentEvalResult`. Set an `output_dir` and it *also* writes a
-**run bundle** to disk. The object and the bundle hold the same data — use the object for programmatic
-follow-up, and the bundle (especially `report.html`) to inspect or share a run.
+`AgentEvaluator().run(...)` returns an `AgentEvalResult` and writes nothing. Call `result.persist()` to
+store it as a **run bundle** on disk. The object and the bundle hold the same data — use the object for
+programmatic follow-up, and the bundle (especially `report.html`) to inspect or share a run.
 
 ## The result object
 
@@ -20,7 +20,7 @@ result = await AgentEvaluator().run(tasks=..., target=...)
 | `result.scores` | one entry per **(task, trial, metric)** |
 | `result.trials` | one entry per **trial** |
 | `result.tasks` | the tasks that were evaluated |
-| `result.output_dir` / `result.dashboard_path` | where the bundle and `report.html` were written (when `output_dir` was set) |
+| `result.work_dir` | the directory the run worked in, where runtimes wrote trial evidence (`None` for an in-memory run) |
 
 ### The summary
 
@@ -53,7 +53,7 @@ are the durable, scorer-agnostic record — they can be re-scored offline later.
 
 ## The run bundle
 
-Set `output_dir` and `run()` writes these files (the same data, on disk):
+Call `result.persist()` and it writes these files (the same data, on disk):
 
 | File | Contents |
 |---|---|
@@ -62,26 +62,40 @@ Set `output_dir` and `run()` writes these files (the same data, on disk):
 | `scores.jsonl` | one row per (task, trial, metric) |
 | `trials.jsonl` | one row per trial — output, evidence, status |
 | `tasks.jsonl` | the tasks that were evaluated |
-| `benchmark.json` | benchmark-grouping metadata for the run |
+| `metadata.json` | run provenance — labels, target identity, timings, SDK version |
 | `report.html` | a browsable dashboard — open it in a browser |
 
+`persist()` returns a `BundleLocation` telling you where the bundle landed:
+
 ```python
 from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
 
 result = await AgentEvaluator().run(
-    tasks=..., target=..., config=AgentEvalRunConfig(output_dir="./agent-eval-run"),
+    tasks=..., target=..., config=AgentEvalRunConfig(work_dir="./agent-eval-run"),
 )
+location = result.persist()
 # -> ./agent-eval-run/report.html, summary.json, scores.jsonl, trials.jsonl, ...
+print(location.output_dir, location.dashboard_path)
 ```
 
 `report.html` is the fastest way to eyeball a run or hand it to someone else; the `.jsonl` files are
 convenient for loading scores and trials into your own tooling.
 
 <Note>
 
-`report.html` is written only when `AgentEvalRunConfig.write_dashboard` is `True` (the default). Set
-`write_dashboard=False` to emit just the JSON/JSONL artifacts and skip the HTML; the `.json` and
-`.jsonl` files are always written whenever `output_dir` is set.
+`persist()` defaults to `work_dir`, which is where the run's trial evidence already lives — that keeps
+the bundle self-contained, so it survives being moved or copied. Passing an explicit
+`persist("./elsewhere")` is supported, but the bundle's evidence references still point back at the
+original directory and only resolve while it exists.
+
+A run with no `work_dir` and no explicit target raises rather than inventing a directory.
+
+</Note>
+
+<Note>
+
+`report.html` is written unless you pass `persist(write_dashboard=False)`, which emits just the
+JSON/JSONL artifacts and skips the HTML. The `.json` and `.jsonl` files are always written.
 
 </Note>
 

packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb

@@ -599,13 +599,16 @@
     "result = evaluator.run_sync(\n",
     "    tasks=suite.tasks,\n",
     "    target=target,\n",
-    "    config=AgentEvalRunConfig(output_dir=OUTPUT_DIR, write_dashboard=True, parallelism=3),\n",
+    "    config=AgentEvalRunConfig(work_dir=OUTPUT_DIR, parallelism=3),\n",
     ")\n",
     "\n",
+    "# Storing the run is its own step; it defaults to the work_dir the config named.\n",
+    "location = result.persist()\n",
+    "\n",
     "print(\"run_id    :\", result.run_id)\n",
     "print(\"tasks     :\", result.summary.task_count)\n",
     "print(\"trials    :\", result.summary.trial_count)\n",
-    "print(\"dashboard :\", result.dashboard_path)"
+    "print(\"dashboard :\", location.dashboard_path)"
    ]
   },
   {
@@ -685,7 +688,7 @@
    "cell_type": "markdown",
    "id": "cell-34",
    "metadata": {},
-   "source": "To see **what the agent actually did** — its output, the files it changed, its step-by-step trajectory\n— open the HTML dashboard at `result.dashboard_path`, or read the persisted bundle under\n`result.output_dir` (`trials.jsonl`, `scores.jsonl`, `summary.json`). The trial evidence (the final\nworkspace and the ATIF trace) is what the metrics above opened to score each run."
+   "source": "To see **what the agent actually did** — its output, the files it changed, its step-by-step trajectory\n— open the HTML dashboard at `location.dashboard_path`, or read the persisted bundle under\n`location.output_dir` (`trials.jsonl`, `scores.jsonl`, `summary.json`). The trial evidence (the final\nworkspace and the ATIF trace) is what the metrics above opened to score each run."
   },
   {
    "cell_type": "markdown",

packages/nemo_evaluator_sdk/examples/codex_docker/example.py

@@ -32,7 +32,7 @@
 
 from nemo_evaluator_sdk import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
 from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
-from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult
+from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult, BundleLocation
 from nemo_evaluator_sdk.agent_eval.runtimes.codex.runtime import CodexDockerCliAgentRuntime
 from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
 from nemo_evaluator_sdk.agent_eval.trials import AgentTaskRunner
@@ -108,8 +108,12 @@ async def evaluate(
     output_dir: str | Path | None = None,
     runtime: AgentTaskRunner | None = None,
     write_dashboard: bool = True,
-) -> AgentEvalResult:
-    """Run one Docker Codex task and score its host-readable workspace evidence."""
+) -> tuple[AgentEvalResult, BundleLocation]:
+    """Run one Docker Codex task, score its workspace evidence, and store the run.
+
+    Returns the result and where it was written: ``run`` itself no longer persists, so storing is an
+    explicit step here.
+    """
     resolved_output_dir = Path(output_dir).expanduser() if output_dir is not None else _new_output_dir()
     target = runtime or _docker_runtime(resolved_output_dir)
 
@@ -127,21 +131,21 @@ async def evaluate(
         metrics=[WorkspaceArtifactMetric()],
     )
 
-    return await AgentEvaluator().run(
+    result = await AgentEvaluator().run(
         tasks=[task],
         target=target,
         config=AgentEvalRunConfig(
-            output_dir=resolved_output_dir,
+            work_dir=resolved_output_dir,
             parallelism=1,
-            write_dashboard=write_dashboard,
             labels={"scenario": "codex-docker-evidence-sanity"},
         ),
     )
+    return result, result.persist(write_dashboard=write_dashboard)
 
 
 async def main() -> None:
     logging.basicConfig(level=logging.INFO, format="%(levelname)s:%(name)s:%(message)s")
-    result = await evaluate()
+    result, location = await evaluate()
 
     trial = result.trials[0]
     if trial.output is None or trial.evidence is None:
@@ -156,7 +160,7 @@ async def main() -> None:
     print(f"artifact contents: {artifact.read_text(encoding='utf-8').strip()}")
     print(f"workspace_artifact.output_matches: {scores['workspace_artifact.output_matches']}")
     print(f"workspace_artifact.artifact_matches: {scores['workspace_artifact.artifact_matches']}")
-    print(f"run bundle: {result.output_dir}")
+    print(f"run bundle: {location.output_dir}")
 
 
 if __name__ == "__main__":

packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py

@@ -63,7 +63,7 @@ async def main() -> int:
     )
 
     output_dir = Path(os.environ.get("FABRIC_OUTPUT_DIR", "/tmp/fabric-container-e2e"))
-    (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(output_dir=output_dir))
+    (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(work_dir=output_dir))
 
     print("=== TRIAL ===")
     print("status:", trial.status)

packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py

@@ -99,15 +99,18 @@ async def _main(args: argparse.Namespace) -> int:
     result = await AgentEvaluator().run(
         tasks=tasks,
         target=runner,
-        config=AgentEvalRunConfig(output_dir=output_dir, parallelism=1),
+        config=AgentEvalRunConfig(work_dir=output_dir, parallelism=1),
     )
+    # Storing the run is its own step. Defaults to the run's work_dir, so the bundle contains the
+    # evidence the trials point at.
+    location = result.persist()
 
     print("=== RESULT ===")
     print(f"tasks: {result.summary.task_count}  trials: {result.summary.trial_count}")
     print("aggregate scores:")
     for aggregate in result.summary.scores.scores:
         print(f"  {aggregate.name}: mean={aggregate.mean}")
-    print(f"\nRun bundle (run.json, trials.jsonl, scores.jsonl, report.html): {output_dir}")
+    print(f"\nRun bundle (run.json, trials.jsonl, scores.jsonl, report.html): {location.output_dir}")
     return 0
 
 

packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.py

@@ -93,13 +93,15 @@ async def _main(args: argparse.Namespace) -> None:
         flush=True,
     )
 
-    # The SDK's imported-trials path: no agent runs — it just scores the stored trials with our metric and
-    # writes a full bundle (run.json / trials.jsonl / scores.jsonl / summary.json / report.html).
+    # The SDK's imported-trials path: no agent runs — it just scores the stored trials with our metric.
     result = await AgentEvaluator().run(
         tasks=tasks,
         trials=trials,
-        config=AgentEvalRunConfig(output_dir=output_dir, parallelism=args.parallelism),
+        config=AgentEvalRunConfig(work_dir=output_dir, parallelism=args.parallelism),
     )
+    # Storing is explicit: writes the full bundle (run.json / trials.jsonl / scores.jsonl /
+    # summary.json / report.html).
+    result.persist()
 
     print(f"{'task (area)':30s} {'original':>10s} {'rescored':>10s}")
     print("-" * 54)

packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.py

@@ -142,8 +142,9 @@ async def _main(args: argparse.Namespace) -> None:
     result = await AgentEvaluator().run(
         tasks=tasks,
         target=runtime,
-        config=AgentEvalRunConfig(output_dir=Path(args.output_dir), parallelism=args.parallelism),
+        config=AgentEvalRunConfig(work_dir=Path(args.output_dir), parallelism=args.parallelism),
     )
+    result.persist()
 
     print(f"run_id: {result.run_id}  tasks: {result.summary.task_count}  trials: {result.summary.trial_count}")
     print("Aggregate scores:")

packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py

@@ -114,7 +114,7 @@ def _selected_task_names(args: argparse.Namespace) -> list[str] | None:
 
 async def _run_reward_only(args: argparse.Namespace) -> AgentEvalResult:
     """Minimal plumbing: discover + run + score LAB's official reward in one call."""
-    run_config = AgentEvalRunConfig(output_dir=Path(args.output_dir), parallelism=args.parallelism)
+    run_config = AgentEvalRunConfig(work_dir=Path(args.output_dir), parallelism=args.parallelism)
     return await run_harbor_eval(
         _build_config(args),
         args.dataset_path,
@@ -135,7 +135,7 @@ async def _run_with_components(args: argparse.Namespace) -> AgentEvalResult:
 
     # Restrict the Harbor run itself to the selected tasks (not just the scoring).
     runner = HarborAgentTaskRunner(config=_build_config(args), task_names=[task.id for task in tasks])
-    run_config = AgentEvalRunConfig(output_dir=Path(args.output_dir), parallelism=args.parallelism)
+    run_config = AgentEvalRunConfig(work_dir=Path(args.output_dir), parallelism=args.parallelism)
     return await AgentEvaluator().run(tasks=tasks, target=runner, config=run_config)
 
 
@@ -145,6 +145,9 @@ async def _main(args: argparse.Namespace) -> None:
     else:
         result = await _run_reward_only(args)
 
+    # Storing is explicit; both modes persist into the run's work_dir.
+    result.persist()
+
     print(f"run_id: {result.run_id}  tasks: {result.summary.task_count}  trials: {result.summary.trial_count}")
     print("Aggregate scores:")
     for aggregate in result.summary.scores.scores:

packages/nemo_evaluator_sdk/examples/profbench/README.md

@@ -131,13 +131,14 @@ async def main() -> None:
         tasks=benchmark.tasks,
         target=DockerSandboxAgentRuntime(model="gpt-4.1-mini", timeout_s=180),
         config=AgentEvalRunConfig(
-            output_dir=output_dir,
+            work_dir=output_dir,
             run_id="profbench-code-sandbox-smoke",
             parallelism=1,
             labels={**{k: str(v) for k, v in benchmark.metadata.items()}, "score_source": "docker_sandbox_and_live_judge"},
-            write_dashboard=False,
         ),
     )
+    # This example renders its own dashboards below, so persistence skips the built-in one.
+    result.persist(write_dashboard=False)
     sdk_dashboard_path, dashboard_path = write_example_dashboards(result, output_dir)
     print(f"SDK dashboard: {sdk_dashboard_path}")
     print(f"Dashboard: {dashboard_path}")
@@ -258,7 +259,7 @@ result = await AgentEvaluator().run(
     tasks=benchmark.tasks,
     target=evaluated_model,
     config=AgentEvalRunConfig(
-        output_dir=output_dir,
+        work_dir=output_dir,
         params=params,
         labels={**{k: str(v) for k, v in benchmark.metadata.items()}, "score_source": "fresh_candidate_and_live_judge"},
         ...

packages/nemo_evaluator_sdk/examples/profbench/runner.py

@@ -121,13 +121,14 @@ async def run_profbench_mode(
         trials=trials,
         target=target,
         config=AgentEvalRunConfig(
-            output_dir=output_dir,
+            work_dir=output_dir,
             run_id=f"{run_instance_id}-{mode.value}",
             params=params,
             labels=benchmark_labels,
-            write_dashboard=False,
         ),
     )
+    # This example renders its own dashboards below, so persistence skips the built-in one.
+    result.persist(write_dashboard=False)
     sdk_dashboard_path, dashboard_path = write_example_dashboards(result, output_dir)
 
     overall = _profbench_overall(result)