diff --git a/docs/evaluator/agent-eval/harbor-runner.mdx b/docs/evaluator/agent-eval/harbor-runner.mdx index 1e11aec2e7..a432e357e7 100644 --- a/docs/evaluator/agent-eval/harbor-runner.mdx +++ b/docs/evaluator/agent-eval/harbor-runner.mdx @@ -135,8 +135,8 @@ Swap `agent_name` (or `agent_import_path`) for your own agent to get a real scor 2. The runner reads that reward onto each trial's metadata, and `HarborRewardMetric` scores it — the `reward` value, or `0.0` for a trial whose verifier emitted none. 3. `result.summary` aggregates the reward across tasks (`harbor_reward.reward`), `result.trials` holds - each trial's status and evidence, and — if you pass a `config` with an `output_dir` to `run()` — the - run **bundle** (including `report.html`) is written like any other agent-eval run. + each trial's status and evidence, and — if you call `result.persist()` — the run **bundle** + (including `report.html`) is written like any other agent-eval run. ## Attempts, concurrency, and caching diff --git a/docs/evaluator/agent-eval/quickstart.mdx b/docs/evaluator/agent-eval/quickstart.mdx index 1e95bcae9d..b6baaecf0d 100644 --- a/docs/evaluator/agent-eval/quickstart.mdx +++ b/docs/evaluator/agent-eval/quickstart.mdx @@ -106,8 +106,8 @@ async def my_agent(task: AgentEvalTask) -> str: ## 4. Run the evaluation `AgentEvaluator.run()` sends the tasks to the runner, collects the trials, scores them, and returns -an `AgentEvalResult`. Setting `output_dir` also writes a run bundle to disk. (`run()` is async, so it -lives inside an `async` function — see the full script below.) +an `AgentEvalResult`. It writes nothing on its own — call `result.persist()` to store a run bundle to +disk. (`run()` is async, so it lives inside an `async` function — see the full script below.) ```python from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator @@ -117,8 +117,9 @@ from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig result = await AgentEvaluator().run( tasks=make_tasks(), target=CallableAgentTaskRunner(my_agent), - config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2), + config=AgentEvalRunConfig(work_dir="./agent-eval-run", parallelism=2), ) +result.persist() for aggregate in result.summary.scores.scores: print(f"{aggregate.name}: {aggregate.mean}") @@ -135,7 +136,8 @@ for aggregate in result.summary.scores.scores: state, logs), and its status (`completed` / `partial` / `failed`). - `result.run_id` — a stable identifier for this run. -Because you set `output_dir`, the same data was also written to `./agent-eval-run/` as a run bundle: +Because you called `persist()`, the same data was also written to `./agent-eval-run/` as a run bundle +(it defaults to `work_dir`): | File | Contents | |---|---| @@ -144,7 +146,7 @@ Because you set `output_dir`, the same data was also written to `./agent-eval-ru | `trials.jsonl` | one row per trial — the agent's output, its evidence, and status | | `tasks.jsonl` | the tasks that were evaluated | | `run.json` | the run manifest — the run id and a map of the artifact files | -| `benchmark.json` | benchmark-grouping metadata for the run | +| `metadata.json` | run provenance — labels, target identity, timings, SDK version | | `report.html` | a browsable dashboard of the run — open it in a browser | The in-memory result and the on-disk bundle hold the same information: use the result object for @@ -209,8 +211,9 @@ async def main() -> None: result = await AgentEvaluator().run( tasks=make_tasks(), target=CallableAgentTaskRunner(my_agent), - config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2), + config=AgentEvalRunConfig(work_dir="./agent-eval-run", parallelism=2), ) + result.persist() for aggregate in result.summary.scores.scores: print(f"{aggregate.name}: {aggregate.mean}") diff --git a/docs/evaluator/agent-eval/reading-results.mdx b/docs/evaluator/agent-eval/reading-results.mdx index 1df54b1fdf..108fe55b9b 100644 --- a/docs/evaluator/agent-eval/reading-results.mdx +++ b/docs/evaluator/agent-eval/reading-results.mdx @@ -1,11 +1,11 @@ --- title: "Reading Results" -description: "Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle it writes when you set output_dir, including the browsable HTML report." +description: "Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle you get by calling persist(), including the browsable HTML report." --- -`AgentEvaluator().run(...)` returns an `AgentEvalResult`. Set an `output_dir` and it *also* writes a -**run bundle** to disk. The object and the bundle hold the same data — use the object for programmatic -follow-up, and the bundle (especially `report.html`) to inspect or share a run. +`AgentEvaluator().run(...)` returns an `AgentEvalResult` and writes nothing. Call `result.persist()` to +store it as a **run bundle** on disk. The object and the bundle hold the same data — use the object for +programmatic follow-up, and the bundle (especially `report.html`) to inspect or share a run. ## The result object @@ -20,7 +20,7 @@ result = await AgentEvaluator().run(tasks=..., target=...) | `result.scores` | one entry per **(task, trial, metric)** | | `result.trials` | one entry per **trial** | | `result.tasks` | the tasks that were evaluated | -| `result.output_dir` / `result.dashboard_path` | where the bundle and `report.html` were written (when `output_dir` was set) | +| `result.work_dir` | the directory the run worked in, where runtimes wrote trial evidence (`None` for an in-memory run) | ### The summary @@ -53,7 +53,7 @@ are the durable, scorer-agnostic record — they can be re-scored offline later. ## The run bundle -Set `output_dir` and `run()` writes these files (the same data, on disk): +Call `result.persist()` and it writes these files (the same data, on disk): | File | Contents | |---|---| @@ -62,16 +62,20 @@ Set `output_dir` and `run()` writes these files (the same data, on disk): | `scores.jsonl` | one row per (task, trial, metric) | | `trials.jsonl` | one row per trial — output, evidence, status | | `tasks.jsonl` | the tasks that were evaluated | -| `benchmark.json` | benchmark-grouping metadata for the run | +| `metadata.json` | run provenance — labels, target identity, timings, SDK version | | `report.html` | a browsable dashboard — open it in a browser | +`persist()` returns a `BundleLocation` telling you where the bundle landed: + ```python from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig result = await AgentEvaluator().run( - tasks=..., target=..., config=AgentEvalRunConfig(output_dir="./agent-eval-run"), + tasks=..., target=..., config=AgentEvalRunConfig(work_dir="./agent-eval-run"), ) +location = result.persist() # -> ./agent-eval-run/report.html, summary.json, scores.jsonl, trials.jsonl, ... +print(location.output_dir, location.dashboard_path) ``` `report.html` is the fastest way to eyeball a run or hand it to someone else; the `.jsonl` files are @@ -79,9 +83,19 @@ convenient for loading scores and trials into your own tooling. -`report.html` is written only when `AgentEvalRunConfig.write_dashboard` is `True` (the default). Set -`write_dashboard=False` to emit just the JSON/JSONL artifacts and skip the HTML; the `.json` and -`.jsonl` files are always written whenever `output_dir` is set. +`persist()` defaults to `work_dir`, which is where the run's trial evidence already lives — that keeps +the bundle self-contained, so it survives being moved or copied. Passing an explicit +`persist("./elsewhere")` is supported, but the bundle's evidence references still point back at the +original directory and only resolve while it exists. + +A run with no `work_dir` and no explicit target raises rather than inventing a directory. + + + + + +`report.html` is written unless you pass `persist(write_dashboard=False)`, which emits just the +JSON/JSONL artifacts and skips the HTML. The `.json` and `.jsonl` files are always written. diff --git a/packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb b/packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb index 7414ba387e..ae75c40f85 100644 --- a/packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb +++ b/packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb @@ -599,13 +599,16 @@ "result = evaluator.run_sync(\n", " tasks=suite.tasks,\n", " target=target,\n", - " config=AgentEvalRunConfig(output_dir=OUTPUT_DIR, write_dashboard=True, parallelism=3),\n", + " config=AgentEvalRunConfig(work_dir=OUTPUT_DIR, parallelism=3),\n", ")\n", "\n", + "# Storing the run is its own step; it defaults to the work_dir the config named.\n", + "location = result.persist()\n", + "\n", "print(\"run_id :\", result.run_id)\n", "print(\"tasks :\", result.summary.task_count)\n", "print(\"trials :\", result.summary.trial_count)\n", - "print(\"dashboard :\", result.dashboard_path)" + "print(\"dashboard :\", location.dashboard_path)" ] }, { @@ -685,7 +688,7 @@ "cell_type": "markdown", "id": "cell-34", "metadata": {}, - "source": "To see **what the agent actually did** — its output, the files it changed, its step-by-step trajectory\n— open the HTML dashboard at `result.dashboard_path`, or read the persisted bundle under\n`result.output_dir` (`trials.jsonl`, `scores.jsonl`, `summary.json`). The trial evidence (the final\nworkspace and the ATIF trace) is what the metrics above opened to score each run." + "source": "To see **what the agent actually did** — its output, the files it changed, its step-by-step trajectory\n— open the HTML dashboard at `location.dashboard_path`, or read the persisted bundle under\n`location.output_dir` (`trials.jsonl`, `scores.jsonl`, `summary.json`). The trial evidence (the final\nworkspace and the ATIF trace) is what the metrics above opened to score each run." }, { "cell_type": "markdown", diff --git a/packages/nemo_evaluator_sdk/examples/codex_docker/example.py b/packages/nemo_evaluator_sdk/examples/codex_docker/example.py index 49c46c9d68..e8640adf00 100644 --- a/packages/nemo_evaluator_sdk/examples/codex_docker/example.py +++ b/packages/nemo_evaluator_sdk/examples/codex_docker/example.py @@ -32,7 +32,7 @@ from nemo_evaluator_sdk import MetricInput, MetricOutput, MetricOutputSpec, MetricResult from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator -from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult +from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult, BundleLocation from nemo_evaluator_sdk.agent_eval.runtimes.codex.runtime import CodexDockerCliAgentRuntime from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask from nemo_evaluator_sdk.agent_eval.trials import AgentTaskRunner @@ -108,8 +108,12 @@ async def evaluate( output_dir: str | Path | None = None, runtime: AgentTaskRunner | None = None, write_dashboard: bool = True, -) -> AgentEvalResult: - """Run one Docker Codex task and score its host-readable workspace evidence.""" +) -> tuple[AgentEvalResult, BundleLocation]: + """Run one Docker Codex task, score its workspace evidence, and store the run. + + Returns the result and where it was written: ``run`` itself no longer persists, so storing is an + explicit step here. + """ resolved_output_dir = Path(output_dir).expanduser() if output_dir is not None else _new_output_dir() target = runtime or _docker_runtime(resolved_output_dir) @@ -127,21 +131,21 @@ async def evaluate( metrics=[WorkspaceArtifactMetric()], ) - return await AgentEvaluator().run( + result = await AgentEvaluator().run( tasks=[task], target=target, config=AgentEvalRunConfig( - output_dir=resolved_output_dir, + work_dir=resolved_output_dir, parallelism=1, - write_dashboard=write_dashboard, labels={"scenario": "codex-docker-evidence-sanity"}, ), ) + return result, result.persist(write_dashboard=write_dashboard) async def main() -> None: logging.basicConfig(level=logging.INFO, format="%(levelname)s:%(name)s:%(message)s") - result = await evaluate() + result, location = await evaluate() trial = result.trials[0] if trial.output is None or trial.evidence is None: @@ -156,7 +160,7 @@ async def main() -> None: print(f"artifact contents: {artifact.read_text(encoding='utf-8').strip()}") print(f"workspace_artifact.output_matches: {scores['workspace_artifact.output_matches']}") print(f"workspace_artifact.artifact_matches: {scores['workspace_artifact.artifact_matches']}") - print(f"run bundle: {result.output_dir}") + print(f"run bundle: {location.output_dir}") if __name__ == "__main__": diff --git a/packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py b/packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py index 6005d244c2..5ef3d0980f 100644 --- a/packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py +++ b/packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py @@ -63,7 +63,7 @@ async def main() -> int: ) output_dir = Path(os.environ.get("FABRIC_OUTPUT_DIR", "/tmp/fabric-container-e2e")) - (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(output_dir=output_dir)) + (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(work_dir=output_dir)) print("=== TRIAL ===") print("status:", trial.status) diff --git a/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py b/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py index 53cebfea44..72a856399e 100644 --- a/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py +++ b/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py @@ -99,15 +99,18 @@ async def _main(args: argparse.Namespace) -> int: result = await AgentEvaluator().run( tasks=tasks, target=runner, - config=AgentEvalRunConfig(output_dir=output_dir, parallelism=1), + config=AgentEvalRunConfig(work_dir=output_dir, parallelism=1), ) + # Storing the run is its own step. Defaults to the run's work_dir, so the bundle contains the + # evidence the trials point at. + location = result.persist() print("=== RESULT ===") print(f"tasks: {result.summary.task_count} trials: {result.summary.trial_count}") print("aggregate scores:") for aggregate in result.summary.scores.scores: print(f" {aggregate.name}: mean={aggregate.mean}") - print(f"\nRun bundle (run.json, trials.jsonl, scores.jsonl, report.html): {output_dir}") + print(f"\nRun bundle (run.json, trials.jsonl, scores.jsonl, report.html): {location.output_dir}") return 0 diff --git a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.py b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.py index e70b7744d7..7b9239bbc4 100644 --- a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.py +++ b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.py @@ -93,13 +93,15 @@ async def _main(args: argparse.Namespace) -> None: flush=True, ) - # The SDK's imported-trials path: no agent runs — it just scores the stored trials with our metric and - # writes a full bundle (run.json / trials.jsonl / scores.jsonl / summary.json / report.html). + # The SDK's imported-trials path: no agent runs — it just scores the stored trials with our metric. result = await AgentEvaluator().run( tasks=tasks, trials=trials, - config=AgentEvalRunConfig(output_dir=output_dir, parallelism=args.parallelism), + config=AgentEvalRunConfig(work_dir=output_dir, parallelism=args.parallelism), ) + # Storing is explicit: writes the full bundle (run.json / trials.jsonl / scores.jsonl / + # summary.json / report.html). + result.persist() print(f"{'task (area)':30s} {'original':>10s} {'rescored':>10s}") print("-" * 54) diff --git a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.py b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.py index 6fd08f6f8e..338839079d 100644 --- a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.py +++ b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.py @@ -142,8 +142,9 @@ async def _main(args: argparse.Namespace) -> None: result = await AgentEvaluator().run( tasks=tasks, target=runtime, - config=AgentEvalRunConfig(output_dir=Path(args.output_dir), parallelism=args.parallelism), + config=AgentEvalRunConfig(work_dir=Path(args.output_dir), parallelism=args.parallelism), ) + result.persist() print(f"run_id: {result.run_id} tasks: {result.summary.task_count} trials: {result.summary.trial_count}") print("Aggregate scores:") diff --git a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py index 9c420d0021..616fc4c1e8 100644 --- a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py +++ b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py @@ -114,7 +114,7 @@ def _selected_task_names(args: argparse.Namespace) -> list[str] | None: async def _run_reward_only(args: argparse.Namespace) -> AgentEvalResult: """Minimal plumbing: discover + run + score LAB's official reward in one call.""" - run_config = AgentEvalRunConfig(output_dir=Path(args.output_dir), parallelism=args.parallelism) + run_config = AgentEvalRunConfig(work_dir=Path(args.output_dir), parallelism=args.parallelism) return await run_harbor_eval( _build_config(args), args.dataset_path, @@ -135,7 +135,7 @@ async def _run_with_components(args: argparse.Namespace) -> AgentEvalResult: # Restrict the Harbor run itself to the selected tasks (not just the scoring). runner = HarborAgentTaskRunner(config=_build_config(args), task_names=[task.id for task in tasks]) - run_config = AgentEvalRunConfig(output_dir=Path(args.output_dir), parallelism=args.parallelism) + run_config = AgentEvalRunConfig(work_dir=Path(args.output_dir), parallelism=args.parallelism) return await AgentEvaluator().run(tasks=tasks, target=runner, config=run_config) @@ -145,6 +145,9 @@ async def _main(args: argparse.Namespace) -> None: else: result = await _run_reward_only(args) + # Storing is explicit; both modes persist into the run's work_dir. + result.persist() + print(f"run_id: {result.run_id} tasks: {result.summary.task_count} trials: {result.summary.trial_count}") print("Aggregate scores:") for aggregate in result.summary.scores.scores: diff --git a/packages/nemo_evaluator_sdk/examples/profbench/README.md b/packages/nemo_evaluator_sdk/examples/profbench/README.md index cc622daae2..50a7b34618 100644 --- a/packages/nemo_evaluator_sdk/examples/profbench/README.md +++ b/packages/nemo_evaluator_sdk/examples/profbench/README.md @@ -131,13 +131,14 @@ async def main() -> None: tasks=benchmark.tasks, target=DockerSandboxAgentRuntime(model="gpt-4.1-mini", timeout_s=180), config=AgentEvalRunConfig( - output_dir=output_dir, + work_dir=output_dir, run_id="profbench-code-sandbox-smoke", parallelism=1, labels={**{k: str(v) for k, v in benchmark.metadata.items()}, "score_source": "docker_sandbox_and_live_judge"}, - write_dashboard=False, ), ) + # This example renders its own dashboards below, so persistence skips the built-in one. + result.persist(write_dashboard=False) sdk_dashboard_path, dashboard_path = write_example_dashboards(result, output_dir) print(f"SDK dashboard: {sdk_dashboard_path}") print(f"Dashboard: {dashboard_path}") @@ -258,7 +259,7 @@ result = await AgentEvaluator().run( tasks=benchmark.tasks, target=evaluated_model, config=AgentEvalRunConfig( - output_dir=output_dir, + work_dir=output_dir, params=params, labels={**{k: str(v) for k, v in benchmark.metadata.items()}, "score_source": "fresh_candidate_and_live_judge"}, ... diff --git a/packages/nemo_evaluator_sdk/examples/profbench/runner.py b/packages/nemo_evaluator_sdk/examples/profbench/runner.py index 7c3146d308..f66b904eb6 100644 --- a/packages/nemo_evaluator_sdk/examples/profbench/runner.py +++ b/packages/nemo_evaluator_sdk/examples/profbench/runner.py @@ -121,13 +121,14 @@ async def run_profbench_mode( trials=trials, target=target, config=AgentEvalRunConfig( - output_dir=output_dir, + work_dir=output_dir, run_id=f"{run_instance_id}-{mode.value}", params=params, labels=benchmark_labels, - write_dashboard=False, ), ) + # This example renders its own dashboards below, so persistence skips the built-in one. + result.persist(write_dashboard=False) sdk_dashboard_path, dashboard_path = write_example_dashboards(result, output_dir) overall = _profbench_overall(result) diff --git a/packages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.py b/packages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.py index d45e3f78c7..851159303d 100644 --- a/packages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.py +++ b/packages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.py @@ -74,7 +74,7 @@ async def run_tasks( target=target, config=self._run_config(output_dir=output_dir, run_id=run_id, labels=labels), ) - self._maybe_write_gate(result) + self._persist_and_gate(result, output_dir) return result async def score_trials( @@ -93,7 +93,7 @@ async def score_trials( trials=list(trials), config=self._run_config(output_dir=output_dir, run_id=run_id, labels=labels), ) - self._maybe_write_gate(result) + self._persist_and_gate(result, output_dir) return result def _run_config( @@ -104,10 +104,9 @@ def _run_config( labels: dict[str, str] | None, ) -> AgentEvalRunConfig: return AgentEvalRunConfig( - output_dir=output_dir, + work_dir=output_dir, run_id=run_id, parallelism=self.config.parallelism, - write_dashboard=self.config.write_dashboard, labels=dict(labels or {}), ) @@ -122,8 +121,12 @@ def _with_extra_metrics(self, task: AgentEvalTask) -> AgentEvalTask: return task return task.model_copy(update={"metrics": metrics + appended}) - def _maybe_write_gate(self, result: AgentEvalResult) -> None: - if not (self.config.write_gate and result.output_dir is not None): + def _persist_and_gate(self, result: AgentEvalResult, output_dir: Path | None) -> None: + """Store the run (when a directory was given) and write the gate report beside it.""" + if output_dir is None: + return + location = result.persist(write_dashboard=self.config.write_dashboard) + if not self.config.write_gate: return baseline = ( load_baseline_summary(self.config.baseline_summary_path) @@ -131,7 +134,7 @@ def _maybe_write_gate(self, result: AgentEvalResult) -> None: else None ) report = evaluate_gate(result, thresholds=self.config.gate_thresholds, baseline_summary=baseline) - write_gate_report(report, result.output_dir) + write_gate_report(report, location.output_dir) __all__ = [ diff --git a/packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py b/packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py index f8dc743ca8..a067331f88 100644 --- a/packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py +++ b/packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py @@ -106,7 +106,7 @@ def task_image_tag(task_id: str) -> str: def resolve_run_layout(task: AgentEvalTask, config: AgentEvalRunConfig | None) -> AgenticRunLayout: """Resolve/create the on-disk layout for one task run.""" - output_dir = config.output_dir if config is not None else None + output_dir = config.work_dir if config is not None else None run_dir = resolve_run_dir(output_dir, lambda: Path.cwd() / "nat-jobs" / task.id) / task.id base = prepare_run_layout(run_dir, str(task.inputs.get("instruction") or task.intent)) state_dir = base.run_dir / "state" diff --git a/packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py b/packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py index 60239e9cf1..ebe70b9be0 100644 --- a/packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py +++ b/packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py @@ -159,9 +159,9 @@ def _print_result(result: AgentEvalResult) -> None: if score.mean is not None: print(f" {score.name}: mean={score.mean:.3f}") _print_measurements(result) - if result.output_dir is not None: - print(f"output_dir: {result.output_dir}") - print(f"gate: {result.output_dir / 'gate.json'}") + if result.work_dir is not None: + print(f"work_dir: {result.work_dir}") + print(f"gate: {result.work_dir / 'gate.json'}") def _print_measurements(result: AgentEvalResult) -> None: diff --git a/packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py b/packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py index d9284ff47c..5b1daf4887 100644 --- a/packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py +++ b/packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py @@ -192,7 +192,7 @@ def _format_command(self, instruction_path: Path, workspace_dir: Path, input_jso return [substitutions.get(token, token) for token in self.config.command] def _run_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: - root = (config.output_dir or Path.cwd()) / "evidence" / RUNTIME_NAME + root = (config.work_dir or Path.cwd()) / "evidence" / RUNTIME_NAME return root / (_safe_name(task.id) or f"task-{index}") diff --git a/packages/nemo_evaluator_sdk/examples/skill_eval/run_skill_eval.py b/packages/nemo_evaluator_sdk/examples/skill_eval/run_skill_eval.py index 0693efe21e..5927eada8c 100644 --- a/packages/nemo_evaluator_sdk/examples/skill_eval/run_skill_eval.py +++ b/packages/nemo_evaluator_sdk/examples/skill_eval/run_skill_eval.py @@ -258,15 +258,17 @@ async def _main() -> int: baseline = await AgentEvaluator().run( tasks=tasks, target=baseline_runtime, - config=AgentEvalRunConfig(run_id="baseline", output_dir=output_dir / "baseline", write_dashboard=False), + config=AgentEvalRunConfig(run_id="baseline", work_dir=output_dir / "baseline"), ) + baseline.persist(write_dashboard=False) treated = await AgentEvaluator().run( tasks=tasks, target=baseline_runtime.with_skill( skill ), # We include the skill in the treated arm, so the two runs differ in *exactly* the skill. - config=AgentEvalRunConfig(run_id="treated", output_dir=output_dir / "treated", write_dashboard=False), + config=AgentEvalRunConfig(run_id="treated", work_dir=output_dir / "treated"), ) + treated.persist(write_dashboard=False) except SkillInjectionError as exc: print(f"skill eval failed to load the bundled skill: {exc}") return 1 diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py index 6e0cb46370..720c94888b 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py @@ -21,8 +21,6 @@ import httpx import nemo_evaluator_sdk.inference as inference -from nemo_evaluator_sdk.agent_eval.dashboard import write_dashboard -from nemo_evaluator_sdk.agent_eval.persistence import persist_run from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult, AgentEvalSummary, RunMetadata from nemo_evaluator_sdk.agent_eval.scores import ( AgentEvalDiagnostic, @@ -196,10 +194,9 @@ async def run( scores=scores, summary=AgentEvalSummary.from_scores(scores, tasks=task_list, extra_scores=runner_scores), metadata=metadata, + work_dir=runtime_config.work_dir, ) - if runtime_config.output_dir is not None: - result = _persist_with_optional_dashboard(result, runtime_config.output_dir, runtime_config.write_dashboard) return result def run_sync( @@ -342,8 +339,8 @@ async def generate_one(index: int, task: AgentEvalTask) -> AgentEvalTrial: "invocation_id": f"{config.run_id}:{task.id}:{target.name}", } evidence_dir = ( - _task_evidence_dir(Path(config.output_dir), index=index, task_id=task.id) - if config.output_dir is not None and isinstance(target, AgentBase) + _task_evidence_dir(Path(config.work_dir), index=index, task_id=task.id) + if config.work_dir is not None and isinstance(target, AgentBase) else None ) resolved_inference_fn = self.inference_fn @@ -770,18 +767,6 @@ def _collect_runner_aggregate_scores(target: object) -> list[AggregateScore]: return collected -def _persist_with_optional_dashboard( - result: AgentEvalResult, - output_dir: Path, - write_html: bool, -) -> AgentEvalResult: - path = Path(output_dir) - dashboard_path = None - if write_html: - dashboard_path = write_dashboard(result.model_copy(update={"output_dir": path}), path / "report.html") - return persist_run(result.model_copy(update={"output_dir": path, "dashboard_path": dashboard_path}), path) - - def _new_run_id() -> str: timestamp = datetime.now(UTC).strftime("%Y%m%d%H%M%S") return f"agent-eval-{timestamp}-{uuid.uuid4().hex[:8]}" diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py index baa184fc5f..ec7210f46c 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py @@ -10,40 +10,59 @@ from pathlib import Path from typing import Any -from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult +from nemo_evaluator_sdk.agent_eval.dashboard import write_dashboard +from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult, BundleLocation from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial from pydantic import BaseModel +#: Filename of the rendered HTML dashboard inside a bundle. +DASHBOARD_FILENAME = "report.html" -def persist_run(result: AgentEvalResult, output_dir: str | Path) -> AgentEvalResult: - """Persist a completed run bundle to ``output_dir``.""" + +def persist_run( + result: AgentEvalResult, + output_dir: str | Path, + *, + write_html_dashboard: bool = True, +) -> BundleLocation: + """Write a completed run to a bundle at ``output_dir`` and report where it landed. + + Explicit rather than a side effect of :meth:`AgentEvaluator.run`: computing an evaluation and + storing one are different decisions, and folding them together is what forced the result object to + carry paths it could not know at construction time. (Same reasoning as ``publish_to_intake``.) + + Set ``write_html_dashboard=False`` to skip rendering ``report.html`` — the dashboard is written + here so the manifest can record it in a single pass. + """ path = Path(output_dir) path.mkdir(parents=True, exist_ok=True) + # Render first so the manifest below can name it; the dashboard reads only the run's own contents. + dashboard_path = write_dashboard(result, path / DASHBOARD_FILENAME) if write_html_dashboard else None + _write_json(path / "metadata.json", result.metadata) _write_jsonl(path / "tasks.jsonl", result.tasks) _write_trials(path / "trials.jsonl", result.trials, base=path) _write_jsonl(path / "scores.jsonl", result.scores) _write_json(path / "summary.json", result.summary) - updated = result.model_copy(update={"output_dir": path}) - _write_json(path / "run.json", _run_manifest(updated)) - return updated + location = BundleLocation(output_dir=path, dashboard_path=dashboard_path) + _write_json(path / "run.json", _run_manifest(result, location)) + return location -def _run_manifest(result: AgentEvalResult) -> dict[str, Any]: - artifacts = { - "metadata": "metadata.json", - "tasks": "tasks.jsonl", - "trials": "trials.jsonl", - "scores": "scores.jsonl", - "summary": "summary.json", - } +def _run_manifest(result: AgentEvalResult, location: BundleLocation) -> dict[str, Any]: return { "run_id": result.run_id, - "output_dir": str(result.output_dir) if result.output_dir is not None else None, - "dashboard_path": str(result.dashboard_path) if result.dashboard_path is not None else None, - "artifacts": artifacts, + "output_dir": str(location.output_dir), + "dashboard_path": str(location.dashboard_path) if location.dashboard_path is not None else None, + "artifacts": { + "metadata": "metadata.json", + "tasks": "tasks.jsonl", + "trials": "trials.jsonl", + "scores": "scores.jsonl", + "summary": "summary.json", + }, } diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py index 065cc09b03..b31d1b5078 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py @@ -109,8 +109,30 @@ class RunMetadata(BaseModel): sdk_version: str | None = Field(default=None, description="nemo-evaluator-sdk version that produced the run.") +class BundleLocation(BaseModel): + """Where a run was written, returned by :meth:`AgentEvalResult.persist`. + + Kept off :class:`AgentEvalResult` because it is not a property of the evaluation — it is the + outcome of choosing to store it. Holding one means the bundle exists, so there is no optional to + re-check; a run that was never persisted simply has no ``BundleLocation``. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + output_dir: Path = Field(description="Directory the run bundle was written to.") + dashboard_path: Path | None = Field( + default=None, + description="Path to the rendered HTML dashboard, or None when dashboard writing was disabled.", + ) + + class AgentEvalResult(BaseModel): - """Root result for a completed agent evaluation: tasks, trials, scores, summary, and bundle metadata.""" + """Root result for a completed agent evaluation: tasks, trials, scores, and summary. + + Describes the evaluation and nothing else — storing it is a separate decision, made by calling + :meth:`persist`. Because the result carries no paths, it never holds a location that was unknown + when it was constructed, and nothing has to mutate it after the fact. + """ model_config = ConfigDict(extra="forbid") @@ -123,8 +145,39 @@ class AgentEvalResult(BaseModel): default_factory=RunMetadata, description="Run provenance: labels, target identity, timings, SDK version.", ) - output_dir: Path | None = Field(default=None, description="Directory the run bundle was written to, if any.") - dashboard_path: Path | None = Field(default=None, description="Path to the rendered dashboard, if written.") + work_dir: Path | None = Field( + default=None, + description="Directory the run worked in, where its runtimes wrote trial evidence. Known " + "before the run starts (it comes from the run config), so unlike a bundle location it is " + "never attached after the fact. None for a purely in-memory run.", + ) + + def persist(self, output_dir: str | Path | None = None, *, write_dashboard: bool = True) -> BundleLocation: + """Write this run to a bundle and return where it landed. + + Deliberately a call rather than something ``AgentEvaluator.run`` does for you: computing an + evaluation and storing one are separate decisions (the same reasoning as ``publish_to_intake``). + + Defaults to :attr:`work_dir`, which is the directory the trials' evidence already lives under — + so the bundle is self-contained and survives being moved. Passing a different ``output_dir`` + leaves those evidence references pointing back at the original directory. That is supported (a + re-scored run may reference an earlier run's deliverables) but the resulting bundle only + resolves while the original directory is still there. + + Set ``write_dashboard=False`` to skip rendering ``report.html``. + """ + # Imported here rather than at module scope: persistence imports this module for the types it + # writes, so a top-level import would be circular. + from nemo_evaluator_sdk.agent_eval.persistence import persist_run + + target = output_dir if output_dir is not None else self.work_dir + if target is None: + raise ValueError( + "this run has no work_dir to persist into (it ran in memory); pass an explicit " + "output_dir, or set work_dir on the AgentEvalRunConfig so evidence and bundle share " + "a directory" + ) + return persist_run(self, target, write_html_dashboard=write_dashboard) def _aggregate_scores( diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py index abef83d4a4..75b0af038b 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py @@ -288,7 +288,7 @@ def _validate_artifact_permissions(self, evidence_dir: Path) -> None: def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: root = self._work_root if root is None: - root = (config.output_dir or Path.cwd()) / "evidence" / "codex" + root = (config.work_dir or Path.cwd()) / "evidence" / "codex" safe_task_id = _safe_path_name(task.id) task_dir = f"{index:06d}-{safe_task_id}" if safe_task_id else f"task-{index:06d}" return Path(root) / task_dir diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/docker_sandbox.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/docker_sandbox.py index 7c8cdccb17..298ef1335f 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/docker_sandbox.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/docker_sandbox.py @@ -289,7 +289,7 @@ def _failed_trial(self, task: AgentEvalTask, exc: Exception, evidence_dir: Path) ) def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: - root = config.output_dir if config.output_dir is not None else self._work_root + root = config.work_dir if config.work_dir is not None else self._work_root if root is None: root = Path(tempfile.gettempdir()) / "nemo-evaluator-agent-runtime" run_id = config.run_id or _new_runtime_run_id() diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py index 0d19c9de41..2af24ecefb 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py @@ -485,7 +485,7 @@ def _failed_trial( def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: # Evidence lands under the run's output dir (like every other runtime); the container's own # working state lives at /out inside the sandbox and is downloaded here. - root = (config.output_dir or Path.cwd()) / "evidence" / "fabric_container" + root = (config.work_dir or Path.cwd()) / "evidence" / "fabric_container" return root / _common.task_subdir_name(index, task.id) diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py index 16395d80b2..22b650efb5 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py @@ -635,7 +635,7 @@ def _relay_config( def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: root = self._work_root if root is None: - root = (config.output_dir or Path.cwd()) / "evidence" / "fabric" + root = (config.work_dir or Path.cwd()) / "evidence" / "fabric" # The run id isolates this run's evidence from other runs sharing the same root (A/B baseline # vs. skilled); run_tasks always populates it, so the fallback only guards a direct call. run_id = config.run_id or _new_run_id() diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py index 73ae5efdb3..8a968dba8f 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py @@ -412,8 +412,8 @@ async def run_tasks( # phases — parallelism bounds concurrent scoring (SDK-side, cheap), while Gym's `--concurrency` # bounds concurrent rollouts against the model endpoint during collection (tuned to that endpoint's # limits via GymRuntimeConfig.concurrency). - if config is not None and config.output_dir is not None: - work_dir = Path(config.output_dir) / "gym_run" + if config is not None and config.work_dir is not None: + work_dir = Path(config.work_dir) / "gym_run" else: work_dir = Path(tempfile.mkdtemp(prefix="gym_run_")) work_dir.mkdir(parents=True, exist_ok=True) diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py index 72d94941ba..e003199bda 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py @@ -1338,7 +1338,7 @@ async def run_harbor_eval( return await AgentEvaluator().run( tasks=tasks, target=runner, - config=run_config or AgentEvalRunConfig(write_dashboard=False), + config=run_config or AgentEvalRunConfig(), ) diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py index 163a3b8547..b1b6a0f1c0 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py @@ -214,9 +214,11 @@ class AgentEvalRunConfig(BaseModel): model_config = ConfigDict(extra="forbid") - output_dir: Path | None = Field( + work_dir: Path | None = Field( default=None, - description="Directory where the run bundle is written; in-memory only when omitted.", + description="Directory the run works in: runtimes write trial evidence beneath it, and it is " + "the default target for AgentEvalResult.persist so the bundle contains that evidence. Purely " + "in-memory when omitted.", ) run_id: str | None = Field(default=None, description="Explicit run identifier; generated when omitted.") prompt_template: str | dict[str, Any] | None = Field( @@ -228,7 +230,6 @@ class AgentEvalRunConfig(BaseModel): description="Inference/run parameters used when producing trials online.", ) parallelism: int = Field(default=4, ge=1, description="Maximum number of tasks scored concurrently.") - write_dashboard: bool = Field(default=True, description="Whether to render an HTML dashboard for the run.") labels: dict[str, str] = Field( default_factory=dict, description="Caller-supplied tags recorded on the run's metadata (e.g. benchmark, mode, backend, " diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py index a5d0a385f3..c52c31d71d 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py @@ -295,9 +295,10 @@ class AggregateScoreBase(BaseModel): name: str = Field(description="Name of the score.") count: int | None = Field( default=None, - description="Number of samples evaluated (excluding NaN). Serialized as null when the sample size is unknown " + description="Number of samples evaluated (excluding NaN). Omitted when the sample size is unknown " "— e.g. a figure imported from a backend that reports statistics without the n behind them. " - "Distinct from 0, which asserts that nothing was evaluated.", + "Distinct from 0, which asserts that nothing was evaluated. (``None`` on the model; the result " + "routes serialize with exclude_none, so the field is absent from the response rather than null.)", ) nan_count: int = Field(description="Number of samples that produced NaN scores.") sum: float | None = Field(default=None, description="Sum of all score values.") @@ -386,7 +387,8 @@ class AggregateScalarScore(AggregateScoreBase): For figures a backend reports as one number (e.g. an environment's own ``pass@1`` or Elo) rather than a set of per-sample values the SDK could aggregate itself. ``value`` carries the number; - ``mean``/``min``/``max`` are left unset because there is no sample to describe. Distinct from + ``mean``/``min``/``max`` are optional and normally unset, since there is no sample to describe — + a producer may still supply them, but readers key off ``score_type`` and read ``value``. Distinct from :class:`AggregateRangeScore` so a reader can tell "this is the whole story" from "this summarizes ``count`` samples", instead of seeing a range score with a suspicious ``count`` of 1. """ diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_docker_example.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_docker_example.py index da9d5d0553..d7536db36f 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_docker_example.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_docker_example.py @@ -66,7 +66,7 @@ def test_default_output_dir_is_under_repo_temp(monkeypatch: pytest.MonkeyPatch) @pytest.mark.asyncio async def test_codex_docker_example_scores_workspace_artifact(tmp_path: Path) -> None: - result = await codex_docker.evaluate( + result, location = await codex_docker.evaluate( output_dir=tmp_path / "run", runtime=_FakeCodexRuntime(tmp_path / "workspace"), write_dashboard=False, @@ -80,6 +80,8 @@ async def test_codex_docker_example_scores_workspace_artifact(tmp_path: Path) -> "workspace_artifact.output_matches": True, } assert (tmp_path / "run" / "run.json").is_file() + assert location.output_dir == tmp_path / "run" + assert location.dashboard_path is None # write_dashboard=False @pytest.mark.asyncio diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_runtime.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_runtime.py index bdee5769d3..711044e693 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_runtime.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_runtime.py @@ -14,7 +14,7 @@ from nemo_evaluator_sdk.agent_eval import workspace_seeds from nemo_evaluator_sdk.agent_eval.runtimes.codex import runtime as codex_runtime from nemo_evaluator_sdk.agent_eval.runtimes.docker_sandbox import DockerSandboxAgentRuntime -from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask +from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask from pydantic import BaseModel # The runtime *selection* (local vs docker-cli vs docker-sandbox) is generic and lives here; only the @@ -69,6 +69,22 @@ def test_resolve_codex_runtime_docker_falls_back_to_cli_without_sdk_key(tmp_path assert effective == codex_runtime.EffectiveCodexRuntime.DOCKER_CLI +def test_evidence_dir_prefers_an_explicit_work_root_over_the_run_config(tmp_path: Path) -> None: + # An explicit ``work_root`` is a caller decision about where evidence goes, so it wins over the + # run's ``work_dir``; without one the runtime derives ``/evidence/codex``. Pointing + # ``work_root`` outside ``work_dir`` is legal and keeps working -- persist() leaves such refs + # absolute (test_persist_and_read_keep_external_evidence_refs_absolute) rather than dropping them, + # so the bundle resolves for as long as that directory is there. + task = AgentEvalTask(id="taskA", intent="do a thing", inputs={}) + config = AgentEvalRunConfig(work_dir=tmp_path / "run") + + explicit = codex_runtime.CodexCliAgentRuntime(work_root=tmp_path / "elsewhere") + assert explicit._evidence_dir(0, task, config) == tmp_path / "elsewhere" / "000000-taskA" + + derived = codex_runtime.CodexCliAgentRuntime() + assert derived._evidence_dir(0, task, config) == tmp_path / "run" / "evidence" / "codex" / "000000-taskA" + + def test_list_codex_agent_models_prints_visible_models( monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_docker_sandbox_runtime.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_docker_sandbox_runtime.py index ad62495708..c33faae163 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_docker_sandbox_runtime.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_docker_sandbox_runtime.py @@ -272,7 +272,7 @@ async def test_completed_run_writes_artifacts_and_evidence(monkeypatch: pytest.M trials = await runtime.run_tasks( [_task()], - config=AgentEvalRunConfig(output_dir=tmp_path, run_id="run-1", parallelism=1), + config=AgentEvalRunConfig(work_dir=tmp_path, run_id="run-1", parallelism=1), ) evidence_dir = tmp_path / "agent-runtime" / "run-1" / "000000-task-1" @@ -304,7 +304,7 @@ async def test_runtime_creates_and_deletes_one_sandbox_per_task( await runtime.run_tasks( [_task(task_id="task-1"), _task(task_id="task-2")], - config=AgentEvalRunConfig(output_dir=tmp_path, run_id="run-1", parallelism=2), + config=AgentEvalRunConfig(work_dir=tmp_path, run_id="run-1", parallelism=2), ) assert len(client.created) == 2 @@ -322,7 +322,7 @@ async def test_direct_runtime_call_uses_one_generated_run_id( await runtime.run_tasks( [_task(task_id="task-1"), _task(task_id="task-2")], - config=AgentEvalRunConfig(output_dir=tmp_path, parallelism=2), + config=AgentEvalRunConfig(work_dir=tmp_path, parallelism=2), ) run_dirs = list((tmp_path / "agent-runtime").iterdir()) @@ -343,7 +343,7 @@ async def test_parallelism_limits_concurrent_task_runs( await runtime.run_tasks( [_task(task_id=f"task-{index}") for index in range(4)], - config=AgentEvalRunConfig(output_dir=tmp_path, run_id="run-1", parallelism=2), + config=AgentEvalRunConfig(work_dir=tmp_path, run_id="run-1", parallelism=2), ) assert runner.max_active == 2 @@ -360,7 +360,7 @@ async def test_runtime_exception_returns_failed_trial( trials = await runtime.run_tasks( [_task()], - config=AgentEvalRunConfig(output_dir=tmp_path, run_id="run-1", parallelism=1), + config=AgentEvalRunConfig(work_dir=tmp_path, run_id="run-1", parallelism=1), ) error_path = tmp_path / "agent-runtime" / "run-1" / "000000-task-1" / "error.json" diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py index f221924518..94d4efec56 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py @@ -298,16 +298,38 @@ def test_run_rejects_trials_and_target_together() -> None: ) +@pytest.mark.asyncio +async def test_run_writes_nothing_until_persist_is_called(tmp_path: Path) -> None: + # The point of the change: computing an evaluation and storing one are separate decisions, so a + # run given a work_dir still leaves it empty until the caller asks for a bundle. + result = await AgentEvaluator().run( + tasks=[_task()], + trials=[_candidate_trial()], + config=AgentEvalRunConfig(work_dir=tmp_path, parallelism=1), + ) + + assert not (tmp_path / "run.json").exists() + assert not (tmp_path / "report.html").exists() + # work_dir comes from the config, so it is known at construction — never patched on afterwards. + assert result.work_dir == tmp_path + + result.persist() + assert (tmp_path / "run.json").is_file() + + @pytest.mark.asyncio async def test_scores_imported_trials_with_metric_and_persists_bundle(tmp_path: Path) -> None: result = await AgentEvaluator().run( tasks=[_task()], trials=[_candidate_trial()], - config=AgentEvalRunConfig(output_dir=tmp_path, parallelism=1), + config=AgentEvalRunConfig(work_dir=tmp_path, parallelism=1), ) + # run() no longer writes anything; persisting is the caller's call and defaults to the work_dir. + location = result.persist() assert _score(result.summary, "constant_metric.score").mean == 0.75 - assert result.dashboard_path == tmp_path / "report.html" + assert location.output_dir == tmp_path + assert location.dashboard_path == tmp_path / "report.html" assert (tmp_path / "run.json").exists() assert (tmp_path / "scores.jsonl").exists() assert "run_id" not in json.loads((tmp_path / "metadata.json").read_text(encoding="utf-8")) @@ -656,7 +678,6 @@ async def test_generation_boundary_names_agent_eval_context() -> None: config=AgentEvalRunConfig( run_id="run-123", params=RunConfigOnline(parallelism=1), - write_dashboard=False, ), ) @@ -692,10 +713,9 @@ async def test_default_agent_invocation_receives_run_context_and_evidence_dir(tm target=agent, config=AgentEvalRunConfig( run_id="run-123", - output_dir=tmp_path, + work_dir=tmp_path, prompt_template=prompt_template, params=RunConfigOnline(parallelism=1), - write_dashboard=False, ), ) @@ -736,9 +756,8 @@ async def fake_invoke(agent: Agent, request: dict[str, Any], **kwargs: Any) -> A target=agent, config=AgentEvalRunConfig( run_id="run-123", - output_dir=tmp_path, + work_dir=tmp_path, params=RunConfigOnline(parallelism=1), - write_dashboard=False, ), ) @@ -776,9 +795,8 @@ def factory(context: AgentInferenceContext): target=agent, config=AgentEvalRunConfig( run_id="run-123", - output_dir=tmp_path, + work_dir=tmp_path, params=RunConfigOnline(parallelism=1), - write_dashboard=False, ), ) diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py index e9a6f3694b..1ef7e2b764 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py @@ -135,7 +135,7 @@ def _runtime(provider: _FakeProvider, **kwargs: object) -> FabricContainerRuntim async def _run(runtime: FabricContainerRuntime, tasks: list[AgentEvalTask], tmp_path: Path) -> Sequence[AgentEvalTrial]: - return await runtime.run_tasks(tasks, AgentEvalRunConfig(output_dir=tmp_path)) + return await runtime.run_tasks(tasks, AgentEvalRunConfig(work_dir=tmp_path)) def _task() -> AgentEvalTask: @@ -480,7 +480,7 @@ async def test_native_skill_preserves_preconfigured_skill_paths( skill = AgentSkill.from_directory(_skill_bundle(tmp_path / "src")) provider = _FakeProvider() runtime = FabricContainerRuntime(config, provider=provider, skills=[skill]) # type: ignore[arg-type] - await runtime.run_tasks([_task()], AgentEvalRunConfig(output_dir=tmp_path)) + await runtime.run_tasks([_task()], AgentEvalRunConfig(work_dir=tmp_path)) paths = _seeded_skill_paths(provider) assert paths[:2] == ["/pre/existing-a", "/pre/existing-b"] @@ -497,7 +497,7 @@ async def test_native_skill_on_runtime_discovered_adapter(tmp_path: Path, monkey skill = AgentSkill.from_directory(_skill_bundle(tmp_path / "src")) provider = _FakeProvider() runtime = FabricContainerRuntime(custom, provider=provider, skills=[skill]) # type: ignore[arg-type] - (trial,) = await runtime.run_tasks([_task()], AgentEvalRunConfig(output_dir=tmp_path)) + (trial,) = await runtime.run_tasks([_task()], AgentEvalRunConfig(work_dir=tmp_path)) assert "/in/skills/code-review" in _seeded_skill_paths(provider) assert trial.metadata["skill"]["mode"] == "native" @@ -520,7 +520,7 @@ async def download_dir(self, handle: SandboxHandle, source_dir: str, target_dir: skill = AgentSkill.from_directory(_skill_bundle(tmp_path / "src")) provider = _CodexWorkspaceProvider() runtime = FabricContainerRuntime(_CODEX_CONFIG, provider=provider, skills=[skill]) # type: ignore[arg-type] - (trial,) = await runtime.run_tasks([_task()], AgentEvalRunConfig(output_dir=tmp_path)) + (trial,) = await runtime.run_tasks([_task()], AgentEvalRunConfig(work_dir=tmp_path)) # Codex discovers agentskills from .agents/skills/ in its working dir, so the bundle is seeded there in # the workspace (not /in), for the harness to self-discover during the run. @@ -545,7 +545,7 @@ async def test_skill_on_unsupported_adapter_fails_fast(tmp_path: Path, monkeypat runtime = FabricContainerRuntime(unsupported, provider=_FakeProvider(), skills=[skill]) # type: ignore[arg-type] with pytest.raises(RuntimeError, match="no known skill-injection strategy"): - await runtime.run_tasks([_task()], AgentEvalRunConfig(output_dir=tmp_path)) + await runtime.run_tasks([_task()], AgentEvalRunConfig(work_dir=tmp_path)) async def test_no_skill_leaves_metadata_none_and_skips_planner(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: @@ -605,7 +605,7 @@ async def download_dir(self, handle: SandboxHandle, source_dir: str, target_dir: ] provider = _CodexWorkspaceProvider() runtime = FabricContainerRuntime(_CODEX_CONFIG, provider=provider, skills=skills) # type: ignore[arg-type] - (trial,) = await runtime.run_tasks([_task()], AgentEvalRunConfig(output_dir=tmp_path)) + (trial,) = await runtime.run_tasks([_task()], AgentEvalRunConfig(work_dir=tmp_path)) # Both bundles seeded under the codex discovery dir, no skills path, all scrubbed from evidence. assert provider.seeded["/out/workspace/.agents/skills/docx/SKILL.md"].startswith("---") @@ -663,7 +663,7 @@ async def test_same_skill_from_both_injection_and_task_files_fails_task( "files": {".agents/skills/code-review/SKILL.md": "# override"}, }, ) - (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(output_dir=tmp_path)) + (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(work_dir=tmp_path)) assert trial.status == AgentEvalTrialStatus.FAILED error = json.loads(Path(trial.evidence.require("error").ref).read_text()) # type: ignore[arg-type] @@ -692,7 +692,7 @@ async def test_task_seeded_skill_coexists_with_a_different_injected_skill( "files": {".agents/skills/style-guide/SKILL.md": "---\nname: style-guide\n---\n"}, }, ) - (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(output_dir=tmp_path)) + (trial,) = await runtime.run_tasks([task], AgentEvalRunConfig(work_dir=tmp_path)) assert trial.status == AgentEvalTrialStatus.COMPLETED assert "/out/workspace/.agents/skills/code-review/SKILL.md" in provider.seeded diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py index fd75eca7f8..4e55ec16f7 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py @@ -210,7 +210,7 @@ def __init__(self, **kwargs: Any) -> None: result = AgentEvaluator().run_sync( tasks=[_task()], target=runtime, - config=AgentEvalRunConfig(output_dir=tmp_path / "out", parallelism=1, write_dashboard=False), + config=AgentEvalRunConfig(work_dir=tmp_path / "out", parallelism=1), ) trial = result.trials[0] @@ -293,7 +293,7 @@ def test_fabric_codex_live_eval_captures_atif_trajectory(tmp_path: Path) -> None result = AgentEvaluator().run_sync( tasks=[_task()], target=runtime, - config=AgentEvalRunConfig(output_dir=tmp_path / "out", parallelism=1, write_dashboard=False), + config=AgentEvalRunConfig(work_dir=tmp_path / "out", parallelism=1), ) trial = result.trials[0] diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py index 7b53c1c6c8..bf19d2ac02 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py @@ -88,7 +88,7 @@ async def test_harbor_runner_scores_through_agent_evaluator_and_adapts_legacy_pa # run_job is awaited exactly once, then the job dir is adapted and scored end-to-end. calls = [] runner = HarborAgentTaskRunner(job_dir=job_dir, run_job=lambda: _record(calls)) - result = await AgentEvaluator().run(tasks=tasks, target=runner, config=AgentEvalRunConfig(write_dashboard=False)) + result = await AgentEvaluator().run(tasks=tasks, target=runner, config=AgentEvalRunConfig()) assert calls == ["ran"] rewards_by_task = {score.task_id: score.outputs[0].value for score in result.scores if score.outputs} diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_persistence.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_persistence.py index eed52ca583..521e3c6d68 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_persistence.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_persistence.py @@ -9,6 +9,7 @@ import shutil from pathlib import Path +import pytest from nemo_evaluator_sdk.agent_eval.persistence import persist_run, read_trials from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult, AgentEvalSummary from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial, AgentEvalTrialStatus, AgentOutput @@ -120,3 +121,46 @@ def test_persist_and_read_keep_external_evidence_refs_absolute(tmp_path: Path) - (trial,) = read_trials(bundle) assert trial.evidence.require("workspace").ref == external_ref # retained as-is + + +def test_persist_defaults_to_the_work_dir_so_evidence_lands_inside_the_bundle(tmp_path: Path) -> None: + # Defaulting to work_dir is what keeps a bundle self-contained: the evidence the trials point at + # is already underneath it, so persist can rewrite the refs bundle-relative. + workspace = tmp_path / "evidence" / "000000-taskA" / "workspace" + workspace.mkdir(parents=True) + result = AgentEvalResult( + run_id="r", + tasks=[], + trials=[_trial_with_workspace(str(workspace))], + scores=[], + summary=AgentEvalSummary.from_scores([], tasks=[]), + work_dir=tmp_path, + ) + + location = result.persist(write_dashboard=False) + + assert location.output_dir == tmp_path + stored = json.loads((tmp_path / "trials.jsonl").read_text(encoding="utf-8"))["evidence"]["descriptors"] + assert stored["workspace"]["ref"] == "evidence/000000-taskA/workspace" # relative => self-contained + + +def test_persist_without_a_work_dir_or_an_explicit_target_is_an_error() -> None: + # An in-memory run has nowhere to go; failing loudly beats inventing a directory. + result = AgentEvalResult( + run_id="r", tasks=[], trials=[], scores=[], summary=AgentEvalSummary.from_scores([], tasks=[]) + ) + + with pytest.raises(ValueError, match="no work_dir"): + result.persist() + + +def test_persist_accepts_an_explicit_target_for_an_in_memory_run(tmp_path: Path) -> None: + # The counterpart to the error above: an in-memory run has no default, but naming one works. + result = AgentEvalResult( + run_id="r", tasks=[], trials=[], scores=[], summary=AgentEvalSummary.from_scores([], tasks=[]) + ) + + location = result.persist(tmp_path, write_dashboard=False) + + assert location.output_dir == tmp_path + assert json.loads((tmp_path / "run.json").read_text(encoding="utf-8"))["run_id"] == "r" diff --git a/plugins/nemo-evaluator/openapi/openapi.yaml b/plugins/nemo-evaluator/openapi/openapi.yaml index 366017a676..e4a08e7c26 100644 --- a/plugins/nemo-evaluator/openapi/openapi.yaml +++ b/plugins/nemo-evaluator/openapi/openapi.yaml @@ -2676,10 +2676,12 @@ components: description: Name of the score. count: title: Count - description: "Number of samples evaluated (excluding NaN). Serialized as\ - \ null when the sample size is unknown \u2014 e.g. a figure imported from\ - \ a backend that reports statistics without the n behind them. Distinct\ - \ from 0, which asserts that nothing was evaluated." + description: "Number of samples evaluated (excluding NaN). Omitted when\ + \ the sample size is unknown \u2014 e.g. a figure imported from a backend\ + \ that reports statistics without the n behind them. Distinct from 0,\ + \ which asserts that nothing was evaluated. (``None`` on the model; the\ + \ result routes serialize with exclude_none, so the field is absent from\ + \ the response rather than null.)" type: integer nan_count: type: integer @@ -2759,10 +2761,12 @@ components: description: Name of the score. count: title: Count - description: "Number of samples evaluated (excluding NaN). Serialized as\ - \ null when the sample size is unknown \u2014 e.g. a figure imported from\ - \ a backend that reports statistics without the n behind them. Distinct\ - \ from 0, which asserts that nothing was evaluated." + description: "Number of samples evaluated (excluding NaN). Omitted when\ + \ the sample size is unknown \u2014 e.g. a figure imported from a backend\ + \ that reports statistics without the n behind them. Distinct from 0,\ + \ which asserts that nothing was evaluated. (``None`` on the model; the\ + \ result routes serialize with exclude_none, so the field is absent from\ + \ the response rather than null.)" type: integer nan_count: type: integer @@ -2844,10 +2848,12 @@ components: description: Name of the score. count: title: Count - description: "Number of samples evaluated (excluding NaN). Serialized as\ - \ null when the sample size is unknown \u2014 e.g. a figure imported from\ - \ a backend that reports statistics without the n behind them. Distinct\ - \ from 0, which asserts that nothing was evaluated." + description: "Number of samples evaluated (excluding NaN). Omitted when\ + \ the sample size is unknown \u2014 e.g. a figure imported from a backend\ + \ that reports statistics without the n behind them. Distinct from 0,\ + \ which asserts that nothing was evaluated. (``None`` on the model; the\ + \ result routes serialize with exclude_none, so the field is absent from\ + \ the response rather than null.)" type: integer nan_count: type: integer @@ -2914,23 +2920,15 @@ components: - nan_count - value title: AggregateScalarScore - description: 'A single pre-computed value with no underlying distribution available. - - - For figures a backend reports as one number (e.g. an environment''s own ``pass@1`` - or Elo) rather - - than a set of per-sample values the SDK could aggregate itself. ``value`` - carries the number; - - ``mean``/``min``/``max`` are left unset because there is no sample to describe. - Distinct from - - :class:`AggregateRangeScore` so a reader can tell "this is the whole story" - from "this summarizes - - ``count`` samples", instead of seeing a range score with a suspicious ``count`` - of 1.' + description: "A single pre-computed value with no underlying distribution available.\n\ + \nFor figures a backend reports as one number (e.g. an environment's own ``pass@1``\ + \ or Elo) rather\nthan a set of per-sample values the SDK could aggregate\ + \ itself. ``value`` carries the number;\n``mean``/``min``/``max`` are optional\ + \ and normally unset, since there is no sample to describe \u2014\na producer\ + \ may still supply them, but readers key off ``score_type`` and read ``value``.\ + \ Distinct from\n:class:`AggregateRangeScore` so a reader can tell \"this\ + \ is the whole story\" from \"this summarizes\n``count`` samples\", instead\ + \ of seeing a range score with a suspicious ``count`` of 1." AggregatedMetricResult: properties: scores: diff --git a/plugins/nemo-evaluator/src/nemo_evaluator/intake/publish.py b/plugins/nemo-evaluator/src/nemo_evaluator/intake/publish.py index be5bf60709..45270a4876 100644 --- a/plugins/nemo-evaluator/src/nemo_evaluator/intake/publish.py +++ b/plugins/nemo-evaluator/src/nemo_evaluator/intake/publish.py @@ -212,8 +212,14 @@ def _publish_failure_message( report: PublishReport, failures: list[tuple[str, BaseException]], ) -> str: - """Build an actionable error: what failed, where the results are cached, how to recover.""" - location = f"cached locally at {result.output_dir}" if result.output_dir is not None else "in the local run bundle" + """Build an actionable error: what failed, what survives locally, how to recover.""" + # work_dir is where the runtimes wrote trial evidence, NOT a bundle location: a bundle exists only + # if the caller chose to persist(), and persist() can be pointed elsewhere. Say only what is true. + location = ( + f"still in memory, and the run's trial evidence is under {result.work_dir}" + if result.work_dir is not None + else "still in memory and have not been written to disk (call result.persist() to keep them)" + ) detail = "\n ".join(f"{trial_id}: {type(error).__name__}: {error}" for trial_id, error in failures) return ( f"publish_to_intake: {len(failures)} of {len(result.trials)} trial(s) failed to publish " diff --git a/plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py b/plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py index b1acb2ac3b..d3ed7b9110 100644 --- a/plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py +++ b/plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py @@ -43,7 +43,6 @@ from nemo_evaluator.shared.metric_bundles.bundles import unbundle_metric from nemo_evaluator.task_refs import resolve_agent_eval_tasks from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator -from nemo_evaluator_sdk.agent_eval.persistence import persist_run from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult from nemo_evaluator_sdk.agent_eval.runtimes.codex.runtime import CodexCliAgentRuntime from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime @@ -373,7 +372,9 @@ def _resolve_target( def _write_result_files(result: AgentEvalResult, persistent_dir: Path) -> AgentEvalResultFiles: """Persist the run bundle (trials/scores/tasks/summary) under the job's storage.""" bundle_dir = persistent_dir / AGENT_BUNDLE_DIR - persist_run(result, bundle_dir) + # No HTML dashboard for job runs: the artifact is consumed programmatically, and the job + # config asked for no dashboard before persistence became an explicit call. + result.persist(bundle_dir, write_dashboard=False) return AgentEvalResultFiles(bundle_dir=bundle_dir, summary=bundle_dir / SUMMARY_FILE_NAME) def run( @@ -394,7 +395,6 @@ def run( parallelism=spec.max_concurrent_tasks, labels=spec.labels, fail_fast=spec.fail_fast, - write_dashboard=False, ) # `run` may be injected a sync `sdk` (submitted jobs, via get_task_sdk) and/or an # `async_sdk`; forward whichever identity is present, preferring async when both are — the diff --git a/plugins/nemo-optimization/src/nemo_optimization/backends/optuna/fabric_trial.py b/plugins/nemo-optimization/src/nemo_optimization/backends/optuna/fabric_trial.py index 200f868a96..d109efee18 100644 --- a/plugins/nemo-optimization/src/nemo_optimization/backends/optuna/fabric_trial.py +++ b/plugins/nemo-optimization/src/nemo_optimization/backends/optuna/fabric_trial.py @@ -100,14 +100,16 @@ def evaluate( tasks=tasks, target=runtime, config=AgentEvalRunConfig( - output_dir=self._trial_output_dir(trial_number, rep), + work_dir=self._trial_output_dir(trial_number, rep), parallelism=self._parallelism, - write_dashboard=False, # Keep scoring the rest of the dataset when one metric raises; # reduce_agent_eval_scores skips FAILED task scores. fail_fast=False, ), ) + # ``run`` no longer stores the bundle; keep writing one per trial as before. No dashboard — + # each trial bundle is intermediate evidence for the study, not something anyone opens. + result.persist(write_dashboard=False) self._record_traces(result, trial_number=trial_number, rep=rep) self._write_trace_map() return reduce_agent_eval_scores(result.scores, self._metric_names) diff --git a/plugins/nemo-optimization/tests/test_fabric_trial.py b/plugins/nemo-optimization/tests/test_fabric_trial.py index ac209e2794..a37f8ebab4 100644 --- a/plugins/nemo-optimization/tests/test_fabric_trial.py +++ b/plugins/nemo-optimization/tests/test_fabric_trial.py @@ -291,6 +291,9 @@ def run_sync(self, *, tasks, target, config): # noqa: ANN001 trials=[trial], scores=[score], summary=AgentEvalSummary.from_scores([score], tasks=tasks), + # The real evaluator carries the config's work_dir onto the result; the trial + # evaluator persists into it, so the fake has to do the same to stay faithful. + work_dir=config.work_dir, ) monkeypatch.setattr("nemo_optimization.backends.optuna.fabric_trial.FabricAgentRuntime", FakeRuntime) @@ -322,6 +325,10 @@ def run_sync(self, *, tasks, target, config): # noqa: ANN001 assert "optimizer" not in captured["runtime"]["config"] assert "eval" not in captured["runtime"]["config"] assert (tmp_path / "out" / "trial_trace_map.json").is_file() + # Storing is now an explicit persist() call, so assert the per-trial bundle still lands. + bundle = tmp_path / "out" / "agent_eval" / "trial-007" / "rep-000" + assert (bundle / "run.json").is_file() + assert not (bundle / "report.html").exists() # write_dashboard=False trace_map = json.loads((tmp_path / "out" / "trial_trace_map.json").read_text(encoding="utf-8")) assert trace_map[0]["experiment_id"] == "exp-test" assert trace_map[0]["row_id"] == "1" diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py index fe354b9968..bae3b15c29 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py @@ -21,8 +21,6 @@ import httpx import nemo_platform.beta.evaluator.inference as inference -from nemo_platform.beta.evaluator.agent_eval.dashboard import write_dashboard -from nemo_platform.beta.evaluator.agent_eval.persistence import persist_run from nemo_platform.beta.evaluator.agent_eval.results import AgentEvalResult, AgentEvalSummary, RunMetadata from nemo_platform.beta.evaluator.agent_eval.scores import ( AgentEvalDiagnostic, @@ -196,10 +194,9 @@ async def run( scores=scores, summary=AgentEvalSummary.from_scores(scores, tasks=task_list, extra_scores=runner_scores), metadata=metadata, + work_dir=runtime_config.work_dir, ) - if runtime_config.output_dir is not None: - result = _persist_with_optional_dashboard(result, runtime_config.output_dir, runtime_config.write_dashboard) return result def run_sync( @@ -342,8 +339,8 @@ async def generate_one(index: int, task: AgentEvalTask) -> AgentEvalTrial: "invocation_id": f"{config.run_id}:{task.id}:{target.name}", } evidence_dir = ( - _task_evidence_dir(Path(config.output_dir), index=index, task_id=task.id) - if config.output_dir is not None and isinstance(target, AgentBase) + _task_evidence_dir(Path(config.work_dir), index=index, task_id=task.id) + if config.work_dir is not None and isinstance(target, AgentBase) else None ) resolved_inference_fn = self.inference_fn @@ -770,18 +767,6 @@ def _collect_runner_aggregate_scores(target: object) -> list[AggregateScore]: return collected -def _persist_with_optional_dashboard( - result: AgentEvalResult, - output_dir: Path, - write_html: bool, -) -> AgentEvalResult: - path = Path(output_dir) - dashboard_path = None - if write_html: - dashboard_path = write_dashboard(result.model_copy(update={"output_dir": path}), path / "report.html") - return persist_run(result.model_copy(update={"output_dir": path, "dashboard_path": dashboard_path}), path) - - def _new_run_id() -> str: timestamp = datetime.now(UTC).strftime("%Y%m%d%H%M%S") return f"agent-eval-{timestamp}-{uuid.uuid4().hex[:8]}" diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.py index ac152bf09a..0f8d158c97 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.py @@ -10,40 +10,59 @@ from pathlib import Path from typing import Any -from nemo_platform.beta.evaluator.agent_eval.results import AgentEvalResult +from nemo_platform.beta.evaluator.agent_eval.dashboard import write_dashboard +from nemo_platform.beta.evaluator.agent_eval.results import AgentEvalResult, BundleLocation from nemo_platform.beta.evaluator.agent_eval.trials import AgentEvalTrial from pydantic import BaseModel +#: Filename of the rendered HTML dashboard inside a bundle. +DASHBOARD_FILENAME = "report.html" -def persist_run(result: AgentEvalResult, output_dir: str | Path) -> AgentEvalResult: - """Persist a completed run bundle to ``output_dir``.""" + +def persist_run( + result: AgentEvalResult, + output_dir: str | Path, + *, + write_html_dashboard: bool = True, +) -> BundleLocation: + """Write a completed run to a bundle at ``output_dir`` and report where it landed. + + Explicit rather than a side effect of :meth:`AgentEvaluator.run`: computing an evaluation and + storing one are different decisions, and folding them together is what forced the result object to + carry paths it could not know at construction time. (Same reasoning as ``publish_to_intake``.) + + Set ``write_html_dashboard=False`` to skip rendering ``report.html`` — the dashboard is written + here so the manifest can record it in a single pass. + """ path = Path(output_dir) path.mkdir(parents=True, exist_ok=True) + # Render first so the manifest below can name it; the dashboard reads only the run's own contents. + dashboard_path = write_dashboard(result, path / DASHBOARD_FILENAME) if write_html_dashboard else None + _write_json(path / "metadata.json", result.metadata) _write_jsonl(path / "tasks.jsonl", result.tasks) _write_trials(path / "trials.jsonl", result.trials, base=path) _write_jsonl(path / "scores.jsonl", result.scores) _write_json(path / "summary.json", result.summary) - updated = result.model_copy(update={"output_dir": path}) - _write_json(path / "run.json", _run_manifest(updated)) - return updated + location = BundleLocation(output_dir=path, dashboard_path=dashboard_path) + _write_json(path / "run.json", _run_manifest(result, location)) + return location -def _run_manifest(result: AgentEvalResult) -> dict[str, Any]: - artifacts = { - "metadata": "metadata.json", - "tasks": "tasks.jsonl", - "trials": "trials.jsonl", - "scores": "scores.jsonl", - "summary": "summary.json", - } +def _run_manifest(result: AgentEvalResult, location: BundleLocation) -> dict[str, Any]: return { "run_id": result.run_id, - "output_dir": str(result.output_dir) if result.output_dir is not None else None, - "dashboard_path": str(result.dashboard_path) if result.dashboard_path is not None else None, - "artifacts": artifacts, + "output_dir": str(location.output_dir), + "dashboard_path": str(location.dashboard_path) if location.dashboard_path is not None else None, + "artifacts": { + "metadata": "metadata.json", + "tasks": "tasks.jsonl", + "trials": "trials.jsonl", + "scores": "scores.jsonl", + "summary": "summary.json", + }, } diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/results.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/results.py index 5ef83f94b7..cddf78e2ef 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/results.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/results.py @@ -109,8 +109,30 @@ class RunMetadata(BaseModel): sdk_version: str | None = Field(default=None, description="nemo-evaluator-sdk version that produced the run.") +class BundleLocation(BaseModel): + """Where a run was written, returned by :meth:`AgentEvalResult.persist`. + + Kept off :class:`AgentEvalResult` because it is not a property of the evaluation — it is the + outcome of choosing to store it. Holding one means the bundle exists, so there is no optional to + re-check; a run that was never persisted simply has no ``BundleLocation``. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + output_dir: Path = Field(description="Directory the run bundle was written to.") + dashboard_path: Path | None = Field( + default=None, + description="Path to the rendered HTML dashboard, or None when dashboard writing was disabled.", + ) + + class AgentEvalResult(BaseModel): - """Root result for a completed agent evaluation: tasks, trials, scores, summary, and bundle metadata.""" + """Root result for a completed agent evaluation: tasks, trials, scores, and summary. + + Describes the evaluation and nothing else — storing it is a separate decision, made by calling + :meth:`persist`. Because the result carries no paths, it never holds a location that was unknown + when it was constructed, and nothing has to mutate it after the fact. + """ model_config = ConfigDict(extra="forbid") @@ -123,8 +145,39 @@ class AgentEvalResult(BaseModel): default_factory=RunMetadata, description="Run provenance: labels, target identity, timings, SDK version.", ) - output_dir: Path | None = Field(default=None, description="Directory the run bundle was written to, if any.") - dashboard_path: Path | None = Field(default=None, description="Path to the rendered dashboard, if written.") + work_dir: Path | None = Field( + default=None, + description="Directory the run worked in, where its runtimes wrote trial evidence. Known " + "before the run starts (it comes from the run config), so unlike a bundle location it is " + "never attached after the fact. None for a purely in-memory run.", + ) + + def persist(self, output_dir: str | Path | None = None, *, write_dashboard: bool = True) -> BundleLocation: + """Write this run to a bundle and return where it landed. + + Deliberately a call rather than something ``AgentEvaluator.run`` does for you: computing an + evaluation and storing one are separate decisions (the same reasoning as ``publish_to_intake``). + + Defaults to :attr:`work_dir`, which is the directory the trials' evidence already lives under — + so the bundle is self-contained and survives being moved. Passing a different ``output_dir`` + leaves those evidence references pointing back at the original directory. That is supported (a + re-scored run may reference an earlier run's deliverables) but the resulting bundle only + resolves while the original directory is still there. + + Set ``write_dashboard=False`` to skip rendering ``report.html``. + """ + # Imported here rather than at module scope: persistence imports this module for the types it + # writes, so a top-level import would be circular. + from nemo_platform.beta.evaluator.agent_eval.persistence import persist_run + + target = output_dir if output_dir is not None else self.work_dir + if target is None: + raise ValueError( + "this run has no work_dir to persist into (it ran in memory); pass an explicit " + "output_dir, or set work_dir on the AgentEvalRunConfig so evidence and bundle share " + "a directory" + ) + return persist_run(self, target, write_html_dashboard=write_dashboard) def _aggregate_scores( diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/codex/runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/codex/runtime.py index ec8cb76601..a66e4c99a9 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/codex/runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/codex/runtime.py @@ -288,7 +288,7 @@ def _validate_artifact_permissions(self, evidence_dir: Path) -> None: def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: root = self._work_root if root is None: - root = (config.output_dir or Path.cwd()) / "evidence" / "codex" + root = (config.work_dir or Path.cwd()) / "evidence" / "codex" safe_task_id = _safe_path_name(task.id) task_dir = f"{index:06d}-{safe_task_id}" if safe_task_id else f"task-{index:06d}" return Path(root) / task_dir diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py index 6ac195d5a2..ac7f2544d5 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py @@ -289,7 +289,7 @@ def _failed_trial(self, task: AgentEvalTask, exc: Exception, evidence_dir: Path) ) def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: - root = config.output_dir if config.output_dir is not None else self._work_root + root = config.work_dir if config.work_dir is not None else self._work_root if root is None: root = Path(tempfile.gettempdir()) / "nemo-evaluator-agent-runtime" run_id = config.run_id or _new_runtime_run_id() diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/container_runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/container_runtime.py index b145ae6c5c..882919befc 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/container_runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/container_runtime.py @@ -485,7 +485,7 @@ def _failed_trial( def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: # Evidence lands under the run's output dir (like every other runtime); the container's own # working state lives at /out inside the sandbox and is downloaded here. - root = (config.output_dir or Path.cwd()) / "evidence" / "fabric_container" + root = (config.work_dir or Path.cwd()) / "evidence" / "fabric_container" return root / _common.task_subdir_name(index, task.id) diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py index 44ef96086b..712128bd4e 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py @@ -635,7 +635,7 @@ def _relay_config( def _evidence_dir(self, index: int, task: AgentEvalTask, config: AgentEvalRunConfig) -> Path: root = self._work_root if root is None: - root = (config.output_dir or Path.cwd()) / "evidence" / "fabric" + root = (config.work_dir or Path.cwd()) / "evidence" / "fabric" # The run id isolates this run's evidence from other runs sharing the same root (A/B baseline # vs. skilled); run_tasks always populates it, so the fallback only guards a direct call. run_id = config.run_id or _new_run_id() diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py index 4f45b6313a..6667fe93f9 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py @@ -412,8 +412,8 @@ async def run_tasks( # phases — parallelism bounds concurrent scoring (SDK-side, cheap), while Gym's `--concurrency` # bounds concurrent rollouts against the model endpoint during collection (tuned to that endpoint's # limits via GymRuntimeConfig.concurrency). - if config is not None and config.output_dir is not None: - work_dir = Path(config.output_dir) / "gym_run" + if config is not None and config.work_dir is not None: + work_dir = Path(config.work_dir) / "gym_run" else: work_dir = Path(tempfile.mkdtemp(prefix="gym_run_")) work_dir.mkdir(parents=True, exist_ok=True) diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/harbor_runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/harbor_runtime.py index 5683ac0719..01e0092703 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/harbor_runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/harbor_runtime.py @@ -1338,7 +1338,7 @@ async def run_harbor_eval( return await AgentEvaluator().run( tasks=tasks, target=runner, - config=run_config or AgentEvalRunConfig(write_dashboard=False), + config=run_config or AgentEvalRunConfig(), ) diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.py index e2b377c886..b80dabb31b 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.py @@ -214,9 +214,11 @@ class AgentEvalRunConfig(BaseModel): model_config = ConfigDict(extra="forbid") - output_dir: Path | None = Field( + work_dir: Path | None = Field( default=None, - description="Directory where the run bundle is written; in-memory only when omitted.", + description="Directory the run works in: runtimes write trial evidence beneath it, and it is " + "the default target for AgentEvalResult.persist so the bundle contains that evidence. Purely " + "in-memory when omitted.", ) run_id: str | None = Field(default=None, description="Explicit run identifier; generated when omitted.") prompt_template: str | dict[str, Any] | None = Field( @@ -228,7 +230,6 @@ class AgentEvalRunConfig(BaseModel): description="Inference/run parameters used when producing trials online.", ) parallelism: int = Field(default=4, ge=1, description="Maximum number of tasks scored concurrently.") - write_dashboard: bool = Field(default=True, description="Whether to render an HTML dashboard for the run.") labels: dict[str, str] = Field( default_factory=dict, description="Caller-supplied tags recorded on the run's metadata (e.g. benchmark, mode, backend, " diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py index 47083eee8f..9cf87e2674 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py @@ -295,9 +295,10 @@ class AggregateScoreBase(BaseModel): name: str = Field(description="Name of the score.") count: int | None = Field( default=None, - description="Number of samples evaluated (excluding NaN). Serialized as null when the sample size is unknown " + description="Number of samples evaluated (excluding NaN). Omitted when the sample size is unknown " "— e.g. a figure imported from a backend that reports statistics without the n behind them. " - "Distinct from 0, which asserts that nothing was evaluated.", + "Distinct from 0, which asserts that nothing was evaluated. (``None`` on the model; the result " + "routes serialize with exclude_none, so the field is absent from the response rather than null.)", ) nan_count: int = Field(description="Number of samples that produced NaN scores.") sum: float | None = Field(default=None, description="Sum of all score values.") @@ -386,7 +387,8 @@ class AggregateScalarScore(AggregateScoreBase): For figures a backend reports as one number (e.g. an environment's own ``pass@1`` or Elo) rather than a set of per-sample values the SDK could aggregate itself. ``value`` carries the number; - ``mean``/``min``/``max`` are left unset because there is no sample to describe. Distinct from + ``mean``/``min``/``max`` are optional and normally unset, since there is no sample to describe — + a producer may still supply them, but readers key off ``score_type`` and read ``value``. Distinct from :class:`AggregateRangeScore` so a reader can tell "this is the whole story" from "this summarizes ``count`` samples", instead of seeing a range score with a suspicious ``count`` of 1. """