\n Build an agentic evaluation from scratch: define a small suite of realistic coding tasks\n (fix a bug, write tests, write docs), run them against the Codex coding agent driven by\n NeMo Fabric, score each against held-out ground truth the agent can't touch, and roll\n heterogeneous per-task metrics up into one comparable correctness score.\n
\n
"
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-01",
+ "metadata": {},
+ "source": "## What is an *agentic* evaluation?\n\nA classic evaluation scores a **model's answer** against a reference. An *agentic* evaluation scores an\n**agent's behavior on a task** — the agent reads files, runs commands, edits a workspace — and what we\njudge is the *result of that work*.\n\n```\nAgentEvalTask → run against a target → AgentEvalTrial(s) → Metrics → AgentEvalResult\n(intent, inputs, (Codex, driven by (what the agent did: (score each (trials, scores,\n reference, metrics, NeMo Fabric) output + workspace trial vs. summary)\n views) + trajectory) held-out truth)\n```\n\n- **`AgentEvalTask`** — one unit of work: an `intent`, `inputs` (files seeded into the workspace), a\n grader-only **`reference`** (held-out ground truth), the `metrics` that score it, and optional `views`.\n- **Target** — what runs the task. We use **`FabricAgentRuntime`**: NeMo Fabric drives the Codex CLI,\n captures the final workspace **and** the agent's execution trajectory.\n- **`AgentEvalTrial`** — the captured run: the agent's output plus **evidence** (its final `workspace`\n and its ATIF `trace`).\n- **Metric** — scores one trial. It can open the workspace evidence (does a file exist? do the tests\n pass?) and read the task's held-out `reference` — which the agent never saw.\n- **View** — maps a task's *own* metrics into a shared, task-agnostic score. \"Correct\" means different\n things per task (tests pass vs. a docstring exists); a view named `correctness` lets us compare and\n roll them up. **This is how you avoid hand-writing pass/fail logic.**\n- **`AgentEvalResult`** — trials, per-metric scores, and an aggregated summary (including `view.*`\n rollups).\n\n> This uses **`AgentEvaluator`**, the SDK-native path that runs locally and returns results in-process.\n> The evaluator *measures*; deciding pass/fail thresholds is an application concern."
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-02",
+ "metadata": {},
+ "source": "## Notebook map\n\n1. [Install & prerequisites](#prereqs)\n2. [Set up a workspace](#setup)\n3. [Metrics: how a trial gets scored](#metrics)\n4. [Define the task suite](#tasks) — [fix a bug](#task-fix-bug) · [write tests](#task-write-tests) · [write docs](#task-write-docs)\n5. [Pick the target: Codex via NeMo Fabric](#target)\n6. [Run the evaluation](#run)\n7. [Read the results](#results)\n8. [Where to go next](#next)"
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-03",
+ "metadata": {},
+ "source": "\n## 1. Install & prerequisites\n\nThis tutorial evaluates **Codex driven by NeMo Fabric**. Fabric is a native (Rust) component, so —\nunlike a pure-Python SDK — this notebook does **not** run from a plain `pip install`. You need a real\ntoolchain:\n\n- **NeMo Evaluator SDK** — `pip install \"nemo-platform-sdk[nemo-evaluator-sdk]\"` and `pip install pytest` (some metrics run the agent's tests).\n- **NeMo Fabric** (native) with the coding-agent + trajectory extras — `nemo-fabric[codex,relay]` —\n plus a **checkout of the NeMo-Fabric repo**: the Codex adapter registry (`adapters/codex-cli`) is\n resolved relative to it. The repo's `script/dev-install-fabric.sh` installs `nemo-fabric[codex,relay]`\n and builds the `nemo-relay` gateway for you.\n- **Codex CLI** on your `PATH` (`npm install -g @openai/codex`) + auth (`codex login`, or set `OPENAI_API_KEY`).\n- **`nemo-relay` gateway** binary on your `PATH` (built by the script above) — required for ATIF\n trajectory capture.\n\nSet two environment variables before launching (both read from the environment, never written here):\n\n- **`NEMO_FABRIC_REPO`** — path to your NeMo-Fabric checkout (Fabric resolves the adapter registry\n from it).\n- **`NVIDIA_BUILD_API_KEY`** — a build.nvidia.com key for the `write-docs` LLM judge. The cell below\n prompts for it if it isn't already set.\n\nEverything else — defining tasks and metrics, reading results — is plain Python you can read through."
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cell-04",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "# Prompt for the judge's API key unless it's already in the environment. getpass masks the input, so\n",
+ "# the key never lands in the notebook or its saved output.\n",
+ "if not os.environ.get(\"NVIDIA_BUILD_API_KEY\"):\n",
+ " os.environ[\"NVIDIA_BUILD_API_KEY\"] = getpass.getpass(\"build.nvidia.com API key (NVIDIA_BUILD_API_KEY): \")\n",
+ "\n",
+ "# Fabric resolves the Codex adapter registry relative to a NeMo-Fabric checkout. It's a path, not a\n",
+ "# secret, so prompt with plain input() (not getpass) and expand ~ if the environment doesn't set it.\n",
+ "if not os.environ.get(\"NEMO_FABRIC_REPO\"):\n",
+ " os.environ[\"NEMO_FABRIC_REPO\"] = os.path.expanduser(\n",
+ " input(\"Path to your NeMo-Fabric checkout (NEMO_FABRIC_REPO): \").strip()\n",
+ " )\n",
+ "if not os.environ[\"NEMO_FABRIC_REPO\"]:\n",
+ " raise RuntimeError(\"NEMO_FABRIC_REPO is required — set it to your NeMo-Fabric checkout.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-05",
+ "metadata": {},
+ "source": "\n## 2. Set up a workspace\n\nEverything runs locally. We pick an output directory for the run bundle (trials, scores, summary, and\nan HTML dashboard) and import the pieces we'll use."
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cell-06",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import hashlib\n",
+ "import os\n",
+ "import re\n",
+ "import sys\n",
+ "from pathlib import Path\n",
+ "from tempfile import mkdtemp\n",
+ "\n",
+ "from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator\n",
+ "from nemo_evaluator_sdk.agent_eval.metrics import AgentPhaseSuccessMetric\n",
+ "from nemo_evaluator_sdk.agent_eval.tasks import (\n",
+ " AgentEvalRunConfig,\n",
+ " AgentEvalTask,\n",
+ " AgentEvalTaskset,\n",
+ " SemanticReducer,\n",
+ " SemanticView,\n",
+ " ViewSignal,\n",
+ ")\n",
+ "from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult\n",
+ "\n",
+ "OUTPUT_DIR = Path(mkdtemp(prefix=\"agent-eval-\"))\n",
+ "print(\"Run bundle will be written to:\", OUTPUT_DIR)\n",
+ "\n",
+ "\n",
+ "def inline(text: str) -> dict:\n",
+ " \"\"\"An inline seed file — contents carried in the task. Equivalent to passing the bare string.\"\"\"\n",
+ " return {\"kind\": \"inline\", \"content\": text}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-07",
+ "metadata": {},
+ "source": "\n## 3. Metrics: how a trial gets scored\n\nA **metric** implements a tiny protocol — a `type` name, an `output_spec()` declaring the named values\nit emits, and an async `compute_scores(input)` returning them for one trial. The metric receives a\n`MetricInput` whose `candidate` carries the agent's `output_text` **and its evidence** — for a Fabric\ncoding trial that includes the final **`workspace`** directory (and the ATIF **`trace`**). `input.row`\ncarries the task's grader-only **`reference`** at `input.row.data[\"reference\"]`. That combination is\nwhat makes *coding* work scorable against ground truth the agent never saw."
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-08",
+ "metadata": {},
+ "source": "
\n⚠️ Grade only with artifacts the agent cannot edit. \nIf the tests that score a \"fix the bug\" task live in the agent's own writable workspace, a coding\nagent can — and, optimizing for reward, sometimes will — edit the tests instead of fixing the\nbug. The evaluation then measures nothing. Keep ground truth held out: put it in the task's\nreference (grader-only, never seeded into the workspace, never shown to the agent). A\nmetric then either overlays it into a throwaway copy before running a verifier, or checksums\na file the agent was told not to touch. The metrics below do both.\n
"
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-09",
+ "metadata": {},
+ "source": "We define three metrics. `AgentPhaseSuccessMetric` (did the agent finish cleanly?) ships with the SDK;\nthe other two are small classes you can read through.\n\n`PytestResults` leans on **`LocalFilesystemEvidence.run_verifier`**, which copies the workspace to a\nthrowaway directory and runs a command there (so scoring never mutates the trial's evidence). Its\n`overlay_files` argument writes trusted files *over* that copy first — that's how the held-out tests\nget in without ever being in the agent's workspace."
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cell-10",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class WorkspaceFileContains:\n",
+ " \"\"\"True when `path` exists in the agent's workspace and includes the given substring (case-insensitive).\"\"\"\n",
+ "\n",
+ " def __init__(self, *, name: str, path: str, contains: str) -> None:\n",
+ " self._name = name\n",
+ " self._path = path\n",
+ " self._contains = contains\n",
+ "\n",
+ " @property\n",
+ " def type(self) -> str:\n",
+ " return self._name\n",
+ "\n",
+ " def output_spec(self) -> list[MetricOutputSpec]:\n",
+ " return [MetricOutputSpec.boolean(\"present\")]\n",
+ "\n",
+ " async def compute_scores(self, input: MetricInput) -> MetricResult:\n",
+ " present = False\n",
+ " evidence = input.candidate.evidence\n",
+ " if evidence is not None and evidence.get(\"workspace\") is not None:\n",
+ " ws = await evidence.filesystem(\"workspace\")\n",
+ " if await ws.exists(self._path):\n",
+ " present = self._contains.lower() in (await ws.read_text(self._path)).lower()\n",
+ " return MetricResult(outputs=[MetricOutput(name=\"present\", value=present)])\n",
+ "\n",
+ "\n",
+ "class WorkspaceFileUnchanged:\n",
+ " \"\"\"True when a workspace file byte-matches the held-out reference — i.e. the agent left it alone.\n",
+ "\n",
+ " For a \"write tests\" task the deliverable *is* the tests, so they can't be held out. The reward\n",
+ " hack instead is editing the code under test so weak tests pass. We keep the authoritative source in\n",
+ " ``reference`` and checksum the agent's copy against it, turning \"don't touch the module\" into an\n",
+ " explicit score rather than trusting the agent.\n",
+ " \"\"\"\n",
+ "\n",
+ " def __init__(self, *, name: str, path: str, reference_key: str) -> None:\n",
+ " self._name = name\n",
+ " self._path = path\n",
+ " self._reference_key = reference_key\n",
+ "\n",
+ " @property\n",
+ " def type(self) -> str:\n",
+ " return self._name\n",
+ "\n",
+ " def output_spec(self) -> list[MetricOutputSpec]:\n",
+ " return [MetricOutputSpec.boolean(\"unchanged\")]\n",
+ "\n",
+ " async def compute_scores(self, input: MetricInput) -> MetricResult:\n",
+ " unchanged = False\n",
+ " expected = (input.row.data.get(\"reference\") or {}).get(self._reference_key, {}).get(self._path)\n",
+ " evidence = input.candidate.evidence\n",
+ " if expected is not None and evidence is not None and evidence.get(\"workspace\") is not None:\n",
+ " ws = await evidence.filesystem(\"workspace\")\n",
+ " if await ws.exists(self._path):\n",
+ " actual = await ws.read_text(self._path)\n",
+ " unchanged = _sha256(actual) == _sha256(expected)\n",
+ " return MetricResult(outputs=[MetricOutput(name=\"unchanged\", value=unchanged)])\n",
+ "\n",
+ "\n",
+ "def _sha256(text: str) -> str:\n",
+ " return hashlib.sha256(text.encode(\"utf-8\")).hexdigest()\n",
+ "\n",
+ "\n",
+ "def _parse_pytest_counts(text: str) -> tuple[int, int]:\n",
+ " \"\"\"Pull (passed, failed+errored) counts out of pytest's summary line.\"\"\"\n",
+ "\n",
+ " def n(word: str) -> int:\n",
+ " match = re.search(rf\"(\\d+) {word}\", text)\n",
+ " return int(match.group(1)) if match else 0\n",
+ "\n",
+ " return n(\"passed\"), n(\"failed\") + n(\"error\")\n",
+ "\n",
+ "\n",
+ "class PytestResults:\n",
+ " \"\"\"Run pytest against the agent's solution and report several outputs, not just a boolean.\n",
+ "\n",
+ " If ``overlay_reference_key`` is set, the files under ``reference[key]`` are staged *over* a\n",
+ " throwaway copy of the workspace before pytest runs — held-out tests the agent never had and could\n",
+ " not edit. Otherwise the agent's own workspace (its files and any tests it wrote) is run as-is.\n",
+ " \"\"\"\n",
+ "\n",
+ " def __init__(self, *, overlay_reference_key: str | None = None) -> None:\n",
+ " self._overlay_reference_key = overlay_reference_key\n",
+ "\n",
+ " @property\n",
+ " def type(self) -> str:\n",
+ " return \"pytest_results\"\n",
+ "\n",
+ " def output_spec(self) -> list[MetricOutputSpec]:\n",
+ " return [\n",
+ " MetricOutputSpec.boolean(\"all_passed\"),\n",
+ " MetricOutputSpec.continuous_score(\"pass_rate\"),\n",
+ " MetricOutputSpec.discrete_score(\"num_run\"),\n",
+ " MetricOutputSpec.discrete_score(\"num_failed\"),\n",
+ " ]\n",
+ "\n",
+ " async def compute_scores(self, input: MetricInput) -> MetricResult:\n",
+ " passed, failed, ok = 0, 0, False\n",
+ " overlay = {}\n",
+ " if self._overlay_reference_key:\n",
+ " overlay = (input.row.data.get(\"reference\") or {}).get(self._overlay_reference_key, {})\n",
+ " evidence = input.candidate.evidence\n",
+ " if evidence is not None and evidence.get(\"workspace\") is not None:\n",
+ " ws = await evidence.filesystem(\"workspace\")\n",
+ " result = await ws.run_verifier(\n",
+ " [sys.executable, \"-m\", \"pytest\", \"-q\"],\n",
+ " overlay_files=overlay or None,\n",
+ " timeout_s=120,\n",
+ " )\n",
+ " passed, failed = _parse_pytest_counts(result.stdout + result.stderr)\n",
+ " ok = result.ok\n",
+ " num_run = passed + failed\n",
+ " return MetricResult(\n",
+ " outputs=[\n",
+ " MetricOutput(name=\"all_passed\", value=ok and num_run > 0),\n",
+ " MetricOutput(name=\"pass_rate\", value=(passed / num_run) if num_run else 0.0),\n",
+ " MetricOutput(name=\"num_run\", value=num_run),\n",
+ " MetricOutput(name=\"num_failed\", value=failed),\n",
+ " ]\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-11",
+ "metadata": {},
+ "source": "\n## 4. Define the task suite\n\nThree coding tasks. Each seeds starter files into the agent's workspace via `inputs[\"files\"]` — a\n`{path: source}` map the runner stages before the agent starts (a `source` is `inline` contents here;\nit can also be a local `path` or a stored `fileset` reference, picked by a `kind` field — a bare\nstring is shorthand for inline text). What the agent is graded against lives in **`reference`**, which\nis *never* seeded into the workspace. Each task attaches **its own** metrics and a `correctness`\n**view** that maps those task-specific metrics onto one shared pass/fail score.\n\nNote how each `ViewSignal` refers to a metric by its `.type` rather than a hand-typed string — the\nreference stays correct if a metric's type name ever changes."
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cell-12",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Metric instances. fix-bug overlays held-out tests; write-tests runs the agent's own tests but\n",
+ "# checksums the module it must not modify. Referencing `metric.type` in views keeps the wiring honest.\n",
+ "phase_success = AgentPhaseSuccessMetric()\n",
+ "pytest_with_held_out_tests = PytestResults(overlay_reference_key=\"tests\")\n",
+ "pytest_agent_tests = PytestResults()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-13",
+ "metadata": {},
+ "source": "\n### 4a. Fix a bug\n\nWe seed a buggy `calculator.py` — **and nothing else**. The test suite is *held out* in `reference`\nand overlaid into a throwaway copy at scoring time, so the agent physically cannot edit the tests that\ngrade it; it can only fix the code.\nScored by: `pytest_results` (do the held-out tests pass, and at what rate) and `AgentPhaseSuccessMetric`\n(did the agent exit cleanly). `correctness` = *agent finished* **and** *all held-out tests pass*."
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cell-14",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "BUGGY_CALCULATOR = \"\"\"def add(a, b):\n",
+ " # BUG: subtracts instead of adding\n",
+ " return a - b\n",
+ "\"\"\"\n",
+ "\n",
+ "CALCULATOR_TESTS = \"\"\"from calculator import add\n",
+ "\n",
+ "\n",
+ "def test_add():\n",
+ " assert add(2, 3) == 5\n",
+ " assert add(-1, 1) == 0\n",
+ "\"\"\"\n",
+ "\n",
+ "fix_bug = AgentEvalTask(\n",
+ " id=\"fix-bug\",\n",
+ " intent=\"Fix the bug in calculator.py so that add() returns the sum and the hidden tests pass.\",\n",
+ " inputs={\n",
+ " \"instruction\": (\n",
+ " \"calculator.py has a bug: add(a, b) subtracts instead of adding. Fix calculator.py so that \"\n",
+ " \"add() returns the sum. You are graded by a hidden test suite you cannot see or edit.\"\n",
+ " ),\n",
+ " # Only the buggy source is seeded. The test file is NOT here — it is held out in `reference`.\n",
+ " \"files\": {\"calculator.py\": inline(BUGGY_CALCULATOR)},\n",
+ " },\n",
+ " # Grader-only ground truth: overlaid by PytestResults at scoring time, never seen by the agent.\n",
+ " reference={\"tests\": {\"test_calculator.py\": CALCULATOR_TESTS}},\n",
+ " metrics=[phase_success, pytest_with_held_out_tests],\n",
+ " views={\n",
+ " \"correctness\": SemanticView(\n",
+ " reducer=SemanticReducer.ALL,\n",
+ " signals=[\n",
+ " ViewSignal(metric=phase_success.type, output=\"agent_phase_success\"),\n",
+ " ViewSignal(metric=pytest_with_held_out_tests.type, output=\"all_passed\"),\n",
+ " ],\n",
+ " )\n",
+ " },\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-15",
+ "metadata": {},
+ "source": "\n### 4b. Write tests\n\nWe seed a correct `stringutils.py` so the agent can read it. **The agent must write a passing pytest\nfile for it.** Here the deliverable *is* the tests, so they can't be held out — but the reward hack is\nediting the module so weak tests pass. So we keep the authoritative `stringutils.py` in `reference`\nand **checksum** the agent's copy against it: `impl_unchanged` is a first-class signal, and the\n`correctness` view fails if the agent touched the module. Scored by: `wrote_test_file` (did a\n`test_stringutils.py` referencing `slugify` appear), `impl_unchanged` (module left alone), and\n`pytest_results` (the agent's tests pass). A different metric set from the bug task."
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cell-16",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "STRINGUTILS = \"\"\"import re\n",
+ "\n",
+ "\n",
+ "def slugify(text: str) -> str:\n",
+ " \\\"\\\"\\\"Lowercase text and turn runs of non-alphanumerics into single hyphens.\\\"\\\"\\\"\n",
+ " return re.sub(r\"[^a-z0-9]+\", \"-\", text.lower()).strip(\"-\")\n",
+ "\"\"\"\n",
+ "\n",
+ "wrote_test_file = WorkspaceFileContains(name=\"wrote_test_file\", path=\"test_stringutils.py\", contains=\"slugify\")\n",
+ "impl_unchanged = WorkspaceFileUnchanged(name=\"impl_unchanged\", path=\"stringutils.py\", reference_key=\"protected\")\n",
+ "\n",
+ "write_tests = AgentEvalTask(\n",
+ " id=\"write-tests\",\n",
+ " intent=\"Write pytest tests for the slugify() function in stringutils.py.\",\n",
+ " inputs={\n",
+ " \"instruction\": (\n",
+ " \"Write a pytest file named test_stringutils.py that imports slugify from stringutils and \"\n",
+ " \"covers a few cases. The tests must pass. Do not modify stringutils.py.\"\n",
+ " ),\n",
+ " \"files\": {\"stringutils.py\": inline(STRINGUTILS)},\n",
+ " },\n",
+ " # The authoritative module the agent must not touch; impl_unchanged checksums against it.\n",
+ " reference={\"protected\": {\"stringutils.py\": STRINGUTILS}},\n",
+ " metrics=[wrote_test_file, impl_unchanged, pytest_agent_tests],\n",
+ " views={\n",
+ " \"correctness\": SemanticView(\n",
+ " reducer=SemanticReducer.ALL,\n",
+ " signals=[\n",
+ " ViewSignal(metric=wrote_test_file.type, output=\"present\"),\n",
+ " ViewSignal(metric=impl_unchanged.type, output=\"unchanged\"),\n",
+ " ViewSignal(metric=pytest_agent_tests.type, output=\"all_passed\"),\n",
+ " ],\n",
+ " )\n",
+ " },\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cell-17",
+ "metadata": {},
+ "source": "\n### 4c. Write docs\n\nWe seed an undocumented `widget.py`. **The agent must add a docstring and a README.**\n\nPresence is easy to check deterministically — but *is the documentation any good?* That's subjective,\nso we add an **LLM-as-judge** metric alongside the deterministic ones. Deterministic checks feed\n`correctness` (did the files appear); the judge scores `doc_quality` against a **rubric** (it picks a\nlabel like `excellent`/`good`/…, which maps to a 1–5 value) for how clear and complete they are."
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cell-18",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from nemo_evaluator_sdk.metrics.llm_judge import LLMJudgeMetric\n",
+ "from nemo_evaluator_sdk.metrics.protocol import CandidateOutput, DatasetRow\n",
+ "from nemo_evaluator_sdk.values import InferenceParams, Model, SecretRef\n",
+ "from nemo_evaluator_sdk.values.scores import Rubric, RubricScore\n",
+ "\n",
+ "# The judge model's API key is injected via this env var — it never appears in the notebook.\n",
+ "JUDGE_API_KEY_ENV = \"NVIDIA_BUILD_API_KEY\"\n",
+ "\n",
+ "\n",
+ "class LlmDocReview:\n",
+ " \"\"\"LLM-as-judge over the docs the agent produced.\n",
+ "\n",
+ " Composition, not inheritance: this metric *holds* an LLMJudgeMetric (which owns the model call and\n",
+ " the structured-JSON parsing) and adds only what is specific here — pulling the README and module\n",
+ " source out of the workspace evidence and handing them to the judge to grade.\n",
+ " \"\"\"\n",
+ "\n",
+ " def __init__(self, judge: LLMJudgeMetric) -> None:\n",
+ " self._judge = judge\n",
+ "\n",
+ " @property\n",
+ " def type(self) -> str:\n",
+ " return \"doc_review\"\n",
+ "\n",
+ " def output_spec(self) -> list[MetricOutputSpec]:\n",
+ " return self._judge.output_spec()\n",
+ "\n",
+ " async def compute_scores(self, input: MetricInput) -> MetricResult:\n",
+ " readme, module_source = \"\", \"\"\n",
+ " evidence = input.candidate.evidence\n",
+ " if evidence is not None and evidence.get(\"workspace\") is not None:\n",
+ " ws = await evidence.filesystem(\"workspace\")\n",
+ " if await ws.exists(\"README.md\"):\n",
+ " readme = await ws.read_text(\"README.md\")\n",
+ " if await ws.exists(\"widget.py\"):\n",
+ " module_source = await ws.read_text(\"widget.py\")\n",
+ " # Hand the artifacts to the judge as its item fields; the judge owns the model call.\n",
+ " judge_input = MetricInput(\n",
+ " row=DatasetRow(row_index=0, data={\"readme\": readme, \"module_source\": module_source}),\n",
+ " candidate=CandidateOutput(output_text=readme),\n",
+ " )\n",
+ " return await self._judge.compute_scores(judge_input)\n",
+ "\n",
+ "\n",
+ "doc_judge = LlmDocReview(\n",
+ " LLMJudgeMetric(\n",
+ " model=Model(\n",
+ " url=\"https://integrate.api.nvidia.com/v1/chat/completions\",\n",
+ " name=os.environ.get(\"JUDGE_MODEL\", \"nvidia/nvidia-nemotron-nano-9b-v2\"),\n",
+ " format=\"openai\",\n",
+ " # api_key_secret names the env var to read; model.api_key resolves os.environ[JUDGE_API_KEY_ENV].\n",
+ " api_key_secret=SecretRef(JUDGE_API_KEY_ENV),\n",
+ " ),\n",
+ " scores=[\n",
+ " RubricScore(\n",
+ " name=\"doc_quality\",\n",
+ " description=\"Overall documentation quality: clarity, accuracy, and completeness.\",\n",
+ " # A rubric fits an LLM judge better than a bare 1-5 range: each level is an explicit,\n",
+ " # describable bar. The judge returns a label; the parser maps it to `value`.\n",
+ " rubric=[\n",
+ " Rubric(\n",
+ " label=\"excellent\",\n",
+ " value=5,\n",
+ " description=\"Clear, accurate, and complete, with a helpful usage example.\",\n",
+ " ),\n",
+ " Rubric(label=\"good\", value=4, description=\"Mostly clear and complete; minor gaps or rough edges.\"),\n",
+ " Rubric(label=\"adequate\", value=3, description=\"Understandable but missing detail or an example.\"),\n",
+ " Rubric(label=\"poor\", value=2, description=\"Vague, partly inaccurate, or hard to follow.\"),\n",
+ " Rubric(label=\"missing\", value=1, description=\"Essentially undocumented or incorrect.\"),\n",
+ " ],\n",
+ " )\n",
+ " ],\n",
+ " prompt_template={\n",
+ " \"messages\": [\n",
+ " {\n",
+ " \"role\": \"system\",\n",
+ " \"content\": \"You are a senior engineer reviewing documentation. Respond with JSON only.\",\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": (\n",
+ " \"A developer documented this module:\\n```python\\n{{item.module_source}}\\n```\\n\\n\"\n",
+ " \"and wrote this README:\\n```markdown\\n{{item.readme}}\\n```\\n\\n\"\n",
+ " \"Rate the documentation on clarity, accuracy, and completeness with one of \"\n",
+ " \"these labels — excellent, good, adequate, poor, missing — then \"\n",
+ " 'return JSON only: {\"doc_quality\": \"