feat(evaluator-sdk)!: install NeMo Fabric 0.1.0rc6 from published wheels and align the runtime

.github/actions/changes/action.yaml

@@ -13,6 +13,9 @@ outputs:
   deps:
     description: "'true' if any dependency-related files changed"
     value: ${{ steps.filter.outputs.deps }}
+  fabric:
+    description: "'true' if the Fabric agent-eval runtime, its tests, or its dependency extra changed"
+    value: ${{ steps.filter.outputs.fabric }}
   e2e:
     description: "'true' if any e2e test files changed"
     value: ${{ steps.filter.outputs.e2e }}
@@ -74,6 +77,12 @@ runs:
             - 'pyproject.toml'
             - 'uv.lock'
             - '.pre-commit-config.yaml'
+          fabric:
+            - 'packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/**'
+            - 'packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_*.py'
+            - 'packages/nemo_evaluator_sdk/pyproject.toml'
+            - '.github/workflows/ci.yaml'
+            - '.github/actions/changes/action.yaml'
           e2e:
             - 'e2e/**'
           docs:

.github/wheel-constraints/nemo-platform-services.txt

@@ -42,8 +42,8 @@ langchain-openai==1.4.0
 langchain==1.3.14
 lark==1.3.1
 nemo-anonymizer==0.3.1
-nemo-fabric==0.1.0a20260723
-nemo-relay==0.4.0
+nemo-fabric==0.1.0rc6
+nemo-relay==0.6.0
 nemo-safe-synthesizer==0.1.7
 nemoguardrails==0.23.0
 ngcsdk==4.21.0

.github/workflows/ci.yaml

@@ -37,6 +37,7 @@ jobs:
       openapi: ${{ steps.changes.outputs.openapi }}
       test: ${{ steps.changes.outputs.test }}
       deps: ${{ steps.changes.outputs.deps }}
+      fabric: ${{ steps.changes.outputs.fabric }}
       e2e: ${{ steps.changes.outputs.e2e }}
       docs: ${{ steps.changes.outputs.docs }}
       web-studio: ${{ steps.changes.outputs.web-studio }}
@@ -857,6 +858,96 @@ jobs:
             coverage.json
             coverage-html/
 
+  # WHY THIS JOB EXISTS (it looks redundant, it isn't): `--extra fabric` appears nowhere else in this
+  # file and there is no `--all-extras`, so no other job installs nemo-fabric. Every other Fabric test
+  # either fakes `nemo_fabric` or `importorskip`s it, which means the hermetic suite keeps passing when
+  # the real API moves — that is how the enable_relay keyword, the `.cli`/`.sdk` adapter ids, and the
+  # entire profile mechanism each drifted underneath us while the unit tests stayed green. Deleting
+  # this job takes real-Fabric coverage to zero. The bare import is also the ONLY check on runtime.py's
+  # TYPE_CHECKING import block (RunResult/RunOutput/...), which ty cannot verify because nemo-fabric is
+  # absent from the type-check environment.
+  fabric-wheel-smoke:
+    name: Fabric wheel install smoke (Linux, py${{ matrix.python-version }})
+    needs: [changes]
+    # A Fabric wheel bump lands in uv.lock (deps), and edits to the runtime's own Fabric call sites or
+    # its tests land under the fabric filter — either can surface API drift, so run on both.
+    if: >
+      !cancelled() &&
+      (needs.changes.outputs.deps == 'true' || needs.changes.outputs.fabric == 'true')
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        # One interpreter on purpose. nemo-fabric-runtime ships a single cp311-abi3 wheel and the
+        # adapters are pure Python, so extra legs mostly re-check the same dependency closure; the
+        # 3.12-3.13 range is already asserted by uv.lock resolving universally.
+        #
+        # It must satisfy uv.lock's `requires-python` (currently >=3.12,<3.14) because this job runs
+        # `uv sync` on the WORKSPACE. That ceiling is the intersection across workspace members —
+        # nemo-rl, automodel, unsloth, deployments and experimentalist still cap at <3.14 — so it is
+        # narrower than the root pyproject's <3.15, and a 3.14 leg would fail at `uv sync` before
+        # installing anything.
+        python-version: ["3.12"]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          # This job installs and imports third-party wheels; don't leave GITHUB_TOKEN in .git/config.
+          persist-credentials: false
+      - name: Install uv
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
+        with:
+          # Must satisfy the workspace floor in the root pyproject.toml (`requires-python`), or
+          # `uv sync` refuses the interpreter outright.
+          python-version: ${{ matrix.python-version }}
+          enable-cache: true
+          cache-dependency-glob: uv.lock
+      # Installs the published nemo-fabric wheels (+ codex/claude/deepagents adapters) from the lock.
+      # Linux is where jobs actually execute the runtime, and it validates that the manylinux
+      # nemo-fabric-runtime wheel installs on the runner's glibc.
+      - name: Install nemo-evaluator-sdk[fabric] from the lock
+        run: uv sync --frozen --package nemo-evaluator-sdk --extra fabric
+      # Import the Fabric SDK surface that agent_eval/runtimes/fabric/runtime.py depends on, so the
+      # published package is exercised for real instead of only via the hermetic fake-nemo_fabric tests.
+      - name: Import the Fabric SDK surface the runtime uses
+        run: |
+          uv run --frozen --no-sync python - <<'PY'
+          import nemo_fabric
+          from nemo_fabric import (
+              EnvironmentConfig,
+              Fabric,
+              FabricConfig,
+              RelayAtifConfig,
+              RelayAtofConfig,
+              RelayAtofFileSinkConfig,
+              RelayObservabilityConfig,
+              RunRequest,
+              RunResult,
+          )
+          print("nemo_fabric import OK:", nemo_fabric.__file__)
+          PY
+      # A bare import can't catch API drift in the runtime's *call sites* — the enable_relay keyword and
+      # the harness adapter ids both moved under us while the hermetic fakes kept passing. These contract
+      # tests exercise those call sites against the real wheels (importorskip elsewhere), so a future
+      # drift turns this job red instead of slipping through.
+      - name: Run Fabric runtime contract tests against the real wheels
+        # pytest lives in the root `dev` group, which the minimal `--package ... --extra fabric` sync
+        # above does not pull, so install it (and pytest-asyncio, for the SDK package's
+        # asyncio_mode=auto) into the synced venv rather than bloating the smoke env with the whole
+        # dev group.
+        #
+        # It must go INTO .venv — NOT via `uv run --with`, which layers an ephemeral overlay whose
+        # sys.prefix is a temp build dir. Fabric discovers adapter descriptors under
+        # `<sys.prefix>/share/nemo-fabric/adapters` (that is where the adapter wheels install their
+        # fabric-adapter.json data files), so under an overlay every adapter goes missing and any
+        # resolution fails with `unknown adapter ...; available adapters: []`.
+        run: |
+          uv pip install --python .venv/bin/python pytest pytest-asyncio
+          uv run --frozen --no-sync pytest \
+            packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_surface.py -v
+
   python-integration-test:
     name: Python integration tests
     needs: [policy-wasm]
@@ -1820,6 +1911,7 @@ jobs:
       - policy-wasm
       - python-unit-test-tools
       - python-unit-test
+      - fabric-wheel-smoke
       # Enable if you want this required
       # - python-integration-test
       - require-nvskills

packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb

@@ -22,7 +22,7 @@
    "cell_type": "markdown",
    "id": "cell-03",
    "metadata": {},
-   "source": "<a id=\"prereqs\"></a>\n## 1. Install & prerequisites\n\nThis tutorial evaluates **Codex driven by NeMo Fabric**. Fabric is a native (Rust) component, so —\nunlike a pure-Python SDK — this notebook does **not** run from a plain `pip install`. You need a real\ntoolchain:\n\n- **NeMo Evaluator SDK** — `pip install \"nemo-platform-sdk[nemo-evaluator-sdk]\"` and `pip install pytest` (some metrics run the agent's tests).\n- **NeMo Fabric** (native) with the coding-agent + trajectory extras — `nemo-fabric[codex,relay]` —\n  plus a **checkout of the NeMo-Fabric repo**: the Codex adapter registry (`adapters/codex-cli`) is\n  resolved relative to it. The repo's `script/dev-install-fabric.sh` installs `nemo-fabric[codex,relay]`\n  and builds the `nemo-relay` gateway for you.\n- **Codex CLI** on your `PATH` (`npm install -g @openai/codex`) + auth (`codex login`, or set `OPENAI_API_KEY`).\n- **`nemo-relay` gateway** binary on your `PATH` (built by the script above) — required for ATIF\n  trajectory capture.\n\nSet two environment variables before launching (both read from the environment, never written here):\n\n- **`NEMO_FABRIC_REPO`** — path to your NeMo-Fabric checkout (Fabric resolves the adapter registry\n  from it).\n- **`NVIDIA_BUILD_API_KEY`** — a build.nvidia.com key for the `write-docs` LLM judge. The cell below\n  prompts for it if it isn't already set.\n\nEverything else — defining tasks and metrics, reading results — is plain Python you can read through."
+   "source": "<a id=\"prereqs\"></a>\n## 1. Install & prerequisites\n\nThis tutorial evaluates **Codex driven by NeMo Fabric**. Fabric is a native (Rust) component, so —\nunlike a pure-Python SDK — this notebook does **not** run from a plain `pip install`. You need a real\ntoolchain:\n\n- **NeMo Evaluator SDK** — `pip install \"nemo-platform-sdk[nemo-evaluator-sdk]\"` and `pip install pytest` (some metrics run the agent's tests).\n- **NeMo Fabric harness adapters** — `pip install \"nemo-evaluator-sdk[fabric]\"`. Fabric's SDK is\n  already a dependency of the evaluator SDK; this extra adds the Codex/Claude/deepagents adapters.\n  No NeMo-Fabric checkout is needed: adapter descriptors install with the wheels and Fabric\n  discovers them from the environment.\n- **Codex CLI** on your `PATH` (`npm install -g @openai/codex`) + auth (`codex login`, or set `OPENAI_API_KEY`).\n- **`nemo-relay` gateway** binary on your `PATH` — required for ATIF trajectory capture. It is the one\n  piece not published to PyPI; the repo's `script/dev-install-fabric.sh` downloads it for you.\n\nSet one environment variable before launching (read from the environment, never written here):\n\n- **`NVIDIA_BUILD_API_KEY`** — a build.nvidia.com key for the `write-docs` LLM judge. The cell below\n  prompts for it if it isn't already set.\n\nOptionally set **`CODEX_MODEL`** to pick the model; the Codex adapter requires one and has no default.\n\nEverything else — defining tasks and metrics, reading results — is plain Python you can read through."
   },
   {
    "cell_type": "code",
@@ -37,16 +37,7 @@
     "# Prompt for the judge's API key unless it's already in the environment. getpass masks the input, so\n",
     "# the key never lands in the notebook or its saved output.\n",
     "if not os.environ.get(\"NVIDIA_BUILD_API_KEY\"):\n",
-    "    os.environ[\"NVIDIA_BUILD_API_KEY\"] = getpass.getpass(\"build.nvidia.com API key (NVIDIA_BUILD_API_KEY): \")\n",
-    "\n",
-    "# Fabric resolves the Codex adapter registry relative to a NeMo-Fabric checkout. It's a path, not a\n",
-    "# secret, so prompt with plain input() (not getpass) and expand ~ if the environment doesn't set it.\n",
-    "if not os.environ.get(\"NEMO_FABRIC_REPO\"):\n",
-    "    os.environ[\"NEMO_FABRIC_REPO\"] = os.path.expanduser(\n",
-    "        input(\"Path to your NeMo-Fabric checkout (NEMO_FABRIC_REPO): \").strip()\n",
-    "    )\n",
-    "if not os.environ[\"NEMO_FABRIC_REPO\"]:\n",
-    "    raise RuntimeError(\"NEMO_FABRIC_REPO is required — set it to your NeMo-Fabric checkout.\")"
+    "    os.environ[\"NVIDIA_BUILD_API_KEY\"] = getpass.getpass(\"build.nvidia.com API key (NVIDIA_BUILD_API_KEY): \")"
    ]
   },
   {
@@ -543,15 +534,47 @@
    "cell_type": "markdown",
    "id": "cell-23",
    "metadata": {},
-   "source": "<a id=\"target\"></a>\n## 5. Pick the target: Codex via NeMo Fabric\n\nThe **target** is what runs each task. We use **`FabricAgentRuntime`**: NeMo Fabric drives a harness\n(here the Codex CLI, selected by `harness.adapter_id`) in a fresh per-task workspace, and captures\nboth the final workspace and the agent's execution **trajectory**. Each task's `inputs[\"files\"]` are\nseeded into its workspace, the harness runs there, and the final file tree is exposed as `workspace`\nevidence (what the metrics above open). `capture_trajectory=True` additionally records the agent's\nstep-by-step actions as an ATIF `trace` (via the `nemo-relay` gateway).\n\nThe agent config is built from Fabric's own typed config objects (`FabricConfig`, `HarnessConfig`, …)\nrather than a raw dict, so the harness/runtime/environment fields are checked as you write them.\n`base_dir` points at your NeMo-Fabric checkout so Fabric can resolve the Codex adapter registry."
+   "source": "<a id=\"target\"></a>\n## 5. Pick the target: Codex via NeMo Fabric\n\nThe **target** is what runs each task. We use **`FabricAgentRuntime`**: NeMo Fabric drives a harness\n(here the Codex CLI, selected by `harness.adapter_id`) in a fresh per-task workspace, and captures\nboth the final workspace and the agent's execution **trajectory**. Each task's `inputs[\"files\"]` are\nseeded into its workspace, the harness runs there, and the final file tree is exposed as `workspace`\nevidence (what the metrics above open). `capture_trajectory=True` additionally records the agent's\nstep-by-step actions as an ATIF `trace` (via the `nemo-relay` gateway).\n\nThe agent config is built from Fabric's own typed config objects (`FabricConfig`, `HarnessConfig`, …)\nrather than a raw dict, so the harness/runtime/environment fields are checked as you write them. Fabric\nresolves the Codex adapter from the installed adapter wheels, so no checkout path has to be supplied."
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "id": "cell-24",
    "metadata": {},
    "outputs": [],
-   "source": "from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime\nfrom nemo_fabric import (  # ty: ignore[unresolved-import]\n    EnvironmentConfig,\n    FabricConfig,\n    HarnessConfig,\n    MetadataConfig,\n    RuntimeConfig,\n)\n\n# A typed Fabric agent config: Codex CLI harness, one-shot text in / message out, local execution.\ncodex_via_fabric = FabricConfig(\n    metadata=MetadataConfig(name=\"coding-agent-eval\"),\n    harness=HarnessConfig(\n        adapter_id=\"nvidia.fabric.codex.cli\",  # Fabric drives the Codex CLI under the hood\n        resolution=\"preinstalled\",\n        settings={\"sandbox\": \"workspace-write\", \"skip_git_repo_check\": True, \"timeout_seconds\": 180},\n    ),\n    runtime=RuntimeConfig(mode=\"oneshot\", transport=\"cli\", input_schema=\"text\", output_schema=\"message\"),\n    environment=EnvironmentConfig(provider=\"local\"),  # the per-task workspace is set by the runtime\n)\n\ntarget = FabricAgentRuntime(\n    config=codex_via_fabric,\n    model=os.environ.get(\"CODEX_MODEL\"),  # None → the adapter's default model\n    base_dir=Path(os.environ[\"NEMO_FABRIC_REPO\"]),  # resolves adapters/codex-cli\n    work_root=OUTPUT_DIR / \"fabric\",\n    capture_trajectory=True,  # capture the agent's ATIF trajectory as trace evidence\n)\nprint(\"Target:\", type(target).__name__)"
+   "source": [
+    "from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime\n",
+    "from nemo_fabric import (  # ty: ignore[unresolved-import]\n",
+    "    EnvironmentConfig,\n",
+    "    FabricConfig,\n",
+    "    HarnessConfig,\n",
+    "    MetadataConfig,\n",
+    "    RuntimeConfig,\n",
+    ")\n",
+    "\n",
+    "# A typed Fabric agent config: Codex CLI harness, one-shot text in / message out, local execution.\n",
+    "codex_via_fabric = FabricConfig(\n",
+    "    metadata=MetadataConfig(name=\"coding-agent-eval\"),\n",
+    "    harness=HarnessConfig(\n",
+    "        adapter_id=\"nvidia.fabric.codex\",  # Fabric drives the Codex CLI under the hood\n",
+    "        resolution=\"preinstalled\",\n",
+    "        settings={\"sandbox\": \"workspace-write\", \"skip_git_repo_check\": True, \"timeout_seconds\": 180},\n",
+    "    ),\n",
+    "    runtime=RuntimeConfig.from_mapping(\n",
+    "        # mode/transport are not declared RuntimeConfig fields; Fabric keeps them in its extras bag.\n",
+    "        {\"mode\": \"oneshot\", \"transport\": \"cli\", \"input_schema\": \"text\", \"output_schema\": \"message\"}\n",
+    "    ),\n",
+    "    environment=EnvironmentConfig(provider=\"local\"),  # the per-task workspace is set by the runtime\n",
+    ")\n",
+    "\n",
+    "target = FabricAgentRuntime(\n",
+    "    config=codex_via_fabric,\n",
+    "    model=os.environ.get(\"CODEX_MODEL\", \"openai/gpt-5.4\"),  # required: the codex adapter has no default model\n",
+    "    work_root=OUTPUT_DIR / \"fabric\",\n",
+    "    capture_trajectory=True,  # capture the agent's ATIF trajectory as trace evidence\n",
+    ")\n",
+    "print(\"Target:\", type(target).__name__)"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -668,7 +691,7 @@
    "cell_type": "markdown",
    "id": "cell-35",
    "metadata": {},
-   "source": "<a id=\"next\"></a>\n## 8. Where to go next\n\n- **Swap the harness.** Fabric selects the agent by `harness.adapter_id`. Point it at another adapter\n  (e.g. `nvidia.fabric.hermes.cli`) to evaluate a different coding agent with the same tasks and\n  metrics.\n- **Evaluate your own agent.** Implement the `AgentTaskRunner` protocol — a class with one async\n  `run_tasks(tasks, config)` that runs each task and returns a trial (output + workspace evidence).\n  Everything else in this notebook stays the same.\n- **Re-score without re-running the agent.** Pass precomputed `trials=` instead of `target=` to\n  `run_sync` to apply new metrics to trials you already have.\n- **Grade against the trajectory.** A metric can open the `trace` evidence (ATIF) to score *how* the\n  agent worked — tool calls, retries, steps — not just its final files.\n- **Add signals to a view, or weight them.** `SemanticReducer` also offers `ANY`, `MEAN`, and\n  `WEIGHTED_MEAN` (with `ViewSignal(weight=...)`) — e.g. a partial-credit `correctness` from `pass_rate`.\n- **Keep ground truth held out.** Anything a metric grades on — tests, reference solutions, rubrics —\n  belongs in `reference`, overlaid or checksummed at scoring time, never in the agent's workspace.\n- **Decide pass/fail in your app.** The evaluator reports scores; thresholds and gating belong to your\n  CI/release process, not the evaluation itself."
+   "source": "<a id=\"next\"></a>\n## 8. Where to go next\n\n- **Swap the harness.** Fabric selects the agent by `harness.adapter_id`. Point it at another adapter\n  (e.g. `nvidia.fabric.hermes`) to evaluate a different coding agent with the same tasks and\n  metrics.\n- **Evaluate your own agent.** Implement the `AgentTaskRunner` protocol — a class with one async\n  `run_tasks(tasks, config)` that runs each task and returns a trial (output + workspace evidence).\n  Everything else in this notebook stays the same.\n- **Re-score without re-running the agent.** Pass precomputed `trials=` instead of `target=` to\n  `run_sync` to apply new metrics to trials you already have.\n- **Grade against the trajectory.** A metric can open the `trace` evidence (ATIF) to score *how* the\n  agent worked — tool calls, retries, steps — not just its final files.\n- **Add signals to a view, or weight them.** `SemanticReducer` also offers `ANY`, `MEAN`, and\n  `WEIGHTED_MEAN` (with `ViewSignal(weight=...)`) — e.g. a partial-credit `correctness` from `pass_rate`.\n- **Keep ground truth held out.** Anything a metric grades on — tests, reference solutions, rubrics —\n  belongs in `reference`, overlaid or checksummed at scoring time, never in the agent's workspace.\n- **Decide pass/fail in your app.** The evaluator reports scores; thresholds and gating belong to your\n  CI/release process, not the evaluation itself."
   }
  ],
  "metadata": {
@@ -683,4 +706,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}