feat(evaluator): add NeMo Fabric agent-eval runner with ATIF trajectory capture

[SandyChapman]

What

Adds a fabric agent-eval runner kind alongside codex. FabricAgentRuntime drives an agent harness through the NeMo Fabric SDK, maps each RunResult to an AgentEvalTrial, and captures the agent trajectory as ATIF (via NeMo Relay's file exporter) into trial evidence under the standard trace key so graders can consume it.

• FabricRunnerTarget(kind="fabric") + widened AgentRunnerTarget union + _resolve_target wiring
• Types against Fabric's real RunResult/FabricClient types (TYPE_CHECKING + lazy import; nemo_fabric is an optional native dep)
• ATIF trajectory capture → EVIDENCE_TRACE(format=atif); telemetry.uri/trace_id preserved
• Tests: unit + a hermetic CI integration test (runner → evaluator → metric → evidence, metric actually reads the trajectory) + a gated live fabric→codex e2e
• Regenerated: plugin openapi.yaml + vendored SDK mirror

Verification

• 119 passed / 1 gated-skip across the SDK agent-eval + plugin suites; ruff/ty clean.
• Live fabric→codex→Relay e2e ran green locally (real 330 KB ATIF captured + scored).

Notes / follow-ups

• Packaging (Tier 3): a fabric+trajectory eval needs three things in the jobs image — nemo-fabric, the codex CLI, and the nemo-relay gateway binary (not pip-installable). Adapter discovery also relies on a build-time path or config_root/adapters, so a deployed image must ship adapters/ (or we add a base_dir knob). Not resolved here.
• Follow-up PR: plugin run() + submit() live integration tests, coupled with the adapter-shipping/base_dir decision; plus an ATIF-vs-SDK-TraceHandle schema-compat check.
• Design doc intentionally excluded from this PR.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added a new Fabric-backed agent-evaluation runtime to execute tasks and persist trial outcomes with durable evidence.
  • Introduced Fabric runner targets with configurable profiles, optional model overlay, per-task timeouts, and optional ATIF trajectory capture.
  • Updated the API spec to accept Fabric runner target definitions.
• Tests
  • Added end-to-end and unit/integration coverage validating trajectory evidence, success/failure mapping, timeouts, and optional dependency behavior.
• Chores
  • Added a developer script to install/uninstall local Fabric tooling for live runs.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a Fabric-backed agent runtime that runs agent-eval tasks through nemo_fabric, persists results and errors as evidence, captures optional Relay trajectory traces, wires a new Fabric runner target through job/spec/OpenAPI, and adds unit, integration, and developer-install coverage.

Changes

Fabric Agent Runtime

Layer / File(s)	Summary
FabricAgentRuntime core packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py	Adds the runtime module, constructor state, task orchestration, per-task execution, trajectory profile setup, evidence directory selection, and input/output/error helpers.
Trial and evidence mapping packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py	Persists Fabric results, builds evidence descriptors, maps artifacts and telemetry, and converts failed results or exceptions into failed trials.
Runtime unit tests packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py	Covers success, ATIF evidence, trajectory toggling, failure, timeout, exception, and missing dependency behavior with an in-memory Fabric stub.
Evaluator integration tests packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py	Adds hermetic and live evaluator tests that validate trajectory evidence capture and metric consumption end to end.
Job/spec/OpenAPI wiring plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py, plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py, plugins/nemo-evaluator/openapi/openapi.yaml, plugins/nemo-evaluator/tests/test_agent_evaluate.py	Adds FabricRunnerTarget support in the DTOs, schema, job target resolution, and tests.
Dev install script script/dev-install-fabric.sh	Adds a helper script for installing or uninstalling nemo-fabric and nemo-relay locally.

Possible sequence

sequenceDiagram
  participant Job as AgentEvalJob
  participant Runtime as FabricAgentRuntime
  participant Client as FabricClient
  participant Relay as nemo_relay.observability
  participant FS as evidence directory

  Job->>Runtime: run_tasks(tasks, config)
  Runtime->>Runtime: build profiles
  Runtime->>Relay: create trajectory profile
  loop each task
    Runtime->>FS: create task evidence dir
    Runtime->>Client: run(input, request_id)
    Client-->>Runtime: RunResult / timeout / exception
    Runtime->>FS: write fabric_result.json or error.json
  end
  Runtime-->>Job: AgentEvalTrial list

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#339: Adds the evidence descriptor types this runtime uses for ATIF trace evidence.
• NVIDIA-NeMo/nemo-platform#496: Adds the agent-eval target plumbing extended here with FabricRunnerTarget.

Suggested reviewers: arpitsardhana

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding a NeMo Fabric agent-eval runner with ATIF trajectory capture.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fabric-eval-runner/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

plugins/nemo-evaluator/tests/test_agent_evaluate.py (1)

197-210: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Test reaches into private attrs (_model, _work_root).

Couples the test to implementation internals rather than the public contract; consistent with no bigger issue but slightly fragile.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-evaluator/tests/test_agent_evaluate.py` around lines 197 - 210,
The test is asserting private internals of FabricAgentRuntime, making it
fragile. Update test_resolve_target_builds_fabric_runtime_from_runner_target to
validate the public behavior of AgentEvalJob._resolve_target and
FabricAgentRuntime instead of _model and _work_root; use the runtime’s exposed
API or observable outcomes to confirm the model and work root are set correctly,
while keeping the existing assertions about prompt_template and params.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@plugins/nemo-evaluator/tests/test_agent_evaluate.py`:
- Around line 197-210: The test is asserting private internals of
FabricAgentRuntime, making it fragile. Update
test_resolve_target_builds_fabric_runtime_from_runner_target to validate the
public behavior of AgentEvalJob._resolve_target and FabricAgentRuntime instead
of _model and _work_root; use the runtime’s exposed API or observable outcomes
to confirm the model and work root are set correctly, while keeping the existing
assertions about prompt_template and params.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 822cb06d-2006-4c85-ace9-cbd87253b142

📥 Commits

Reviewing files that changed from the base of the PR and between 2cc1011 and 0102d00.

⛔ Files ignored due to path filters (1)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py is excluded by !sdk/**

📒 Files selected for processing (8)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
• plugins/nemo-evaluator/tests/test_agent_evaluate.py
• script/dev-install-fabric.sh

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	23089/30221	76.4%	61.3%
Integration Tests	13257/28901	45.9%	19.1%

[arpitsardhana]

It would be nice to add Fabric Profile example for different harness to get sense of different runtimes structure

[coderabbitai]

🧹 Nitpick comments (1)

packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py (1)

173-173: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Repeated FabricAgentRuntime(config=_CONFIG, work_root=tmp_path / "fabric") construction.

Same call repeated across 8 tests. A runtime fixture would cut boilerplate.

Also applies to: 214-214, 235-235, 254-254, 279-279, 295-295, 308-308, 323-323

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py` at line
173, The same FabricAgentRuntime construction is repeated across multiple tests,
so factor it into a shared pytest fixture in test_fabric_runtime.py. Create a
fixture that builds FabricAgentRuntime with _CONFIG, model, and the
tmp_path-derived work_root, then update the affected test functions to accept
and use that runtime fixture instead of duplicating the setup.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py`:
- Line 173: The same FabricAgentRuntime construction is repeated across multiple
tests, so factor it into a shared pytest fixture in test_fabric_runtime.py.
Create a fixture that builds FabricAgentRuntime with _CONFIG, model, and the
tmp_path-derived work_root, then update the affected test functions to accept
and use that runtime fixture instead of duplicating the setup.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 46239b4b-58ea-4ad5-9a35-2cec62dc37e9

📥 Commits

Reviewing files that changed from the base of the PR and between 0102d00 and 229a2fe.

⛔ Files ignored due to path filters (1)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py is excluded by !sdk/**

📒 Files selected for processing (8)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
• plugins/nemo-evaluator/tests/test_agent_evaluate.py
• script/dev-install-fabric.sh

🚧 Files skipped from review as they are similar to previous changes (5)

• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
• script/dev-install-fabric.sh
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-evaluator/tests/test_agent_evaluate.py

[SandyChapman]

@coderabbitai The _model/_work_root assertions are intentional — they verify the resolver's wiring (model passthrough + work_root derivation), and there's no public accessor to assert against without a live Fabric dependency. Leaving as-is.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@SandyChapman Fair enough — makes sense given the optional dependency constraint. Withdrawing.