chore(evaluator): detangle plugin from legacy service

[SandyChapman]

Summary

• move plugin-only evaluator pieces, including FilesetRef and plugin examples, out of the standalone SDK
• remove active SDK/plugin references to the legacy evaluator service package
• keep the accepted nmp-common plugin dependency for get_qualified_image/get_task_sdk
• fix bundle metadata generation so stale generated extras stop retaining removed service deps

Validation

• uv run ruff check packages/nemo_evaluator_sdk plugins/nemo-evaluator tools/nemo-platform-sdk-tools/src/nemo_platform_sdk_tools/sdk/vendor/vendor_package.py tools/nemo-platform-sdk-tools/tests/sdk/vendor/test_vendor_package.py
• uv run --frozen --extra cpu ty check ... focused evaluator files
• uv run pytest packages/nemo_evaluator_sdk/tests/test_params.py plugins/nemo-evaluator/tests/test_sdk.py plugins/nemo-evaluator/tests/test_evaluate_job.py tools/nemo-platform-sdk-tools/tests/sdk/vendor/test_vendor_package.py -q
• uv run pytest packages/nemo_evaluator_sdk/tests/execution/test_evaluator.py plugins/nemo-evaluator/tests/test_standalone_sdk_backend.py -q
• tools/lint/lint-all.sh

Summary by CodeRabbit

Release Notes

• New Features

  • Added FilesetRef type with fragment composition support for dataset references
  • Improved task execution with typed exit codes and graceful SIGTERM shutdown handling

• Refactor

  • Consolidated SDK dataset types and utilities to SDK-owned modules
  • Removed remote execution support, focusing on local evaluator workflows
  • Enhanced task initialization error handling with explicit logging

• Documentation

  • Clarified backend adapter ownership in execution design notes

• Chores

  • Removed legacy migration comments across modules
  • Updated dependency declarations and improved dependency group management

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	18419/24402	75.5%	62.0%
Integration Tests	11853/23179	51.1%	26.4%

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR consolidates FilesetRef into the evaluator SDK, adds local dataset utilities for the plugin, removes remote Platform example workflows, establishes a task entrypoint with graceful shutdown, implements lazy SDK exports, updates inferred parameter tests, and refactors dependencies from monolithic to modular packages.

Changes

Evaluator SDK and plugin consolidation

Layer / File(s)	Summary
FilesetRef type definition and imports plugins/nemo-evaluator/src/nemo_evaluator/sdk/values/filesets.py, plugins/nemo-evaluator/src/nemo_evaluator/sdk/types.py, plugins/nemo-evaluator/src/nemo_evaluator/sdk/__init__.py, plugins/nemo-evaluator/examples/plugin_examples.py, plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py, plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py, plugins/nemo-evaluator/tests/*	FilesetRef RootModel defined with with_fragment() method; imported across SDK init, plugin examples/jobs/executor, and tests from new SDK-local values module.
Dataset existence and download utilities plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py	dataset_exists() async, download_dataset() async, and download_dataset_sync() sync functions resolve FilesetRef references via Files API, handling whole-fileset/glob/subpath downloads; resolve_submit_dataset() validates FilesetRef datasets locally.
SDK lazy-loading and executor wiring plugins/nemo-evaluator/src/nemo_evaluator/sdk/__init__.py, plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py	Evaluator/JobResource imports deferred to TYPE_CHECKING; module-level __getattr__ provides on-demand loading; executor re-wired to import utilities from jobs.utils and FilesetRef from SDK values.
Evaluator task entrypoint plugins/nemo-evaluator/src/nemo_evaluator/tasks/evaluate.py, plugins/nemo-evaluator/tests/test_evaluate_job.py	EvaluateTaskExitCode enum with SDK_INITIALIZATION_FAILED code; SIGTERM handler; exception handling wraps SDK creation; test verifies init failure returns correct exit code.
Examples: remove remote Platform workflows packages/nemo_evaluator_sdk/examples/examples.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/README.md	Removes NeMo Platform helpers, run_nmp_* workflows, HelpSteer2 download logic; simplifies imports; updates README to clarify plugin-owned backend adapters.
InferenceParams contract and coercion tests packages/nemo_evaluator_sdk/tests/test_params.py	Tests refocused on SDK InferenceParams contract only; coercion test validates compatible foreign Pydantic model via model_validate and asserts dumped payload.
Legacy header cleanup packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/templates.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/*	Removes "Migrated from" header comments across SDK modules.
Dependencies and vendor tooling packages/nemo_platform/pyproject.toml, plugins/nemo-evaluator/pyproject.toml, tools/nemo-platform-sdk-tools/src/nemo_platform_sdk_tools/sdk/vendor/vendor_package.py, tools/nemo-platform-sdk-tools/tests/sdk/vendor/test_vendor_package.py	Replaces nmp-evaluator/nemo-platform with nemo-platform-sdk/nmp-common; adds _generated_optional_dependency_groups() to detect and regenerate bundled optional-dependency groups.

Suggested reviewers

• ngoncharenko
• arpitsardhana
• mckornfield

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 59.46% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main objective: detangling the evaluator plugin from the legacy service.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch detangle-evaluator-service-deps/schapman

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py (1)

48-54: 💤 Low value

Unused recursive parameter.

Both download_dataset and download_dataset_sync accept recursive but never use it. Either remove it or wire it into the download calls.

Also applies to: 92-100

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py` around lines 48 -
54, The `recursive` parameter on download_dataset and download_dataset_sync is
unused; either remove it or pass it through to the underlying NeMo SDK download
call. Locate the functions download_dataset and download_dataset_sync (they
accept sdk: AsyncNeMoPlatform and FilesetRef) and update the call that performs
the actual download to include the recursive flag (e.g., forward recursive into
the SDK method such as sdk.download_fileset or sdk.download_fileset_sync /
whatever method is used to fetch FilesetRef) so the parameter is honored, or if
recursion is not supported by the SDK drop the parameter from both function
signatures and all callers.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py`:
- Around line 48-54: The `recursive` parameter on download_dataset and
download_dataset_sync is unused; either remove it or pass it through to the
underlying NeMo SDK download call. Locate the functions download_dataset and
download_dataset_sync (they accept sdk: AsyncNeMoPlatform and FilesetRef) and
update the call that performs the actual download to include the recursive flag
(e.g., forward recursive into the SDK method such as sdk.download_fileset or
sdk.download_fileset_sync / whatever method is used to fetch FilesetRef) so the
parameter is honored, or if recursion is not supported by the SDK drop the
parameter from both function signatures and all callers.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1580103a-ae5f-41f8-b285-0664a42c3e1d

📥 Commits

Reviewing files that changed from the base of the PR and between fc9b0c4 and 2c8e330.

⛔ Files ignored due to path filters (7)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/datasets/loader.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/metrics/aggregation.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/templates.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/common.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/datasets.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py is excluded by !sdk/**
• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (23)

• packages/nemo_evaluator_sdk/examples/examples.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/README.md
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/templates.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/common.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/datasets.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/tests/test_params.py
• packages/nemo_platform/pyproject.toml
• plugins/nemo-evaluator/examples/plugin_examples.py
• plugins/nemo-evaluator/pyproject.toml
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/__init__.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/types.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/values/filesets.py
• plugins/nemo-evaluator/src/nemo_evaluator/tasks/evaluate/__main__.py
• plugins/nemo-evaluator/tests/test_evaluate_job.py
• plugins/nemo-evaluator/tests/test_sdk.py
• tools/nemo-platform-sdk-tools/src/nemo_platform_sdk_tools/sdk/vendor/vendor_package.py
• tools/nemo-platform-sdk-tools/tests/sdk/vendor/test_vendor_package.py

💤 Files with no reviewable changes (6)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/templates.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/common.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/datasets.py

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/README.md`:
- Around line 35-36: Replace the vague phrase "plugin package" in the
Evaluator(client=...) description with the concrete package/module name that
contains the platform-specific backend adapters (e.g., the actual backend
adapter package in this repo), and add a markdown link to that package's README
or API docs so users can find the adapters quickly; update the sentence to read
something like "Platform-specific backend adapters are provided by
<package-name> (see <link-to-package-docs>)."

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8d65d6ed-4857-41e4-99f8-bf62f3825cfe

📥 Commits

Reviewing files that changed from the base of the PR and between c8c3356 and 584cce9.

⛔ Files ignored due to path filters (7)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/datasets/loader.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/metrics/aggregation.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/templates.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/common.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/datasets.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py is excluded by !sdk/**
• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (23)

• packages/nemo_evaluator_sdk/examples/examples.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/README.md
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/templates.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/common.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/datasets.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/tests/test_params.py
• packages/nemo_platform/pyproject.toml
• plugins/nemo-evaluator/examples/plugin_examples.py
• plugins/nemo-evaluator/pyproject.toml
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/__init__.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/types.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/values/filesets.py
• plugins/nemo-evaluator/src/nemo_evaluator/tasks/evaluate.py
• plugins/nemo-evaluator/tests/test_evaluate_job.py
• plugins/nemo-evaluator/tests/test_sdk.py
• tools/nemo-platform-sdk-tools/src/nemo_platform_sdk_tools/sdk/vendor/vendor_package.py
• tools/nemo-platform-sdk-tools/tests/sdk/vendor/test_vendor_package.py

💤 Files with no reviewable changes (6)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/templates.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/datasets.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/common.py

🚧 Files skipped from review as they are similar to previous changes (14)

• packages/nemo_platform/pyproject.toml
• plugins/nemo-evaluator/examples/plugin_examples.py
• plugins/nemo-evaluator/pyproject.toml
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/values/filesets.py
• tools/nemo-platform-sdk-tools/tests/sdk/vendor/test_vendor_package.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/types.py
• plugins/nemo-evaluator/tests/test_sdk.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/init.py
• tools/nemo-platform-sdk-tools/src/nemo_platform_sdk_tools/sdk/vendor/vendor_package.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
• packages/nemo_evaluator_sdk/tests/test_params.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
• packages/nemo_evaluator_sdk/examples/examples.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py