refactor(evaluator): move metric resolution to to_spec

[SandyChapman]

Summary

• Rebase the evaluator metric resolution change onto current main.
• Resolve metric model refs during job spec conversion before compile/run.
• Keep the current bundle-based evaluator plugin execution path after the mainline refactors.

Verification

• tools/lint/lint-all.sh
• uv run --frozen ruff check plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py plugins/nemo-evaluator/tests/test_evaluate_job.py plugins/nemo-evaluator/tests/test_sdk.py
• uv run --frozen ty check plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py plugins/nemo-evaluator/tests/test_evaluate_job.py plugins/nemo-evaluator/tests/test_sdk.py
• uv run --frozen pytest plugins/nemo-evaluator/tests/test_evaluate_job.py plugins/nemo-evaluator/tests/test_sdk.py -q

Summary by CodeRabbit

• Refactor

  • Model-reference resolution moved earlier into spec compilation so unresolved model refs are caught before execution; local run flow simplified to operate on canonical specs.

• New Features

  • Introduced a distinct submitter-facing input spec and unified handling so executors accept either input or compiled specs; job payloads accept both forms.

• Tests

  • Updated tests to validate pre-compilation resolution, spec canonicalization, and packaged local-run behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: df89ebb4-126b-44f9-8fd0-161501edcff7

📥 Commits

Reviewing files that changed from the base of the PR and between fdbe198 and aeaf28d.

📒 Files selected for processing (5)

• plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.py
• plugins/nemo-evaluator/tests/test_evaluate_job.py
• plugins/nemo-evaluator/tests/test_sdk.py

✅ Files skipped from review due to trivial changes (1)

• plugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.py

🚧 Files skipped from review as they are similar to previous changes (4)

• plugins/nemo-evaluator/tests/test_evaluate_job.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
• plugins/nemo-evaluator/tests/test_sdk.py

---

📝 Walkthrough

Walkthrough

Refactors evaluator: adds submitter-facing EvaluateInputSpec; moves bundled-metric hydration and model-ref resolution into EvaluateJob.to_spec(); EvaluateSpec now rejects unresolved bundled model refs; executors accept EvaluateInputSpec|EvaluateSpec and canonicalize to resolved EvaluateSpec for local runs; tests updated accordingly.

Changes

Evaluate Job Spec Compilation Workflow

Layer / File(s)	Summary
Input spec, imports, and metric-bundle helpers plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py	Adds EvaluateInputSpec, expands bundling/unbundling and packager imports, and adds module-level helpers to hydrate metrics, find unresolved MetricWithModels roots, apply runtime params, resolve model refs asynchronously, and re-bundle resolved metrics.
Canonical spec compilation plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py	Implements EvaluateJob.to_spec(...) to normalize/validate EvaluateInputSpec, hydrate metrics, apply job params, resolve model refs via async_sdk when provided, re-bundle if modified, and return a validated EvaluateSpec; run() now expects validated EvaluateSpec.
Executor imports and request type plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py	Adds `EvaluateRequestSpec: TypeAlias = EvaluateInputSpec
Local spec canonicalization (sync/async) plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py	Adds _resolve_sync_local_spec and _resolve_async_local_spec to convert an EvaluateRequestSpec into a canonical EvaluateSpec for local runs by invoking EvaluateJob.to_spec (sync via run_sync, async via await).
Sync executor local flow plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py	Sync create/run_local accept EvaluateRequestSpec, canonicalize to EvaluateSpec, and evaluate/evaluate_benchmark build with CloudpickleMetricBundlePackager(), dispatch local run, and parse/validate JSON results from local_result_path.
Async executor local flow plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py	Async create/run_local accept EvaluateRequestSpec, canonicalize via async to_spec, dispatch local run in a worker thread, and load/validate JSON results using asyncio.to_thread.
HTTP payload typing plugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.py	create_job_payload now accepts `EvaluateInputSpec
Job-level spec validation tests plugins/nemo-evaluator/tests/test_evaluate_job.py	Tests updated to await EvaluateJob.to_spec, assert EvaluateSpec rejects unresolved bundled metric model refs, assert to_spec resolves bundled model refs prior to compile, and assert job params/prompt-template defaults apply when no model refs.
SDK executor packaging tests plugins/nemo-evaluator/tests/test_sdk.py	Tests refactored to patch executor.run_local and assert it receives a packaged EvaluateInputSpec with a single exact-match metric, the expected dataset (including FilesetRef fragment selector for glob), and RunConfig(parallelism=2); adds helpers to materialize local run results.

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#46: Introduced bundled-metric/cloudpickle packaging used by this PR's unbundle/rebundle and packager usage.

Suggested reviewers

• ngoncharenko
• arpitsardhana

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 58.54% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Title accurately describes the main refactoring: moving metric model reference resolution from runtime into the to_spec compilation step.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 4050-evaluator-to-spec-resolution/schapman

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py`:
- Line 200: The parameter annotation for async_sdk currently declares
AsyncNeMoPlatform but the function (around the async_sdk None check at line 213)
treats it as optional; update the signature to use AsyncNeMoPlatform | None for
the async_sdk parameter so type checkers reflect the actual None handling
(locate the function/method that accepts async_sdk and change its type
annotation accordingly).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2b19b9d9-15f8-4708-aa30-9f76b436a388

📥 Commits

Reviewing files that changed from the base of the PR and between 254747b and 73bfcbb.

📒 Files selected for processing (5)

• plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py
• plugins/nemo-evaluator/src/nemo_evaluator/sdk/http_utils.py
• plugins/nemo-evaluator/tests/test_evaluate_job.py
• plugins/nemo-evaluator/tests/test_sdk.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	18420/24402	75.5%	62.0%
Integration Tests	11850/23179	51.1%	26.3%

[SandyChapman]

Closing as superseded by #128. The metric resolution/to_spec work has been folded into the evaluator ref-contract cleanup branch there.