Skip to content

feat(evaluator): resolve model refs for plugin submit targets - #126

Closed
SandyChapman wants to merge 1 commit into
mainfrom
evaluator-modelref-target-main/schapman
Closed

feat(evaluator): resolve model refs for plugin submit targets#126
SandyChapman wants to merge 1 commit into
mainfrom
evaluator-modelref-target-main/schapman

Conversation

@SandyChapman

Copy link
Copy Markdown
Contributor

Summary

  • allow evaluator plugin submit targets to accept platform ModelRef values
  • resolve ModelRef targets to concrete SDK Model values before building the evaluator job spec
  • keep SDK/job execution target types concrete and add coverage for config validation

Testing

  • uv run --frozen pytest plugins/nemo-evaluator/tests/test_sdk.py plugins/nemo-evaluator/tests/test_evaluate_job.py -q
  • uv run --frozen ruff check plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py plugins/nemo-evaluator/src/nemo_evaluator/sdk/resources.py plugins/nemo-evaluator/tests/test_sdk.py
  • uv run --frozen --extra cpu ty check plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py plugins/nemo-evaluator/src/nemo_evaluator/sdk/resources.py plugins/nemo-evaluator/tests/test_sdk.py

Signed-off-by: Sandy Chapman <schapman@nvidia.com>
@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 18419/24402 75.5% 62.0%
Integration Tests 11853/23179 51.1% 26.3%

@SandyChapman

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #128. The ModelRef submit-target support from this PR has been folded into the broader evaluator ref-contract cleanup there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant