Skip to content

fix: make Insight evaluation promotion-safe - #907

Merged
ryana merged 1 commit into
ase-625-insight-suite-promotion/rangillyfrom
ase-684-insight-suite-correctness/rangilly
Jul 27, 2026
Merged

fix: make Insight evaluation promotion-safe#907
ryana merged 1 commit into
ase-625-insight-suite-promotion/rangillyfrom
ase-684-insight-suite-correctness/rangilly

Conversation

@ryana

@ryana ryana commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • give finalized Insight Suites content-addressed suite, scorer, task, and verifier identities
  • make evaluation resume-safe and reject stale or incomparable metric evidence
  • add bounded metric validation, conservative promotion gates, deterministic comparison reporting, and fallback-safe finalization
  • use the Experimentalist-owned nemo-experimentalist-insight-suite:// artifact scheme

Stack

This is layer 4 of 4 and must merge after:

  1. feat: author and evaluate insight suite metrics #903
  2. feat: report insight suite optimization metrics #905
  3. feat: suggest Insight Suite tasks for validation #906
  4. this PR

The base branch is ase-625-insight-suite-promotion/rangilly, so this PR displays only its incremental correctness and safety diff.

Provenance

  • Reissues NVIDIA-dev/NeMo-Optimizer#97
  • Linear: ASE-684
  • Migration tracking: ASE-699

Validation

  • uv run --frozen --group experimentalist pytest -q plugins/nemo-experimentalist/tests — 563 passed, 2 skipped
  • uv run --frozen ruff check plugins/nemo-experimentalist — passed
  • uv run --frozen ruff format --check plugins/nemo-experimentalist — passed
  • ty check passed for every changed Python file
  • SKIP=studio-lint-staged uv run pre-commit run -a — all applicable hooks passed

The two skipped tests are existing opt-in checks that require external credentials/runtime. The Studio hook was skipped because this stack is Python-only and the local Studio toolchain is unavailable (Node 22.14 is below the repository's 22.18 minimum, and pnpm is not installed).

Reissues NVIDIA-dev/NeMo-Optimizer#97 for the Experimentalist plugin.\n\nLinear: ASE-684, ASE-699

Signed-off-by: Ryan Angilly <rangilly@nvidia.com>
@ryana
ryana requested review from a team as code owners July 27, 2026 16:26
@ryana
ryana requested a review from aleckhoury July 27, 2026 16:26
@github-actions github-actions Bot added the fix label Jul 27, 2026
@ryana
ryana merged commit 1800292 into ase-625-insight-suite-promotion/rangilly Jul 27, 2026
4 checks passed
@ryana
ryana deleted the ase-684-insight-suite-correctness/rangilly branch July 27, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant