[aleckhoury]

Summary

Refactors Eval Author’s portable metric handoff and moves Experimentalist to metric-contract-driven optimization. Regression metrics are selector context rather than a hard eligibility gate, while candidates must still report every Pareto objective.

Changes

• Stage and hand off authored Eval Author datasets and metric contracts.
• Use objective-aware Pareto ranking, direction-aware peer analysis, and LLM-guided soft regression constraints.
• Safely replace Harbor task directories only after validating the staged copy.
• Update metric configuration documentation and focused regression coverage.

Type of Change

• Code change with documentation updates
• Documentation only
• Contributor tooling or automation
• CI, build, or test infrastructure

Quality Gates

• Tests added or updated for changed behavior
• Documentation updated for user-visible behavior

Verification

• Pull request title follows the repository’s Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• Targeted tests pass
• No secrets, API keys, or credentials are included
• uv run pre-commit run -a passes — blocked by 38 existing type-check diagnostics from rebased main files outside this PR.

Targeted validation:

• .venv/bin/pytest plugins/nemo-experimentalist/tests/test_metric_contract.py plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py plugins/nemo-experimentalist/tests/test_experimentalist_analyzer.py plugins/nemo-experimentalist/tests/experimentalist/test_terminator.py -v — 32 passed.
• .venv/bin/pytest plugins/nemo-experimentalist/tests/test_experimentalist_backend.py -v — 19 passed.
• Focused Ruff lint and formatting checks — passed.
• uv run generate-config-docs && git diff --exit-code docs/set-up/config-reference.mdx — passed.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	31445/40067	78.5%	63.0%
Integration Tests	18319/38019	48.2%	20.8%

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Eval Author now stages and validates train, validation, and Insight inputs, authors shared metrics, and returns metric metadata. Experimentalist now applies objective and regression contracts throughout optimization instead of scalar Insight evaluation and promotion.

Changes

Eval Author metric handoff

Layer / File(s)	Summary
Contracts and input staging plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/{models.py,materialization.py,run.py}, documentation, tests	Added strict result models, metric-contract validation, and staging for train, validation, and task-template inputs.
Metric authoring and repair flow plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py, plugins/nemo-eval-author/tests/test_eval_author_*	Authored shared metrics across all datasets, validated them with repair feedback, and returned metric keys and summary metadata.

Experimentalist multi-objective optimization

Layer / File(s)	Summary
Metric contracts and optimization flow plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/{config.py,entities.py}, experimentalist/components/*	Added objective and regression metric contracts, Insight task distribution, candidate eligibility checks, Pareto-based selection, and Harbor task import support.
Metric-aware analysis and reporting plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/{components,reporting.py}	Propagated metric definitions and selection reasons through analysis, proposals, trace diagnosis, backend output, and reporting. Removed scalar reward reporting and Insight-specific persistence and promotion behavior.
Validation and integration coverage plugins/nemo-eval-author/tests/*, plugins/nemo-experimentalist/tests/*, documentation, examples	Updated tests, documentation, examples, and fixtures for staged datasets, metric contracts, multi-objective reporting, and revised handoff behavior.

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant run_eval_author
  participant DatasetFactory
  participant EvalAuthor
  participant EvolutionaryOptimizer
  participant RunReporter
  Caller->>run_eval_author: provide Eval Author inputs
  run_eval_author->>DatasetFactory: parse staged datasets
  run_eval_author->>EvalAuthor: pass train, validation, and Insight datasets
  EvalAuthor-->>Caller: return authored metrics and staged datasets
  EvolutionaryOptimizer->>EvolutionaryOptimizer: build objective and regression contracts
  EvolutionaryOptimizer->>RunReporter: report metric mappings and objective scores

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#903: Revises the same Eval Author and Experimentalist metric workflow.
• NVIDIA-NeMo/nemo-platform#932: Shares the Eval Author and Experimentalist integration.
• NVIDIA-NeMo/nemo-platform#1008: Shares dataset preparation and Experimentalist loop changes.

Suggested reviewers: arpitsardhana, ngoncharenko

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.83% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary Eval Author dataset augmentation refactor.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

⚔️ Resolve merge conflicts 💡

• Resolve merge conflict in branch ASE-780-eval-author-portable-verifier-patch/akhoury

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ASE-780-eval-author-portable-verifier-patch/akhoury

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py (1)

84-92: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Include URI resolution in the accumulated-failure path.

local_path_from_uri runs outside the try. A non-local task URI raises immediately and discards all failures collected so far. Move it inside the guarded block so the caller sees the full report.

♻️ Proposed change

-            task_dir = local_path_from_uri(
-                task.uri,
-                context=f"{dataset_name} dataset task {task.id!r}",
-            ).resolve()
             try:
+                task_dir = local_path_from_uri(
+                    task.uri,
+                    context=f"{dataset_name} dataset task {task.id!r}",
+                ).resolve()
                 verifier_dir = _verifier_dir(task_dir)
             except (FileNotFoundError, ValueError, tomllib.TOMLDecodeError) as exc:
                 failures.append(f"{dataset_name}/{task.id}: {exc}")
                 continue

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py`
around lines 84 - 92, Move the task.uri resolution and .resolve() logic into the
existing try block surrounding _verifier_dir(task_dir), so local_path_from_uri
failures are caught by the same exception handler. Preserve the current
failures.append and continue behavior to retain accumulated failures and
continue processing later tasks.

plugins/nemo-eval-author/tests/test_eval_author_agent.py (1)

263-263: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Drop raising=False for validate_metric_contracts.

agent.py imports the symbol at module scope, so it exists. With raising=False, a future rename or removal silently leaves the real validator unpatched and the test still passes.

♻️ Proposed change

-    monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts, raising=False)
+    monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py` at line 263, Update
the monkeypatch call for validate_metric_contracts in the test to remove
raising=False, ensuring the patch fails if the module-level symbol is renamed or
removed.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.py`:
- Around line 93-96: Update the validation in the model’s identity-checking path
around insight_suite_identity to require the exact sha256: prefix followed by 64
hexadecimal characters, while retaining the existing required/non-empty
behavior. Reject identities with an invalid prefix, incorrect length, or
non-hexadecimal characters, and add tests covering each case.

---

Nitpick comments:
In
`@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py`:
- Around line 84-92: Move the task.uri resolution and .resolve() logic into the
existing try block surrounding _verifier_dir(task_dir), so local_path_from_uri
failures are caught by the same exception handler. Preserve the current
failures.append and continue behavior to retain accumulated failures and
continue processing later tasks.

In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py`:
- Line 263: Update the monkeypatch call for validate_metric_contracts in the
test to remove raising=False, ensuring the patch fails if the module-level
symbol is renamed or removed.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cc3bd9bb-1a36-4b6e-9482-2ee8713a6ece

📥 Commits

Reviewing files that changed from the base of the PR and between 73670d7 and fc47c4f.

📒 Files selected for processing (23)

• plugins/nemo-eval-author/README.md
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.md
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.py
• plugins/nemo-eval-author/tests/test_eval_author_agent.py
• plugins/nemo-eval-author/tests/test_eval_author_materialization.py
• plugins/nemo-eval-author/tests/test_eval_author_models.py
• plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py
• plugins/nemo-eval-author/tests/test_eval_author_run.py
• plugins/nemo-eval-author/tests/test_plugin_boundary.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py
• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_helpers.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_model_config.py
• plugins/nemo-experimentalist/tests/test_experiment_mirror.py
• plugins/nemo-experimentalist/tests/test_experimentalist_backend.py

💤 Files with no reviewable changes (3)

• plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (3)

plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py (1)

51-72: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add coverage for the decline marker and the n/a path.

Every case here renders ▲. Two branches in candidate_evaluated stay untested: the ▼ marker when an objective moves against its direction, and f"{target.name} n/a" when the evaluator omits an objective metric. The second branch is the one that signals a broken metric contract, so it deserves a test.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py` around
lines 51 - 72, The test
test_candidate_evaluated_shows_all_objectives_and_direction_aware_deltas
currently covers only ▲ deltas; extend coverage for candidate_evaluated to
verify a directionally worsening objective renders ▼ and an omitted objective
metric renders the exact “{target.name} n/a” output. Include assertions for both
branches while preserving the existing objective formatting checks.

plugins/nemo-experimentalist/tests/test_metric_contract.py (1)

103-118: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover the name-collision and empty-key branches of _with_insight_objective.

Two branches stay untested. First, an authored Insight key that matches a configured regression or objective name: the helper drops it from regression_by_name to avoid the overlap ValueError raised by validate_metric_contract. Second, an empty metric_keys tuple, which returns the config unchanged. Both are cheap to assert and both guard against a run-time validation failure.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/tests/test_metric_contract.py` around lines 103
- 118, Add tests for _with_insight_objective covering an authored Insight key
that collides with an existing objective or regression metric, asserting the
resulting configuration passes validation without duplicate names and preserves
the intended metric placement. Add a separate test for an empty metric_keys
tuple, asserting the helper returns the configuration unchanged.

plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py (1)

6-6: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove from __future__ import annotations.

Use concrete annotations to follow the plugin convention. All referenced types are imported at module scope. nooa resolves deferred annotations with typing.get_type_hints, so this is a consistency refactor, not a schema-generation fix.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py`
at line 6, Remove the from __future__ import annotations statement from
selector.py, leaving the existing concrete type annotations and module-level
imports unchanged.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-experimentalist/README.md`:
- Around line 157-172: Split the combined YAML example in the documentation into
two separate fenced YAML blocks, each with its own descriptive introductory
text. Keep the single-metric example and the multi-metric example as
independent, valid YAML documents without duplicate objective_function keys.

In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.py`:
- Around line 417-424: Update the peer-comparison flow around
`_select_divergent_pairs` and `_find_complementary_failures` to accept and apply
each metric’s direction contract before computing deltas or selecting leaders,
so minimize metrics consistently treat lower values as better. Propagate the
metric definitions from the caller through both computations and add coverage
for a mixed maximize/minimize comparison.

In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.py`:
- Around line 1205-1220: Update the task import flow around _from_task_dir to
validate the staged task before renaming over an existing destination, and
retain the backup until reloading the replacement destination succeeds. Restore
the backup and clean up staging if validation or reload fails, preserving the
original task. Add a test covering replacement with malformed task.toml.

In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py`:
- Around line 1418-1442: Update the candidate filtering flow around eligible,
excluded, and ranked to handle the case where every candidate is excluded:
either fail fast with an explicit error or fall back to the original candidates
so the round cannot proceed with zero survivors. Build the exclusion list by
comparing candidate labels against the labels of eligible candidates, rather
than comparing Pydantic model instances by value.

In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.py`:
- Around line 196-200: Update the call to qualitative_stop_check to pass the
resolved active_objectives and active_regressions values instead of
objective_metrics and regression_metrics, preserving the fallback behavior
established when those active lists are resolved.

---

Nitpick comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py`:
- Line 6: Remove the from __future__ import annotations statement from
selector.py, leaving the existing concrete type annotations and module-level
imports unchanged.

In `@plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py`:
- Around line 51-72: The test
test_candidate_evaluated_shows_all_objectives_and_direction_aware_deltas
currently covers only ▲ deltas; extend coverage for candidate_evaluated to
verify a directionally worsening objective renders ▼ and an omitted objective
metric renders the exact “{target.name} n/a” output. Include assertions for both
branches while preserving the existing objective formatting checks.

In `@plugins/nemo-experimentalist/tests/test_metric_contract.py`:
- Around line 103-118: Add tests for _with_insight_objective covering an
authored Insight key that collides with an existing objective or regression
metric, asserting the resulting configuration passes validation without
duplicate names and preserves the intended metric placement. Add a separate test
for an empty metric_keys tuple, asserting the helper returns the configuration
unchanged.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 827baef6-02cc-454c-8a52-097573fa0d87

📥 Commits

Reviewing files that changed from the base of the PR and between fc47c4f and 4c61c4e.

📒 Files selected for processing (22)

• plugins/nemo-experimentalist/README.md
• plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.example
• plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/cache.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/dataset_staging.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_analyzer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py
• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py
• plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_terminator.py
• plugins/nemo-experimentalist/tests/test_experimentalist_analyzer.py
• plugins/nemo-experimentalist/tests/test_metric_contract.py

💤 Files with no reviewable changes (1)

• plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (3)

plugins/nemo-eval-author/tests/test_eval_author_agent.py (1)

263-263: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Drop raising=False.

agent.py imports validate_metric_contracts, so the attribute exists. With raising=False, a future rename silently creates a dead patch and the tests keep passing against unpatched code.

🔧 Proposed fix

-    monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts, raising=False)
+    monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py` at line 263, Update
the monkeypatch in the test around validate_metric_contracts to remove
raising=False, relying on the imported validate_metric_contracts attribute being
present so renamed symbols fail loudly instead of silently creating an
ineffective patch.

plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py (1)

189-190: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Replace the string-based return annotation.

Use typing.Self instead of "EvolutionaryOptimizerConfig".

♻️ Proposed change

     `@model_validator`(mode="after")
-    def validate_metric_contract(self) -> "EvolutionaryOptimizerConfig":
+    def validate_metric_contract(self) -> Self:

Add Self to the typing import at line 23:

-from typing import Any, Literal
+from typing import Any, Literal, Self

As per coding guidelines: "Prefer concrete type hints over string-based type hints".

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py`
around lines 189 - 190, Update the return annotation of
EvolutionaryOptimizerConfig.validate_metric_contract to use typing.Self instead
of the quoted class name, and add Self to the existing typing imports.

Source: Coding guidelines

plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py (1)

555-566: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Do not require exactly one authored metric key.

author_insight_metrics may author several keys. Line 557 fails the canary when the model authors two. Compare every authored key instead.

♻️ Proposed change

-    assert len(violating_metric_names) == 1
-    assert compliant_metric_names == violating_metric_names
-    metric_name = next(iter(violating_metric_names))
-    violating_score = violating_payload[metric_name]
-    compliant_score = compliant_payload[metric_name]
-    assert isinstance(violating_score, int | float) and not isinstance(violating_score, bool)
-    assert isinstance(compliant_score, int | float) and not isinstance(compliant_score, bool)
-    assert 0.0 <= violating_score <= 1.0
-    assert 0.0 <= compliant_score <= 1.0
-    assert compliant_score > violating_score
+    assert violating_metric_names
+    assert compliant_metric_names == violating_metric_names
+    for metric_name in violating_metric_names:
+        violating_score = violating_payload[metric_name]
+        compliant_score = compliant_payload[metric_name]
+        assert isinstance(violating_score, int | float) and not isinstance(violating_score, bool)
+        assert isinstance(compliant_score, int | float) and not isinstance(compliant_score, bool)
+        assert 0.0 <= violating_score <= 1.0
+        assert 0.0 <= compliant_score <= 1.0
+    assert any(
+        compliant_payload[name] > violating_payload[name] for name in violating_metric_names
+    )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py` around lines
555 - 566, Update the assertions in the evaluation comparison around
violating_metric_names and compliant_metric_names so multiple authored metric
keys are supported instead of requiring exactly one. Compare every authored key
present in both payloads, validate each violating_score and compliant_score as
numeric values in [0, 1], and require compliant_score > violating_score for
every key while preserving reward exclusion.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.py`:
- Line 96: Update EvalAuthorResult’s metric_keys validation around
_validate_metric_keys to assign and return the normalized tuple rather than
discarding it. Implement this as a field validator that skips the empty default,
preserving empty defaults while stripping whitespace from supplied keys
consistently with MetricAuthoringResult.

In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py`:
- Around line 83-122: Update satisfies_regression_constraints and
is_eligible_for_metric_contract to handle a baseline_metrics mapping that lacks
declared regression targets before comparing candidate values. Choose and
implement an explicit failure or documented fallback policy so missing round-0
regression metrics cannot silently reject every candidate, while preserving the
existing constraint comparisons when baseline values are present.

---

Nitpick comments:
In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py`:
- Line 263: Update the monkeypatch in the test around validate_metric_contracts
to remove raising=False, relying on the imported validate_metric_contracts
attribute being present so renamed symbols fail loudly instead of silently
creating an ineffective patch.

In `@plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py`:
- Around line 555-566: Update the assertions in the evaluation comparison around
violating_metric_names and compliant_metric_names so multiple authored metric
keys are supported instead of requiring exactly one. Compare every authored key
present in both payloads, validate each violating_score and compliant_score as
numeric values in [0, 1], and require compliant_score > violating_score for
every key while preserving reward exclusion.

In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py`:
- Around line 189-190: Update the return annotation of
EvolutionaryOptimizerConfig.validate_metric_contract to use typing.Self instead
of the quoted class name, and add Self to the existing typing imports.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f7a29ec0-5340-4368-8879-387dc0a453d9

📥 Commits

Reviewing files that changed from the base of the PR and between c71ca67 and 9fc6517.

📒 Files selected for processing (42)

• plugins/nemo-eval-author/README.md
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.md
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.py
• plugins/nemo-eval-author/tests/test_eval_author_agent.py
• plugins/nemo-eval-author/tests/test_eval_author_materialization.py
• plugins/nemo-eval-author/tests/test_eval_author_models.py
• plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py
• plugins/nemo-eval-author/tests/test_eval_author_run.py
• plugins/nemo-eval-author/tests/test_plugin_boundary.py
• plugins/nemo-experimentalist/README.md
• plugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.example
• plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/cache.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/dataset_staging.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_analyzer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py
• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_helpers.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_model_config.py
• plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_terminator.py
• plugins/nemo-experimentalist/tests/test_experiment_mirror.py
• plugins/nemo-experimentalist/tests/test_experimentalist_analyzer.py
• plugins/nemo-experimentalist/tests/test_experimentalist_backend.py
• plugins/nemo-experimentalist/tests/test_metric_contract.py

💤 Files with no reviewable changes (4)

• plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
• plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py

🚧 Files skipped from review as they are similar to previous changes (30)

• plugins/nemo-eval-author/tests/test_plugin_boundary.py
• plugins/nemo-experimentalist/README.md
• plugins/nemo-eval-author/README.md
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.md
• plugins/nemo-experimentalist/tests/experimentalist/test_model_config.py
• plugins/nemo-experimentalist/tests/experimentalist/test_terminator.py
• plugins/nemo-experimentalist/tests/test_experimentalist_backend.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.py
• plugins/nemo-experimentalist/tests/test_experiment_mirror.py
• plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/dataset_staging.py
• plugins/nemo-experimentalist/tests/test_experimentalist_analyzer.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py
• plugins/nemo-eval-author/tests/test_eval_author_models.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.py
• plugins/nemo-experimentalist/tests/test_metric_contract.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/cache.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_analyzer.py
• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/reporting.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.py
• plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py
• plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py