feat: author and evaluate insight suite metrics - #903
Conversation
Reissues NVIDIA-dev/NeMo-Optimizer#91 in the nemo-platform monorepo. Signed-off-by: Ryan Angilly <rangilly@nvidia.com>
Reissues NVIDIA-dev/NeMo-Optimizer#95 in the nemo-platform monorepo. Signed-off-by: Ryan Angilly <rangilly@nvidia.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughInsight suites are staged, validated, finalized as local content-addressed artifacts, evaluated on baseline and generated candidates, persisted as an ChangesInsight suite evaluation
Sequence Diagram(s)sequenceDiagram
participant EvolutionaryOptimizer
participant EvalAuthor
participant InsightSuite
participant Candidate
participant ExperimentMirror
EvolutionaryOptimizer->>EvalAuthor: obtain finalized insight dataset
EvalAuthor->>InsightSuite: author and validate verifier metrics
EvolutionaryOptimizer->>InsightSuite: evaluate baseline and candidate agents
InsightSuite-->>EvolutionaryOptimizer: return metrics and trial results
EvolutionaryOptimizer->>Candidate: persist insight rewards and provenance
EvolutionaryOptimizer->>ExperimentMirror: project the insight split
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/tests/test_experiment_mirror.py`:
- Line 11: Remove the leftover “# VERIFY-2” comment from the nemo_platform
import in the test module, leaving the imported symbols unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ae42fb4d-2a9f-4dd5-9bd5-6531ecf8a033
📒 Files selected for processing (12)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.mdplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/agent.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/models.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experiment_mirror.pyplugins/nemo-experimentalist/tests/experimentalist/test_eval_author_repair_e2e.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/test_eval_author_agent.pyplugins/nemo-experimentalist/tests/test_eval_author_materialization.pyplugins/nemo-experimentalist/tests/test_experiment_mirror.py
Reissues NVIDIA-dev/NeMo-Optimizer#96 for the Experimentalist plugin.\n\nLinear: ASE-625, ASE-699 Signed-off-by: Ryan Angilly <rangilly@nvidia.com>
Reissues NVIDIA-dev/NeMo-Optimizer#97 for the Experimentalist plugin.\n\nLinear: ASE-684, ASE-699 Signed-off-by: Ryan Angilly <rangilly@nvidia.com>
|
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py (1)
1032-1032: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winUse the full insight dataset id here.
HarborDataset.from_path()names insight jobsinsight-<digest>, so"-insight"leavesagent-*insight result dirs behind on rollback.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py` at line 1032, Update the suffix iteration in the rollback cleanup logic to use the full insight dataset identifier generated by HarborDataset.from_path(), matching the insight-<digest> naming convention. Preserve the existing train and validation suffix handling while ensuring agent-* insight result directories are also removed.
🧹 Nitpick comments (4)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py (2)
170-170: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAnnotate the accumulator —
dict[str, list[float]]avoids an inferred partially-unknown type under strict type checking.- values = {metric_name: [] for metric_name in required_metrics} + values: dict[str, list[float]] = {metric_name: [] for metric_name in required_metrics}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py` at line 170, Annotate the values accumulator in the surrounding insight-promotion logic as dict[str, list[float]], while preserving the existing required_metrics-based initialization and accumulation behavior.
219-224: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDead check.
_task_metric_valuesalready rejects any non-completedtrial, socompleted_attempts != total_attemptscan never hold here; only thenot total_attemptsbranch is reachable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py` around lines 219 - 224, In the surrounding trial-validation logic, remove the unreachable completed_attempts calculation and comparison from the total-attempts guard. Retain only the not total_attempts check that can still return None, while preserving the existing _task_metric_values validation.plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py (1)
1426-1428: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRedundant metadata write. Line 1428 is subsumed by the final assignment at Lines 1449-1450; drop it.
Also applies to: 1449-1450
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py` around lines 1426 - 1428, Remove the immediate dataset.metadata["insight_metric_keys"] assignment from the expected_metric_keys initialization branch, while retaining the final assignment that already writes the metric keys after processing.plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.py (1)
98-119: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winDrop persisted per-file hash dicts from
manifest.json.Each task entry stores a full
"files"map (andverifier.files) of every relative path → hash, but_content_provenancealways recomputes these fresh from disk on resolve/finalize and never reads the stored values back — this is write-only data that inflatesmanifest.jsonwith the size/count of files in each task (includingenvironment/build context).♻️ Keep only the digests, not the full file listing
tasks.append( { **task_entry, "content_hash": content_hash, "verifier": { "path": verifier_path, "content_hash": verifier_hash, - "files": verifier_files, }, - "files": files, } )Also applies to: 404-419
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.py` around lines 98 - 119, Remove the persisted per-file hash dictionaries from task manifest entries: omit the top-level "files" field and the verifier "files" field in the task construction around _file_hashes, while retaining the computed content_hash and verifier.content_hash digests and existing provenance calculations. Apply the same change to the corresponding manifest construction around the additional referenced section.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py`:
- Around line 1406-1416: Metric-key comparisons currently depend on input
ordering, causing equivalent sets to be treated inconsistently. In loop.py lines
1406-1416, sort both candidate.insight_metric_keys and dataset_metric_keys
before creating cached_metric_key_sets; in insight_promotion.py lines 194-201,
sort candidate.insight_metric_keys before creating metric_key_sets. Preserve the
existing disagreement and promotion logic after normalizing the keys.
- Around line 1849-1866: Guard the insight-report rendering in the finalize flow
so ValueError from write_insight_comparison_section or
write_insight_promotion_section cannot abort a completed run. Add defensive
handling around the insight sections, matching the existing write_final_report
protection, while preserving result persistence and winner handling when insight
metadata is missing or incompatible.
---
Outside diff comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py`:
- Line 1032: Update the suffix iteration in the rollback cleanup logic to use
the full insight dataset identifier generated by HarborDataset.from_path(),
matching the insight-<digest> naming convention. Preserve the existing train and
validation suffix handling while ensuring agent-* insight result directories are
also removed.
---
Nitpick comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.py`:
- Around line 98-119: Remove the persisted per-file hash dictionaries from task
manifest entries: omit the top-level "files" field and the verifier "files"
field in the task construction around _file_hashes, while retaining the computed
content_hash and verifier.content_hash digests and existing provenance
calculations. Apply the same change to the corresponding manifest construction
around the additional referenced section.
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py`:
- Line 170: Annotate the values accumulator in the surrounding insight-promotion
logic as dict[str, list[float]], while preserving the existing
required_metrics-based initialization and accumulation behavior.
- Around line 219-224: In the surrounding trial-validation logic, remove the
unreachable completed_attempts calculation and comparison from the
total-attempts guard. Retain only the not total_attempts check that can still
return None, while preserving the existing _task_metric_values validation.
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py`:
- Around line 1426-1428: Remove the immediate
dataset.metadata["insight_metric_keys"] assignment from the expected_metric_keys
initialization branch, while retaining the final assignment that already writes
the metric keys after processing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 2d1dd4f9-4db7-43e0-8a4a-5ed5e59933da
📒 Files selected for processing (13)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.mdplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/agent.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/models.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/test_eval_author_agent.pyplugins/nemo-experimentalist/tests/test_eval_author_materialization.pyplugins/nemo-experimentalist/tests/test_experimentalist_backend.py
🚧 Files skipped from review as they are similar to previous changes (3)
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.md
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/agent.py
- plugins/nemo-experimentalist/tests/test_eval_author_agent.py
aleckhoury
left a comment
aleckhoury
left a comment
There was a problem hiding this comment.
I was going to leave more specific line by line reviews, but it may be better to pass this overview to your agent. Really just some areas we can simplify the hashing and copying, then a couple actual bugs:
Keep
Deterministic suite/scorer identities derived from task and verifier content.
Candidate rewards bound to the suite identity.
Resume invalidation when the authored suite changes.
Runtime metric validation: consistent keys, finite [0,1] values, and complete trials.
Conservative baseline-to-winner promotion qualification.
Deterministic Insight comparison reporting.
Simplify
The nemo-optimizer-insight-suite:// URI, resolver, and second copy under artifacts//insight-suite/ appear unnecessary at this stage.
The URI only resolves relative to the same local experiment directory. The suite is not uploaded to Filesets, included in candidate publication, or otherwise made portable. Therefore, this provides content integrity but not actual durability.
For now, I suggest:
Keep one suite at: eval-and-optimize/eval_author//insight-suite/
Compute and persist its suite/scorer identities in place.
Return the local HarborDataset with those identities in its metadata.
Cache and invalidate candidate results using the suite identity alone.
Remove the custom URI, resolver, duplicate artifact copy, and insight_suite_artifact_ref fields.
Report the local task path plus suite/task/verifier hashes in advisory promotion output.
Durable publication should happen later, only when a user explicitly promotes a task into canonical validation.
Two additional fixes
Harbor defaults to n_attempts=1, but promotion qualification requires at least two attempts. As written, suggestions cannot qualify under the default configuration. Insight-suite evaluation should use max(configured_n_attempts, 2) while leaving train and validation unchanged.
Deterministic comparison/promotion rendering currently risks failing an otherwise successful run during finalization. Evaluator and invalid-metric failures should remain fatal, but report enrichment should warn and omit the affected section rather than fail the optimization.
This retains the meaningful ASE-684 correctness guarantees while removing premature durability machinery and making promotion suggestions functional by default.
aleckhoury
left a comment
aleckhoury
left a comment
There was a problem hiding this comment.
Approving, with plan to address comments and test e2e once merged
cbe92c2 to
9cd6ede
Compare
|
@aleckhoury Addressed the review feedback in 9cd6ede:
Validation: 567 Experimentalist tests passed (2 opt-in tests skipped), Ruff lint/format passed, and targeted ty passed. All applicable pre-commit hooks passed; the local Studio lint hook could not start because pnpm 10.32.1 is not installed, and no Studio files changed. |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.md (1)
42-75: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winSplit lifecycle explanation from return-contract reference.
Move the
EvalAuthorResultfield contract to a reference page and cross-link it; keep this section focused on the materialization lifecycle.As per coding guidelines, “Each documentation page should fit ONE Diataxis quadrant; do not mix tutorials with reference tables or how-tos with architecture explanations; use cross-links instead.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.md` around lines 42 - 75, Move the return-contract details for EvalAuthorResult.insight_suite and EvalAuthorResult.insight_suite_identity out of the Materialized Insight Suite lifecycle section into an appropriate reference page, then add a cross-link here. Keep this section focused solely on suite materialization, validation, identity persistence, and resume lifecycle behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py`:
- Around line 196-199: Update the candidate processing in the insight promotion
flow to build a single suite_candidates collection filtered by
provenance.identity. Reuse it for metric-key extraction, trial evidence, and
candidate_count so stale candidates from other suites cannot affect scoring or
promotion suggestions.
---
Outside diff comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.md`:
- Around line 42-75: Move the return-contract details for
EvalAuthorResult.insight_suite and EvalAuthorResult.insight_suite_identity out
of the Materialized Insight Suite lifecycle section into an appropriate
reference page, then add a cross-link here. Keep this section focused solely on
suite materialization, validation, identity persistence, and resume lifecycle
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9a99ee50-b414-4596-9844-96b0b8ad11fa
📒 Files selected for processing (12)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.mdplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/agent.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/models.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/test_eval_author_agent.pyplugins/nemo-experimentalist/tests/test_eval_author_materialization.pyplugins/nemo-experimentalist/tests/test_experiment_mirror.py
💤 Files with no reviewable changes (1)
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.py
🚧 Files skipped from review as they are similar to previous changes (6)
- plugins/nemo-experimentalist/tests/test_experiment_mirror.py
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/models.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/agent.py
- plugins/nemo-experimentalist/tests/test_eval_author_agent.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py
Signed-off-by: Ryan Angilly <rangilly@nvidia.com>
9cd6ede to
06fbf9e
Compare
|
Addressed the second CodeRabbit review round in
Validation: 568 passed, 2 opt-in skipped; Ruff lint/format and targeted |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/REFERENCE.md (1)
6-24: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd the required reference-page sections and examples.
This page needs a prerequisites section at the top, tabbed Python SDK and CLI examples, and a
Next Stepssection with cross-links at the end. The current field table is useful, but does not satisfy those required documentation elements.As per coding guidelines, documentation pages must list prerequisites first, provide Python SDK and CLI examples in tab sets, and end with cross-linked Next Steps.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/REFERENCE.md` around lines 6 - 24, Add the required documentation structure to the Eval Author Python Reference page: place a Prerequisites section first, add tabbed examples for both the Python SDK and CLI, and finish with a Next Steps section containing cross-links to relevant documentation. Preserve the existing EvalAuthorResult field table and explanatory content.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/REFERENCE.md`:
- Around line 6-24: Add the required documentation structure to the Eval Author
Python Reference page: place a Prerequisites section first, add tabbed examples
for both the Python SDK and CLI, and finish with a Next Steps section containing
cross-links to relevant documentation. Preserve the existing EvalAuthorResult
field table and explanatory content.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 70858717-06fa-4b33-b8c6-2353bc4fecb5
📒 Files selected for processing (13)
plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.mdplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/REFERENCE.mdplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/agent.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/models.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/test_eval_author_agent.pyplugins/nemo-experimentalist/tests/test_eval_author_materialization.pyplugins/nemo-experimentalist/tests/test_experiment_mirror.py
💤 Files with no reviewable changes (1)
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.py
🚧 Files skipped from review as they are similar to previous changes (11)
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/README.md
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/models.py
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
- plugins/nemo-experimentalist/tests/test_experiment_mirror.py
- plugins/nemo-experimentalist/tests/test_eval_author_materialization.py
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/materialization.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/eval_author/agent.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py
- plugins/nemo-experimentalist/tests/test_eval_author_agent.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py
2 participants
Summary
Stack
mainLinear
Source
Reissues NVIDIA-dev/NeMo-Optimizer#91 after the Experimentalist plugin moved into this monorepo in #896.
Validation
uv run --frozen --group experimentalist pytest -q plugins/nemo-experimentalist/tests/test_eval_author_agent.py plugins/nemo-experimentalist/tests/test_eval_author_materialization.py plugins/nemo-experimentalist/tests/test_experiment_mirror.py plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py plugins/nemo-experimentalist/tests/experimentalist/test_eval_author_repair_e2e.py— 37 passed, 2 credential-gated skipsuv run --frozen ruff check plugins/nemo-experimentalistuv run --frozen ruff format --check plugins/nemo-experimentalistuv run --frozen ty checkon changed production files and new testsSKIP=studio-lint-staged uv run pre-commit run -a— all applicable hooks passed; Studio was skipped because this is a Python-only change and the optional Node 22.18/pnpm environment is not installedSummary by CodeRabbit