feat(evaluator): refresh evaluator plugin skills - #1055
Merged
Conversation
Contributor
ngoncharenko
commented
Aug 3, 2026
ngoncharenko
commented
Aug 3, 2026
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe change documents local and durable evaluator workflows, converts example specifications to inline metric bundles, expands validation coverage, removes evaluator documentation entry points, and updates Fabric runtime integration. ChangesEvaluator workflow updates
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 10
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-evaluator/tests/test_evaluate_job.py (1)
71-74: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winCover
fabric_agent_eval.jsonin the agent-evaluation compiler test.
test_checked_durable_fabric_job_is_a_valid_agent_eval_specvalidates onlyAgentEvalInputSpec. The compiler test uses a separate synthetic spec. Load this fixture, convert it throughAgentEvalJob.to_spec, and compile it withAgentEvalJob.compile.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/tests/test_evaluate_job.py` around lines 71 - 74, Update the agent-evaluation compiler test around EXAMPLE_SPEC_PATHS and test_checked_durable_fabric_job_is_a_valid_agent_eval_spec to include the fabric_agent_eval.json fixture. Load that fixture, convert it with AgentEvalJob.to_spec, and compile the resulting spec using AgentEvalJob.compile instead of relying on the separate synthetic spec.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/evaluator/agent-eval/index.mdx`:
- Line 16: Hyphenate the compound modifier as “platform-durable” in both
execution-mode notes: update docs/evaluator/agent-eval/index.mdx lines 16-16 to
“Local and platform-durable interfaces are available.” and
docs/evaluator/index.mdx lines 123-123 to “Agent evaluation has local and
platform-durable interfaces.”
- Around line 17-18: Update the local SDK usage note in
docs/evaluator/agent-eval/index.mdx at lines 17-18 to document await
AgentEvaluator().run(tasks=..., target=...). Apply the same await addition to
the corresponding note in docs/evaluator/index.mdx at lines 124-125; no other
changes are needed.
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py`:
- Around line 248-254: Update the requires_live_fabric skip reason to reference
the documented installation command, adding --frozen and --inexact to the
existing uv sync invocation while preserving the rest of the guidance.
In `@plugins/nemo-evaluator/README.md`:
- Around line 74-100: Correct the “Plugin SDK Execution” documentation around
the NeMoPlatform evaluator calls: either add a separate
client.evaluator.run(...) example demonstrating local execution, or revise the
heading and description to state that client.evaluator.submit(...) is durable
execution only.
In `@script/dev-install-fabric.sh`:
- Around line 20-21: Update the comment around the package-scoped command to say
it restores the locked project dependencies rather than returning the
environment to a locked state, and mention using an exact sync when a clean
environment is required.
In `@skills/nemo-evaluator-plugin/references/agent-evaluation.md`:
- Line 8: Update the sentence introducing AgentEvaluator to use correct grammar,
stating that it provides lightweight in-process evaluation without requiring a
running nemo-platform.
In `@skills/nemo-evaluator-plugin/references/execution.md`:
- Line 11: Add the required blank lines around the headings at lines 11, 63, and
196, and before and after the fenced code blocks at lines 69, 77, 89, 130, 137,
141, 158, and 170 in the markdown document, without changing its content.
In `@skills/nemo-evaluator-plugin/references/metric-selection.md`:
- Around line 31-36: Update the metric discovery commands in
skills/nemo-evaluator-plugin/references/metric-selection.md:31-36 to prefix both
nemo invocations with uv run, ensuring repository checkouts use the project CLI.
Apply the same execution-context fix to the secret commands in
skills/nemo-evaluator-plugin/references/api-auth.md:15-28 by prefixing them with
uv run, or explicitly labeling them as installed-platform commands.
In `@skills/nemo-evaluator-plugin/references/resources.md`:
- Around line 21-56: Update the snippet before its first client.evaluator usage
to either initialize client with NeMoPlatform or explicitly state that an
existing client is required; ensure the resulting example is self-contained and
runnable.
In `@skills/nemo-evaluator-plugin/SKILL.md`:
- Around line 38-53: Update the “CLI Interface” prerequisites in SKILL.md to
require setting NMP_BASE_URL to http://localhost:8080, checking whether the
local platform instance is already running, and verifying
$NMP_BASE_URL/health/ready before any CLI commands. Instruct users to stop and
follow SETUP.md if the readiness check fails.
---
Outside diff comments:
In `@plugins/nemo-evaluator/tests/test_evaluate_job.py`:
- Around line 71-74: Update the agent-evaluation compiler test around
EXAMPLE_SPEC_PATHS and
test_checked_durable_fabric_job_is_a_valid_agent_eval_spec to include the
fabric_agent_eval.json fixture. Load that fixture, convert it with
AgentEvalJob.to_spec, and compile the resulting spec using AgentEvalJob.compile
instead of relying on the separate synthetic spec.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 5e46cf47-c6af-43ee-b959-4333635877ad
📒 Files selected for processing (25)
docs/evaluator/agent-eval/index.mdxdocs/evaluator/index.mdxpackages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.pyplugins/nemo-evaluator/README.mdplugins/nemo-evaluator/src/nemo_evaluator/cli.pyplugins/nemo-evaluator/src/nemo_evaluator/docs/index.mdplugins/nemo-evaluator/tests/test_evaluate_job.pyplugins/nemo-evaluator/tests/test_skill_examples.pyscript/dev-install-fabric.shskills/nemo-evaluator-plugin/SKILL.mdskills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.pyskills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.jsonskills/nemo-evaluator-plugin/assets/specs/exact_match_metric.jsonskills/nemo-evaluator-plugin/assets/specs/fabric_agent_eval.jsonskills/nemo-evaluator-plugin/assets/specs/llm_as_judge.jsonskills/nemo-evaluator-plugin/references/agent-evaluation.mdskills/nemo-evaluator-plugin/references/api-auth.mdskills/nemo-evaluator-plugin/references/evaluation-shapes.mdskills/nemo-evaluator-plugin/references/execution.mdskills/nemo-evaluator-plugin/references/llm-judge.mdskills/nemo-evaluator-plugin/references/metric-selection.mdskills/nemo-evaluator-plugin/references/resources.mdskills/nemo-evaluator-plugin/references/troubleshooting.mdskills/nemo-evaluator-plugin/scripts/generate_example_specs.py
💤 Files with no reviewable changes (1)
- skills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.json
ngoncharenko
changed the title
Contributor
|
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
skills/nemo-evaluator-plugin/references/troubleshooting.md (1)
3-4: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd prerequisites before the first command.
Add a
Prerequisitessection beforeInspect the installed contracts. State that thenemo evaluatorCLI must be available and that repository checkouts must run commands throughuv run.As per coding guidelines, documentation pages must list prerequisites at the top.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/troubleshooting.md` around lines 3 - 4, Add a Prerequisites section at the top of the troubleshooting document, before “Inspect the installed contracts,” stating that the nemo evaluator CLI must be available and that repository checkouts must invoke commands through uv run. Preserve the existing command instructions and CLI wording.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@skills/nemo-evaluator-plugin/references/troubleshooting.md`:
- Around line 3-4: Add a Prerequisites section at the top of the troubleshooting
document, before “Inspect the installed contracts,” stating that the nemo
evaluator CLI must be available and that repository checkouts must invoke
commands through uv run. Preserve the existing command instructions and CLI
wording.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d0e6398e-b4c8-45c8-a412-fc81c9c5a842
📒 Files selected for processing (15)
.cursor/hooks/state/continual-learning.jsondocs/evaluator/agent-eval/index.mdxdocs/evaluator/index.mdxpackages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.pyplugins/nemo-evaluator/README.mdplugins/nemo-evaluator/tests/test_agent_evaluate.pyplugins/nemo-evaluator/tests/test_evaluate_job.pyplugins/nemo-evaluator/tests/test_skill_examples.pyscript/dev-install-fabric.shskills/nemo-evaluator-plugin/SKILL.mdskills/nemo-evaluator-plugin/references/agent-evaluation.mdskills/nemo-evaluator-plugin/references/execution.mdskills/nemo-evaluator-plugin/references/metric-selection.mdskills/nemo-evaluator-plugin/references/resources.mdskills/nemo-evaluator-plugin/references/troubleshooting.md
🚧 Files skipped from review as they are similar to previous changes (12)
- docs/evaluator/agent-eval/index.mdx
- script/dev-install-fabric.sh
- skills/nemo-evaluator-plugin/references/metric-selection.md
- skills/nemo-evaluator-plugin/references/resources.md
- plugins/nemo-evaluator/tests/test_skill_examples.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
- skills/nemo-evaluator-plugin/references/agent-evaluation.md
- plugins/nemo-evaluator/README.md
- plugins/nemo-evaluator/tests/test_evaluate_job.py
- skills/nemo-evaluator-plugin/SKILL.md
- docs/evaluator/index.mdx
- skills/nemo-evaluator-plugin/references/execution.md
Contributor
Author
|
/nvskills-ci |
Contributor
|
/nvskills-ci |
SandyChapman
approved these changes
Aug 4, 2026
ngoncharenko
commented
Aug 4, 2026
ngoncharenko
force-pushed
the
ngoncharenko/aalgo-550-eval-plugin-skills
branch
from
August 5, 2026 19:03
d97623d to
1a47c6e
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (4)
plugins/nemo-evaluator/README.md (2)
41-43: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRun documented Python CLI commands through
uv.
plugins/nemo-evaluator/README.md#L41-L43: replacenemo --helpwithuv run nemo --help.plugins/nemo-evaluator/README.md#L76-L81: prefix eachnemo jobscommand withuv run.As per coding guidelines, run Python scripts and tools through
uv.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/README.md` around lines 41 - 43, Update the documented CLI commands in plugins/nemo-evaluator/README.md at lines 41-43 and 76-81: change the nemo --help example to use uv run nemo --help, and prefix every nemo jobs command with uv run. No other documentation changes are needed.Source: Coding guidelines
3-27: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftSeparate the documentation quadrants.
Classify this README as REFERENCE. Move setup and execution procedures to HOW-TO pages. Move evaluation-shape concepts to an EXPLANATION page.
As per coding guidelines, each documentation page must fit one Diataxis quadrant.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/README.md` around lines 3 - 27, Reclassify this README as REFERENCE by retaining only the evaluator plugin interfaces, entry points, and concise behavior descriptions. Move setup and execution procedures to appropriate HOW-TO documentation, and move concepts describing evaluation shapes such as dataset-driven versus task-driven jobs to an EXPLANATION page; update links or references accordingly.Source: Coding guidelines
skills/nemo-evaluator-plugin/references/evaluation-shapes.md (1)
1-3: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the required navigation sections.
Add
PrerequisitesbeforeDifference summary. AddNext Stepsafter the final paragraph with links to the dataset and agent-evaluation procedures.As per coding guidelines, documentation pages must list prerequisites at the top and include a
Next Stepssection.Also applies to: 40-41
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/evaluation-shapes.md` around lines 1 - 3, Update the evaluation-shapes documentation page by adding a Prerequisites section before Difference summary, then add a Next Steps section after the final paragraph. Include links in Next Steps to the dataset and agent-evaluation procedures, preserving the existing content and ordering otherwise.Source: Coding guidelines
skills/nemo-evaluator-plugin/scripts/generate_example_specs.py (1)
62-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAnnotate
metricasMetric.Import
Metricfromnemo_evaluator_sdk.metrics.protocoland replaceAnyin_bundle;bundle_metricrequires this protocol.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/scripts/generate_example_specs.py` around lines 62 - 63, Update the _bundle function to import Metric from nemo_evaluator_sdk.metrics.protocol and annotate its metric parameter as Metric instead of Any, matching the protocol required by bundle_metric.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-evaluator/README.md`:
- Around line 129-139: Update the Fabric submission instructions to copy the
checked spec before submission, replace its target.model placeholder with a real
provider/model identifier, and pass the copied spec file to `agent-evaluate
submit` instead of the unchanged checked file. Preserve the existing environment
and capture_trajectory guidance.
In `@skills/nemo-evaluator-plugin/references/llm-judge.md`:
- Around line 3-4: Add a Prerequisites section before the configuration content
in llm-judge.md, covering the SDK environment, reachable judge endpoint, model
ID, and required credentials. Append a Next Steps section at the end with links
to the API authentication and execution guidance documentation.
- Line 43: Update the online judging template near the example at line 43 and
the related section around lines 75-78 to use the generated sample output field,
sample.output_text, instead of the offline dataset field item.output. Keep the
offline example using item.output, or clearly separate the two templates so each
workflow references its correct output source.
- Line 24: Update the standalone example’s api_key_secret value to use
SecretRef(root="NVIDIA_API_KEY") instead of the platform placeholder. Keep
platform-specific secret remapping only in the authentication guidance,
preserving the example’s standalone environment-variable behavior.
In `@skills/nemo-evaluator-plugin/SKILL.md`:
- Line 134: Update the standalone workflow link in SKILL.md to use the required
execution.md anchor “validate-standalone-then-submit-to-the-platform” instead of
“evaluate-standalone-then-submit-to-the-platform”, preserving the existing link
text and reference target.
---
Nitpick comments:
In `@plugins/nemo-evaluator/README.md`:
- Around line 41-43: Update the documented CLI commands in
plugins/nemo-evaluator/README.md at lines 41-43 and 76-81: change the nemo
--help example to use uv run nemo --help, and prefix every nemo jobs command
with uv run. No other documentation changes are needed.
- Around line 3-27: Reclassify this README as REFERENCE by retaining only the
evaluator plugin interfaces, entry points, and concise behavior descriptions.
Move setup and execution procedures to appropriate HOW-TO documentation, and
move concepts describing evaluation shapes such as dataset-driven versus
task-driven jobs to an EXPLANATION page; update links or references accordingly.
In `@skills/nemo-evaluator-plugin/references/evaluation-shapes.md`:
- Around line 1-3: Update the evaluation-shapes documentation page by adding a
Prerequisites section before Difference summary, then add a Next Steps section
after the final paragraph. Include links in Next Steps to the dataset and
agent-evaluation procedures, preserving the existing content and ordering
otherwise.
In `@skills/nemo-evaluator-plugin/scripts/generate_example_specs.py`:
- Around line 62-63: Update the _bundle function to import Metric from
nemo_evaluator_sdk.metrics.protocol and annotate its metric parameter as Metric
instead of Any, matching the protocol required by bundle_metric.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 4ffa1e84-6c28-46ec-81c8-d62795d31972
📒 Files selected for processing (29)
docs/evaluator/agent-eval/index.mdxdocs/evaluator/index.mdxpackages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.pypackages/nemo_platform/pyproject.tomlplugins/nemo-evaluator/README.mdplugins/nemo-evaluator/pyproject.tomlplugins/nemo-evaluator/src/nemo_evaluator/cli.pyplugins/nemo-evaluator/src/nemo_evaluator/docs.pyplugins/nemo-evaluator/src/nemo_evaluator/docs/index.mdplugins/nemo-evaluator/tests/test_agent_evaluate.pyplugins/nemo-evaluator/tests/test_evaluate_job.pyplugins/nemo-evaluator/tests/test_skill_examples.pyscript/dev-install-fabric.shskills/nemo-evaluator-plugin/SKILL.mdskills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.pyskills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.jsonskills/nemo-evaluator-plugin/assets/specs/exact_match_metric.jsonskills/nemo-evaluator-plugin/assets/specs/fabric_agent_eval.jsonskills/nemo-evaluator-plugin/assets/specs/llm_as_judge.jsonskills/nemo-evaluator-plugin/references/agent-evaluation.mdskills/nemo-evaluator-plugin/references/api-auth.mdskills/nemo-evaluator-plugin/references/evaluation-shapes.mdskills/nemo-evaluator-plugin/references/execution.mdskills/nemo-evaluator-plugin/references/llm-judge.mdskills/nemo-evaluator-plugin/references/metric-selection.mdskills/nemo-evaluator-plugin/references/resources.mdskills/nemo-evaluator-plugin/references/troubleshooting.mdskills/nemo-evaluator-plugin/scripts/generate_example_specs.py
💤 Files with no reviewable changes (5)
- plugins/nemo-evaluator/src/nemo_evaluator/docs.py
- plugins/nemo-evaluator/pyproject.toml
- packages/nemo_platform/pyproject.toml
- skills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.json
- plugins/nemo-evaluator/src/nemo_evaluator/docs/index.md
🚧 Files skipped from review as they are similar to previous changes (17)
- docs/evaluator/index.mdx
- script/dev-install-fabric.sh
- skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json
- plugins/nemo-evaluator/src/nemo_evaluator/cli.py
- skills/nemo-evaluator-plugin/references/execution.md
- plugins/nemo-evaluator/tests/test_agent_evaluate.py
- docs/evaluator/agent-eval/index.mdx
- packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
- packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py
- skills/nemo-evaluator-plugin/references/agent-evaluation.md
- skills/nemo-evaluator-plugin/references/resources.md
- skills/nemo-evaluator-plugin/assets/specs/llm_as_judge.json
- skills/nemo-evaluator-plugin/references/troubleshooting.md
- skills/nemo-evaluator-plugin/references/metric-selection.md
- skills/nemo-evaluator-plugin/references/api-auth.md
- skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py
- plugins/nemo-evaluator/tests/test_evaluate_job.py
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
skills/nemo-evaluator-plugin/references/llm-judge.md (1)
19-24: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMake the Python example executable or label it as a template.
provider.exampleand<judge-model-id>are placeholders, so this code cannot run as written. Use the tested endpoint and model fromskills/nemo-evaluator-plugin/scripts/generate_example_specs.py, or clearly label this block as a non-runnable template.As per coding guidelines, “Ensure all code snippets are tested and actually work before publishing.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/llm-judge.md` around lines 19 - 24, Update the Python example centered on LLMJudgeMetric to either use the tested endpoint and model values from generate_example_specs.py, including the corresponding secret configuration, or clearly label the block as a non-runnable template; do not leave provider.example or <judge-model-id> presented as executable values.Source: Coding guidelines
♻️ Duplicate comments (1)
skills/nemo-evaluator-plugin/references/llm-judge.md (1)
43-44: 🎯 Functional Correctness | 🟠 MajorUse generated output for online judging.
The prompt reads
{{item.output}}, which is an existing dataset response. The online workflow scores generated samples, and the checked specification uses{{sample.output_text}}. Split the offline and online examples, or change the online prompt to use{{sample.output_text}}.Also applies to: 75-78
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/llm-judge.md` around lines 43 - 44, Update the online judging prompt in the relevant example to use the generated response variable sample.output_text instead of the dataset field item.output. Keep the offline example using item.output, or split the examples clearly so each workflow references its correct output source.
🧹 Nitpick comments (1)
skills/nemo-evaluator-plugin/references/llm-judge.md (1)
10-17: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPut Python and CLI examples in a tab set.
Group the SDK example and CLI commands under one tab set. Add equivalent CLI guidance if the current commands only inspect metrics and do not configure the judge workflow.
As per coding guidelines, “Provide both Python SDK and CLI examples in tab-sets for consistency and to support multiple user workflows.”
Also applies to: 62-67
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/llm-judge.md` around lines 10 - 17, Update the documentation section containing the Python SDK example and CLI commands to present both workflows in a single tab set. Ensure the CLI tab includes equivalent judge-workflow configuration guidance, not only metric inspection commands, while preserving the existing Python example content.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@skills/nemo-evaluator-plugin/references/llm-judge.md`:
- Around line 19-24: Update the Python example centered on LLMJudgeMetric to
either use the tested endpoint and model values from generate_example_specs.py,
including the corresponding secret configuration, or clearly label the block as
a non-runnable template; do not leave provider.example or <judge-model-id>
presented as executable values.
---
Duplicate comments:
In `@skills/nemo-evaluator-plugin/references/llm-judge.md`:
- Around line 43-44: Update the online judging prompt in the relevant example to
use the generated response variable sample.output_text instead of the dataset
field item.output. Keep the offline example using item.output, or split the
examples clearly so each workflow references its correct output source.
---
Nitpick comments:
In `@skills/nemo-evaluator-plugin/references/llm-judge.md`:
- Around line 10-17: Update the documentation section containing the Python SDK
example and CLI commands to present both workflows in a single tab set. Ensure
the CLI tab includes equivalent judge-workflow configuration guidance, not only
metric inspection commands, while preserving the existing Python example
content.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b26e4728-d345-447b-b0fc-d468164e8a45
📒 Files selected for processing (8)
skills/nemo-evaluator-plugin/SKILL.mdskills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.pyskills/nemo-evaluator-plugin/assets/specs/fabric_agent_eval.jsonskills/nemo-evaluator-plugin/references/evaluation-shapes.mdskills/nemo-evaluator-plugin/references/execution.mdskills/nemo-evaluator-plugin/references/llm-judge.mdskills/nemo-evaluator-plugin/references/metric-selection.mdskills/nemo-evaluator-plugin/scripts/generate_example_specs.py
🚧 Files skipped from review as they are similar to previous changes (7)
- skills/nemo-evaluator-plugin/references/metric-selection.md
- skills/nemo-evaluator-plugin/references/execution.md
- skills/nemo-evaluator-plugin/references/evaluation-shapes.md
- skills/nemo-evaluator-plugin/assets/specs/fabric_agent_eval.json
- skills/nemo-evaluator-plugin/SKILL.md
- skills/nemo-evaluator-plugin/scripts/generate_example_specs.py
- skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
ngoncharenko
force-pushed
the
ngoncharenko/aalgo-550-eval-plugin-skills
branch
from
August 5, 2026 20:09
221fae7 to
86984f1
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
skills/nemo-evaluator-plugin/references/execution.md (3)
43-43: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse the configured product-name substitution.
Replace
NeMo Platformin prose with the configured documentation substitution. Keep API and CLI identifiers literal.As per coding guidelines: “Never hardcode product names; use substitutions in Sphinx configuration to maintain consistency.”
Also applies to: 124-125
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/execution.md` at line 43, Update the prose around the nemo-evaluator-plugin execution guidance to use the configured Sphinx product-name substitution instead of hardcoded “NeMo Platform”; keep API and CLI identifiers, including client.evaluator.submit, literal.Source: Coding guidelines
32-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse tab sets for SDK and CLI alternatives.
The paired SDK and CLI procedures are sequential. Put each pair in one tab set.
As per coding guidelines: “Use tab sets for parallel alternatives or variants.”
Also applies to: 71-86, 94-137, 147-172, 178-214
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/execution.md` around lines 32 - 63, Restructure the paired Platform CLI and Platform Python SDK procedures into a single tab set, with each procedure as a separate tab presenting the same execution alternative. Apply the same tab-set structure to the corresponding paired sections at the other referenced locations, preserving their existing commands and examples.Source: Coding guidelines
1-7: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the required Diataxis page structure.
Both pages omit prerequisites and a final Next Steps section.
troubleshooting.mdalso mixes REFERENCE failures with a HOW-TO debugging procedure.
skills/nemo-evaluator-plugin/references/execution.md#L1-L7: classify the page as HOW-TO and list prerequisites before the procedures.skills/nemo-evaluator-plugin/references/execution.md#L221-L279: add a final Next Steps section with related cross-links.skills/nemo-evaluator-plugin/references/troubleshooting.md#L1-L4: classify the page as REFERENCE and list prerequisites.skills/nemo-evaluator-plugin/references/troubleshooting.md#L36-L42: move the debugging procedure to a HOW-TO page, then add final Next Steps cross-links.As per coding guidelines: “Each documentation page should fit ONE Diataxis quadrant,” “Always list prerequisites at the top,” and “Include ‘Next Steps’ section at the end.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/references/execution.md` around lines 1 - 7, Update skills/nemo-evaluator-plugin/references/execution.md lines 1-7 to classify the page as HOW-TO and add prerequisites before its procedures; add a final Next Steps section with related cross-links at lines 221-279. Update skills/nemo-evaluator-plugin/references/troubleshooting.md lines 1-4 to classify it as REFERENCE and add prerequisites; move the debugging procedure from lines 36-42 to a HOW-TO page, then end troubleshooting.md with Next Steps cross-links so each page fits one Diataxis quadrant.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-evaluator/README.md`:
- Line 119: Rename the duplicate “CLI Commands” heading in the README to “Agent
evaluation CLI commands” to satisfy markdownlint MD024 while preserving the
section content.
- Around line 6-7: Remove references to the retired `nemo evaluator evaluate
run` workflow from the README’s interface list and examples. Retain
documentation for the durable `submit` workflow, and direct local validation
users to the standalone SDK instead, consistent with the guidance in `SKILL.md`.
In `@skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py`:
- Around line 86-104: Update build_agent_eval_spec so CodexRunnerTarget does not
receive the literal "<codex-model>" placeholder; construct it without a model to
use the CLI default, or add and pass through an optional model argument while
preserving the existing default behavior.
In `@skills/nemo-evaluator-plugin/assets/specs/llm_as_judge.json`:
- Around line 57-62: Update the judge prompt content in the llm-as-judge
specification to clearly delimit both rendered values, item.input and
sample.output_text, and explicitly instruct the judge to treat them as data and
ignore any embedded directives or attempts to alter scoring. Add adversarial
evaluation cases verifying prompt-injected instructions in either value cannot
change the required helpfulness score or JSON-only response format.
---
Nitpick comments:
In `@skills/nemo-evaluator-plugin/references/execution.md`:
- Line 43: Update the prose around the nemo-evaluator-plugin execution guidance
to use the configured Sphinx product-name substitution instead of hardcoded
“NeMo Platform”; keep API and CLI identifiers, including
client.evaluator.submit, literal.
- Around line 32-63: Restructure the paired Platform CLI and Platform Python SDK
procedures into a single tab set, with each procedure as a separate tab
presenting the same execution alternative. Apply the same tab-set structure to
the corresponding paired sections at the other referenced locations, preserving
their existing commands and examples.
- Around line 1-7: Update skills/nemo-evaluator-plugin/references/execution.md
lines 1-7 to classify the page as HOW-TO and add prerequisites before its
procedures; add a final Next Steps section with related cross-links at lines
221-279. Update skills/nemo-evaluator-plugin/references/troubleshooting.md lines
1-4 to classify it as REFERENCE and add prerequisites; move the debugging
procedure from lines 36-42 to a HOW-TO page, then end troubleshooting.md with
Next Steps cross-links so each page fits one Diataxis quadrant.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9cdccf46-8091-4e84-b285-6f447b9d57cc
📒 Files selected for processing (29)
docs/evaluator/agent-eval/index.mdxdocs/evaluator/index.mdxpackages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.pypackages/nemo_platform/pyproject.tomlplugins/nemo-evaluator/README.mdplugins/nemo-evaluator/pyproject.tomlplugins/nemo-evaluator/src/nemo_evaluator/cli.pyplugins/nemo-evaluator/src/nemo_evaluator/docs.pyplugins/nemo-evaluator/src/nemo_evaluator/docs/index.mdplugins/nemo-evaluator/tests/test_agent_evaluate.pyplugins/nemo-evaluator/tests/test_evaluate_job.pyplugins/nemo-evaluator/tests/test_skill_examples.pyscript/dev-install-fabric.shskills/nemo-evaluator-plugin/SKILL.mdskills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.pyskills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.jsonskills/nemo-evaluator-plugin/assets/specs/exact_match_metric.jsonskills/nemo-evaluator-plugin/assets/specs/fabric_agent_eval.jsonskills/nemo-evaluator-plugin/assets/specs/llm_as_judge.jsonskills/nemo-evaluator-plugin/references/agent-evaluation.mdskills/nemo-evaluator-plugin/references/api-auth.mdskills/nemo-evaluator-plugin/references/evaluation-shapes.mdskills/nemo-evaluator-plugin/references/execution.mdskills/nemo-evaluator-plugin/references/llm-judge.mdskills/nemo-evaluator-plugin/references/metric-selection.mdskills/nemo-evaluator-plugin/references/resources.mdskills/nemo-evaluator-plugin/references/troubleshooting.mdskills/nemo-evaluator-plugin/scripts/generate_example_specs.py
💤 Files with no reviewable changes (5)
- plugins/nemo-evaluator/src/nemo_evaluator/docs/index.md
- packages/nemo_platform/pyproject.toml
- plugins/nemo-evaluator/pyproject.toml
- skills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.json
- plugins/nemo-evaluator/src/nemo_evaluator/docs.py
🚧 Files skipped from review as they are similar to previous changes (15)
- plugins/nemo-evaluator/src/nemo_evaluator/cli.py
- docs/evaluator/index.mdx
- skills/nemo-evaluator-plugin/assets/specs/fabric_agent_eval.json
- packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py
- skills/nemo-evaluator-plugin/references/resources.md
- skills/nemo-evaluator-plugin/references/llm-judge.md
- skills/nemo-evaluator-plugin/references/agent-evaluation.md
- skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json
- packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
- plugins/nemo-evaluator/tests/test_agent_evaluate.py
- script/dev-install-fabric.sh
- skills/nemo-evaluator-plugin/references/api-auth.md
- docs/evaluator/agent-eval/index.mdx
- plugins/nemo-evaluator/tests/test_evaluate_job.py
- skills/nemo-evaluator-plugin/references/metric-selection.md
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Contributor
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.run()from SKILLS since it's going awayIn Progress
Attempt #1launched with a custom image locally - pipeline hangsAttempt #2: "Unable to run agent/model preflight: Unable to fetch models from https://inference-api.nvidia.com/v1/models: The read operation timed out" - looking into thatLinear: AALGO-450
Filled out a few stories to resolve some identified gaps: https://linear.app/nvidia/project/evaluator-plugin-agenteval-support-29a12b9a6a95/overview
Summary by CodeRabbit
New Features
agent-evaluateworkflow.Documentation
Tests