feat(evaluator): promote generic agentic runtimes into the Agent-Eval SDK - #383
Merged
Conversation
arpitsardhana
marked this pull request as ready for review
June 20, 2026 21:20
arpitsardhana
force-pushed
the
aalgo-258-nat-runner/arpsingh
branch
from
June 22, 2026 06:36
dd5d412 to
e01b23b
Compare
Contributor
Author
|
@coderabbitai full review |
Contributor
✅ Action performedFull review finished. |
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a complete Agent-Eval SDK Pipeline and Runtime Stack
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 10
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/nemo_evaluator_sdk/examples/run_agent_eval/README.md (1)
1-227: 🛠️ Refactor suggestion | 🟠 Major | 🏗️ Heavy liftSplit this page by Diataxis and move prerequisites to the top.
This page mixes tutorial/how-to/explanation/reference, places prerequisites late, and omits a “Next Steps” section. Restructure into single-quadrant pages with cross-links.
As per coding guidelines, "Each documentation page should fit ONE Diataxis quadrant; do not mix tutorials with reference tables or how-tos with architecture explanations; use cross-links instead", "Always list prerequisites at the top of documentation pages before other content", and "Include 'Next Steps' section at the end with cross-links to related documentation content."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/README.md` around lines 1 - 227, The README.md mixes tutorial content (Run it), how-to guides (Plugging in a real workflow, Running a real tests/agentic-use task), explanation (What it demonstrates, Execution flow), and reference material (Files table) without clear separation, and places prerequisites late in the document. Reorganize the content by moving the prerequisites section to the very top before "Run it", create distinct sections aligned to Diataxis quadrants (Tutorial for "Run it", How-To for the real workflow sections, Explanation for "What it demonstrates" and execution flow, Reference for "Files" and configuration tables), and add a "Next Steps" section at the end with cross-links to related documentation. Each reorganized section should stand independently and reference other sections via cross-links rather than embedding mixed content types together.Source: Coding guidelines
🧹 Nitpick comments (5)
packages/nemo_evaluator_sdk/examples/run_agent_eval/aut_runtime.py (2)
169-169: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick winImport inside function body.
inject_gateway_urlis imported insideprepare_aut_config_for_runtime. Move to module-level for consistency with coding guidelines (prefer concrete imports, not deferred).Proposed fix
from pathlib import Path import yaml +from nemo_agents_plugin.utils import inject_gateway_url from nemo_evaluator_sdk.agent_eval.runtimes.environment import AgentEnvironmentProvider, EnvRunSpecThen remove line 169.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/aut_runtime.py` at line 169, The import statement for `inject_gateway_url` from `nemo_agents_plugin.utils` is currently inside the `prepare_aut_config_for_runtime` function body at line 169. Move this import to the module-level imports at the top of the file with the other import statements, then remove the import statement from inside the function body.Source: Coding guidelines
72-75: 🧹 Nitpick | 🔵 Trivial | ⚖️ Poor tradeoffAPI key interpolation may break on special characters.
The inline Python script at line 73 replaces
${NVIDIA_API_KEY}and${ANTHROPIC_API_KEY}placeholders via simple string replacement. If an API key contains quotes or backslashes, the resulting YAML could be malformed.Consider using proper YAML serialization instead of text replacement.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/aut_runtime.py` around lines 72 - 75, The API key interpolation in the inline Python script uses simple string replacement with the replace() method for NVIDIA_API_KEY and ANTHROPIC_API_KEY placeholders. This approach is fragile and will break if API keys contain special characters like quotes or backslashes, potentially creating malformed YAML. Instead of using str.replace() to substitute these placeholders, load the YAML file using a YAML library like PyYAML, update the parsed dictionary with the actual API key values, and then serialize it back to YAML format. This ensures proper escaping and quoting of special characters in the API keys.packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py (2)
253-264: 🧹 Nitpick | 🔵 Trivial | ⚖️ Poor tradeoffBrittle string replacement for workflow rewriting.
text.replace("http://localhost:8080", ...)and model name replacement assume exact string matches. If the source YAML has different formatting (e.g., quotes, trailing slashes), replacement silently fails.Consider parsing YAML first, modifying the data structure, then serializing—which you already do later for the tracing config (line 266+).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py` around lines 253 - 264, The current implementation uses brittle string replacements (for "http://localhost:8080", model name, and MCP client function groups) that will silently fail if the YAML has different formatting like quotes or spacing. Replace these string-based replacements with a proper YAML parsing approach: parse the workflow_path content using a YAML parser to convert it to a Python dictionary, modify the relevant fields in the data structure (nmp_base_url under appropriate keys, model_name field, and function_groups vs functions), then serialize the modified structure back to YAML text. This approach is more robust and maintainable, and aligns with how the tracing config is already handled later in the function.
546-554: 🧹 Nitpick | 🔵 Trivial | 💤 Low valueSequential task execution limits throughput.
run_tasksprocesses tasks sequentially without parallelism. UnlikeWorkflowAgentRuntimewhich usesasyncio.Semaphore, this awaits each task serially. If intentional (e.g., Docker resource constraints), consider documenting it; otherwise, add concurrency control.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py` around lines 546 - 554, The run_tasks method processes tasks sequentially using a simple for loop with await, which limits throughput. Refactor the task execution to use concurrent processing with asyncio.Semaphore (similar to WorkflowAgentRuntime) to allow multiple tasks to run in parallel while still respecting resource constraints. Replace the sequential loop that awaits self._run_task for each task with either asyncio.gather or asyncio.TaskGroup along with a Semaphore to control concurrency. Alternatively, if sequential execution is intentional due to constraints like Docker resource limitations, add a docstring comment to the run_tasks method documenting this design decision.packages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.py (1)
20-20: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick winScan all namespace roots, not just the first path entry.
Line 20 only checks one
agent_eval.__path__entry. In split namespace installs, this can miss offenders.Suggested fix
-AGENT_EVAL_ROOT = Path(next(iter(agent_eval.__path__))).resolve() +AGENT_EVAL_ROOTS = [Path(p).resolve() for p in agent_eval.__path__] @@ - for path in sorted(AGENT_EVAL_ROOT.rglob("*.py")): - text = path.read_text(encoding="utf-8") - for match in _FORBIDDEN.finditer(text): - line_no = text.count("\n", 0, match.start()) + 1 - offenders.append(f"{path.relative_to(AGENT_EVAL_ROOT)}:{line_no}: {match.group(0).strip()}") + for root in AGENT_EVAL_ROOTS: + for path in sorted(root.rglob("*.py")): + text = path.read_text(encoding="utf-8") + for match in _FORBIDDEN.finditer(text): + line_no = text.count("\n", 0, match.start()) + 1 + offenders.append(f"{path.relative_to(root)}:{line_no}: {match.group(0).strip()}")Also applies to: 32-37
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.py` at line 20, The current implementation using next(iter(agent_eval.__path__)) only examines the first path entry in the namespace, which causes missed violations in split namespace installations. Instead of extracting a single AGENT_EVAL_ROOT, refactor the code to iterate through all entries in agent_eval.__path__ and check each namespace root for import hygiene violations. Apply this same pattern to both the initial AGENT_EVAL_ROOT assignment on line 20 and the related code section mentioned in lines 32-37.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/README.md`:
- Around line 53-61: The markdown file has two linting issues that need to be
fixed. First, add a language identifier (such as json or text) after the opening
backticks for the fenced code blocks at lines 53 and 178 that show the agent
evaluation output and configuration examples respectively. Second, locate the
blockquote section around line 194 and remove the blank line that appears within
or between blockquote elements, as markdown linting requires proper blockquote
spacing without isolated blank lines.
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py`:
- Around line 107-110: The validation for requiring --aut-agent-name when
--backend aut is specified is happening at runtime in the conditional block with
NatAutRuntime instantiation, causing an unhandled ValueError. Move this
validation to the argument parser setup using a custom validation method or
action that checks the mutually dependent arguments (backend and aut_agent_name)
during parse time rather than after, so that argparse can handle the error with
a proper controlled exit message instead of a traceback.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/common_metrics.py`:
- Around line 40-41: The bool() coercion on the metadata.get("agent_ok") value
at line 40 treats any non-empty string (including "false") as True, which
incorrectly marks failed trials as passed. Replace the truthy coercion with
proper boolean conversion that correctly handles string representations of
booleans. Instead of bool(input.candidate.metadata.get("agent_ok")), implement
logic that checks if the value is explicitly True or a string "true"
(case-insensitive), returning False for any other value including the string
"false" or absence of the key.
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/gating.py`:
- Around line 422-429: The `_aggregate_provenance` function attempts to add
provenance field values directly to sets using `.add(value)`, which will raise a
TypeError when the value is an unhashable type like a list or dict. Modify the
code where `observed[field_name].add(value)` is called to handle unhashable
values by either converting them to hashable equivalents (e.g., converting lists
to tuples, dicts to frozensets or their string representations) or wrapping the
add operation in a try-except block to gracefully skip unhashable values. This
ensures the aggregation can complete successfully without aborting summarization
and gate report generation.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/build_spec.py`:
- Around line 169-170: The code at the dependency injection point in the RUN pip
install command is vulnerable to shell command injection because the dependency
strings from spec.python_dependencies are joined and directly interpolated into
a shell command without proper escaping. To fix this, properly escape or quote
each individual dependency value from spec.python_dependencies before joining
them into the deps string, ensuring that any shell metacharacters in the
dependency names cannot be interpreted as commands during Docker image build.
Alternatively, use the exec form of the RUN instruction with a JSON array syntax
to bypass shell interpretation entirely, passing each dependency as a separate
list element.
- Around line 90-98: The python_dependencies parsing in the BuildSpec creation
does not validate that python_deps is actually a list before converting it with
list(). When python_deps is a string like "pytest", list(python_deps) converts
it to individual characters instead of a list containing the single dependency.
Add validation to ensure python_deps is a list before using it in the BuildSpec
constructor. If python_deps is not a list (such as when it's a string), either
raise a validation error to reject the invalid YAML structure or handle the
string case appropriately to produce correct dependency lists. This ensures only
properly formatted dependencies are accepted.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/docker.py`:
- Around line 18-32: The redact_cmd_for_logging function only redacts values in
the KEY=value format but misses sensitive flag patterns like --token secret or
--password secret where the secret value is a separate argument. Enhance the
function to detect sensitive flags (tokens containing KEY, TOKEN, SECRET, or
PASSWORD markers) even without the equals sign, and when such a flag is found,
redact the following token in the command sequence. This will require iterating
through the cmd sequence with index access so you can both identify sensitive
flags and mark the subsequent argument as redacted.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/environment.py`:
- Around line 27-29: The default_image_tag function uses the raw task_id
parameter directly without sanitization, but Docker image tags have character
restrictions and invalid characters will cause image build and runtime failures.
Sanitize the task_id before constructing the tag by converting it to lowercase,
replacing or removing invalid Docker tag characters (such as spaces and special
characters that are not alphanumeric, hyphens, underscores, or periods),
ensuring the resulting tag conforms to Docker naming conventions.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/generalized_agent.py`:
- Around line 124-136: The subprocess created by `self._process_factory` is not
explicitly killed when the `asyncio.wait_for` call times out, which can leave
zombie processes running. Add a finally block after the try-except to ensure the
subprocess is properly terminated by calling `process.kill()` regardless of
whether the operation succeeds or fails. Alternatively, add explicit cleanup in
the exception handler to terminate the process before returning from the
`_failed_trial` call, ensuring the process object has been created before
attempting to kill it.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/verify.py`:
- Around line 51-63: The code creates an inconsistency where the reward value
(read from reward_path file) can contradict the passed status (derived from the
ok variable). When reward.txt contains "1" but ok is False, the metadata becomes
internally inconsistent with passed=False but reward=1. To fix this, ensure that
the reward and passed values are always consistent by having the reward value
determine or validate the passed status, rather than allowing them to diverge.
Apply this consistency check throughout the verification logic, including all
code sections mentioned (lines 51-63 and also applies to 76-84) where this
pattern occurs.
---
Outside diff comments:
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/README.md`:
- Around line 1-227: The README.md mixes tutorial content (Run it), how-to
guides (Plugging in a real workflow, Running a real tests/agentic-use task),
explanation (What it demonstrates, Execution flow), and reference material
(Files table) without clear separation, and places prerequisites late in the
document. Reorganize the content by moving the prerequisites section to the very
top before "Run it", create distinct sections aligned to Diataxis quadrants
(Tutorial for "Run it", How-To for the real workflow sections, Explanation for
"What it demonstrates" and execution flow, Reference for "Files" and
configuration tables), and add a "Next Steps" section at the end with
cross-links to related documentation. Each reorganized section should stand
independently and reference other sections via cross-links rather than embedding
mixed content types together.
---
Nitpick comments:
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/aut_runtime.py`:
- Line 169: The import statement for `inject_gateway_url` from
`nemo_agents_plugin.utils` is currently inside the
`prepare_aut_config_for_runtime` function body at line 169. Move this import to
the module-level imports at the top of the file with the other import
statements, then remove the import statement from inside the function body.
- Around line 72-75: The API key interpolation in the inline Python script uses
simple string replacement with the replace() method for NVIDIA_API_KEY and
ANTHROPIC_API_KEY placeholders. This approach is fragile and will break if API
keys contain special characters like quotes or backslashes, potentially creating
malformed YAML. Instead of using str.replace() to substitute these placeholders,
load the YAML file using a YAML library like PyYAML, update the parsed
dictionary with the actual API key values, and then serialize it back to YAML
format. This ensures proper escaping and quoting of special characters in the
API keys.
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py`:
- Around line 253-264: The current implementation uses brittle string
replacements (for "http://localhost:8080", model name, and MCP client function
groups) that will silently fail if the YAML has different formatting like quotes
or spacing. Replace these string-based replacements with a proper YAML parsing
approach: parse the workflow_path content using a YAML parser to convert it to a
Python dictionary, modify the relevant fields in the data structure
(nmp_base_url under appropriate keys, model_name field, and function_groups vs
functions), then serialize the modified structure back to YAML text. This
approach is more robust and maintainable, and aligns with how the tracing config
is already handled later in the function.
- Around line 546-554: The run_tasks method processes tasks sequentially using a
simple for loop with await, which limits throughput. Refactor the task execution
to use concurrent processing with asyncio.Semaphore (similar to
WorkflowAgentRuntime) to allow multiple tasks to run in parallel while still
respecting resource constraints. Replace the sequential loop that awaits
self._run_task for each task with either asyncio.gather or asyncio.TaskGroup
along with a Semaphore to control concurrency. Alternatively, if sequential
execution is intentional due to constraints like Docker resource limitations,
add a docstring comment to the run_tasks method documenting this design
decision.
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.py`:
- Line 20: The current implementation using next(iter(agent_eval.__path__)) only
examines the first path entry in the namespace, which causes missed violations
in split namespace installations. Instead of extracting a single
AGENT_EVAL_ROOT, refactor the code to iterate through all entries in
agent_eval.__path__ and check each namespace root for import hygiene violations.
Apply this same pattern to both the initial AGENT_EVAL_ROOT assignment on line
20 and the related code section mentioned in lines 32-37.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 99bd2b0a-e2dd-4984-9347-897b60511c0e
⛔ Files ignored due to path filters (12)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/common_metrics.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/gating.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/measurements.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/pipeline.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/build_spec.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/docker.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/environment.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/generalized_agent.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/layout.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/verify.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trial_artifacts.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trials.pyis excluded by!sdk/**
📒 Files selected for processing (30)
packages/nemo_evaluator_sdk/examples/run_agent_eval/.gitignorepackages/nemo_evaluator_sdk/examples/run_agent_eval/README.mdpackages/nemo_evaluator_sdk/examples/run_agent_eval/aut_agent.workspace.example.ymlpackages/nemo_evaluator_sdk/examples/run_agent_eval/aut_runtime.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/mini_agent.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/usage.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/common_metrics.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/gating.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/measurements.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/pipeline.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/build_spec.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/docker.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/environment.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/generalized_agent.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/layout.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/verify.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trial_artifacts.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_common_metrics.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_environment.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_gating.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_generalized_agent.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_measurements.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_pipeline.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_trial_artifacts.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_verify.py
arpitsardhana
force-pushed
the
aalgo-258-nat-runner/arpsingh
branch
from
June 22, 2026 20:42
9d6381d to
6ec07e2
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py`:
- Around line 98-107: The subprocess created by create_subprocess_exec at line
99 is not being terminated when asyncio.wait_for times out at line 104, allowing
orphaned processes to continue running. Modify the exception handling to
specifically catch asyncio.TimeoutError before the general Exception handler,
and when a timeout occurs, call process.kill() on the subprocess object to
ensure it is terminated before returning the failed trial from line 107. This
ensures proper cleanup of the subprocess on timeout, similar to the fix
implemented in generalized_agent.py.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9afab548-cd2c-4b8d-b87c-e004cf2981b2
⛔ Files ignored due to path filters (12)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/common_metrics.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/gating.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/measurements.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/pipeline.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/build_spec.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/docker.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/environment.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/generalized_agent.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/layout.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/verify.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trial_artifacts.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trials.pyis excluded by!sdk/**
📒 Files selected for processing (30)
packages/nemo_evaluator_sdk/examples/run_agent_eval/.gitignorepackages/nemo_evaluator_sdk/examples/run_agent_eval/README.mdpackages/nemo_evaluator_sdk/examples/run_agent_eval/aut_agent.workspace.example.ymlpackages/nemo_evaluator_sdk/examples/run_agent_eval/aut_runtime.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/mini_agent.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/usage.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/common_metrics.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/gating.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/measurements.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/pipeline.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/build_spec.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/docker.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/environment.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/generalized_agent.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/layout.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/verify.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trial_artifacts.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_common_metrics.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_environment.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_gating.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_generalized_agent.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_measurements.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_pipeline.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_trial_artifacts.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_verify.py
✅ Files skipped from review due to trivial changes (6)
- packages/nemo_evaluator_sdk/examples/run_agent_eval/README.md
- packages/nemo_evaluator_sdk/examples/run_agent_eval/aut_agent.workspace.example.yml
- packages/nemo_evaluator_sdk/tests/agent_eval/test_common_metrics.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_trial_artifacts.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/.gitignore
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/pipeline.py
🚧 Files skipped from review as they are similar to previous changes (17)
- packages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trial_artifacts.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/measurements.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_verify.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/layout.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_generalized_agent.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/mini_agent.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/common_metrics.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_gating.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/environment.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_measurements.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/verify.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/usage.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_environment.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/build_spec.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py
Contributor
|
arpitsardhana
force-pushed
the
aalgo-258-nat-runner/arpsingh
branch
from
June 23, 2026 00:36
19af066 to
4825d87
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/build_spec.py`:
- Around line 178-180: The comment at line 178-179 provides incorrect rationale
for why `<` and `>` are excluded from `_DOCKERFILE_UNSAFE_CHARS`. The default
RUN form does process shell redirections via `/bin/sh -c`, so these characters
ARE unsafe and must be included in the set. Fix this by adding `<` and `>` to
the character set in `_DOCKERFILE_UNSAFE_CHARS` (change `set('\n\r"`$;&|')` to
include them), and remove or replace the incorrect comment that claims they are
intentionally allowed. The comment should reflect that these characters are
unsafe because they enable shell redirections that break version pins and enable
file overwrites, so they must be quoted instead.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: fe2d00eb-a5e5-4ffe-ba9d-447c0dc72e22
⛔ Files ignored due to path filters (2)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/measurements.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trial_artifacts.pyis excluded by!sdk/**
📒 Files selected for processing (16)
packages/nemo_evaluator_sdk/examples/run_agent_eval/README.mdpackages/nemo_evaluator_sdk/examples/run_agent_eval/build_spec.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/gating.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/layout.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/verify.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/measurements.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trial_artifacts.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_environment.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_gating.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_measurements.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_pipeline.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_verify.py
💤 Files with no reviewable changes (1)
- packages/nemo_evaluator_sdk/examples/run_agent_eval/layout.py
✅ Files skipped from review due to trivial changes (1)
- packages/nemo_evaluator_sdk/examples/run_agent_eval/README.md
🚧 Files skipped from review as they are similar to previous changes (13)
- packages/nemo_evaluator_sdk/tests/agent_eval/test_measurements.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_verify.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trial_artifacts.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/verify.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/measurements.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_pipeline.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/gating.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_environment.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_gating.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py
arpitsardhana
force-pushed
the
aalgo-258-nat-runner/arpsingh
branch
from
June 23, 2026 01:36
4825d87 to
592f1e5
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py (1)
104-112: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winSubprocess still orphaned on timeout.
asyncio.wait_fortimeout leaves thecreate_subprocess_execchild running; nothing kills it before returning the failed trial.Proposed fix
+ process = None try: process = await asyncio.create_subprocess_exec( *command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=self.config.timeout_s) except Exception as exc: # noqa: BLE001 - any launch/timeout failure is a trial-production failure. + if process is not None: + try: + process.kill() + await process.wait() + except Exception: + pass return self._failed_trial(task, layout.run_dir, exc)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py` around lines 104 - 112, The subprocess created by asyncio.create_subprocess_exec is not being terminated when asyncio.wait_for times out, leaving the child process orphaned. In the exception handler that catches the timeout failure and calls self._failed_trial, add a call to terminate and/or kill the process object (using process.kill() and potentially awaiting process.wait() to ensure it exits) before returning the failed trial, so the subprocess is properly cleaned up regardless of whether the timeout or another exception occurs.packages/nemo_evaluator_sdk/examples/run_agent_eval/build_spec.py (2)
89-90: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winFail fast when
environment.dependenciesis not a mapping.Line 89 and Line 90 silently drop dependencies when
dependenciesis not a dict. That should raise a validation error instead of continuing with an incomplete build spec.Proposed fix
- dependencies = data.get("dependencies") or {} - python_deps = dependencies.get("python") if isinstance(dependencies, dict) else None + dependencies_raw = data.get("dependencies") + if dependencies_raw is None: + python_deps = None + elif isinstance(dependencies_raw, dict): + python_deps = dependencies_raw.get("python") + else: + raise ValueError(f"Invalid build spec in {task_dir}: 'dependencies' must be a mapping")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/build_spec.py` around lines 89 - 90, The code at lines 89-90 silently handles the case where dependencies is not a dictionary by setting python_deps to None, which causes incomplete build specs to proceed without error. Instead of checking isinstance(dependencies, dict) and continuing with None, add validation that raises an error when dependencies is not a dictionary or mapping type. Specifically, after retrieving dependencies from data at line 89, validate that if dependencies is not None and not empty, it must be a dict, and raise an appropriate validation error if this constraint is violated to fail fast rather than silently dropping the dependencies.
179-181: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winQuote dependency tokens before generating
RUN pip install.Line 207 feeds raw tokens into shell-form
RUNat Line 208.numpy<2is parsed as shell redirection, so valid pins break and redirection side effects are possible.Proposed fix
+import shlex @@ -# Note: '<'/'>' are intentionally allowed — pip version pins (``numpy<2``, -# ``pytest>=8``) use them and a Dockerfile ``RUN`` arg doesn't shell-redirect. +# Note: RUN shell form is interpreted by a shell; dependency tokens must be quoted. _DOCKERFILE_UNSAFE_CHARS = set('\n\r"`$;&|') @@ if spec.python_dependencies: - deps = " ".join(spec.python_dependencies) + deps = " ".join(shlex.quote(dep) for dep in spec.python_dependencies) lines.append(f"RUN pip install --no-cache-dir {deps}")In Dockerfile shell-form RUN, is `pip install numpy<2` parsed using shell redirection semantics, and is quoting each dependency token (e.g., via shlex.quote) the correct mitigation?Also applies to: 207-208
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/build_spec.py` around lines 179 - 181, The RUN pip install command being generated at line 208 uses raw dependency tokens from line 207 without shell quoting, which causes shell interpretation of special characters like '<' and '>' as redirection operators rather than part of version specifiers (e.g., numpy<2 gets parsed as redirection). Apply shlex.quote to each dependency token before building the RUN command string to properly escape shell special characters and prevent shell redirection side effects.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py`:
- Around line 159-163: The ATIF format detection on line 162 uses
trace_name.startswith("atif") which is too restrictive and misclassifies files
like "trace.atif.json" as JSON. Instead of checking if the trace_name starts
with "atif", check if the name contains or ends with ".atif" extension (e.g.,
using endswith(".atif") or checking if ".atif" is in the filename) to properly
detect ATIF traces regardless of their complete filename pattern.
---
Duplicate comments:
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/build_spec.py`:
- Around line 89-90: The code at lines 89-90 silently handles the case where
dependencies is not a dictionary by setting python_deps to None, which causes
incomplete build specs to proceed without error. Instead of checking
isinstance(dependencies, dict) and continuing with None, add validation that
raises an error when dependencies is not a dictionary or mapping type.
Specifically, after retrieving dependencies from data at line 89, validate that
if dependencies is not None and not empty, it must be a dict, and raise an
appropriate validation error if this constraint is violated to fail fast rather
than silently dropping the dependencies.
- Around line 179-181: The RUN pip install command being generated at line 208
uses raw dependency tokens from line 207 without shell quoting, which causes
shell interpretation of special characters like '<' and '>' as redirection
operators rather than part of version specifiers (e.g., numpy<2 gets parsed as
redirection). Apply shlex.quote to each dependency token before building the RUN
command string to properly escape shell special characters and prevent shell
redirection side effects.
In `@packages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.py`:
- Around line 104-112: The subprocess created by asyncio.create_subprocess_exec
is not being terminated when asyncio.wait_for times out, leaving the child
process orphaned. In the exception handler that catches the timeout failure and
calls self._failed_trial, add a call to terminate and/or kill the process object
(using process.kill() and potentially awaiting process.wait() to ensure it
exits) before returning the failed trial, so the subprocess is properly cleaned
up regardless of whether the timeout or another exception occurs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 273b6d09-cca3-400f-b84b-3dfb227c2337
⛔ Files ignored due to path filters (3)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/metrics.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/environment.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trials.pyis excluded by!sdk/**
📒 Files selected for processing (22)
packages/nemo_evaluator_sdk/examples/run_agent_eval/.gitignorepackages/nemo_evaluator_sdk/examples/run_agent_eval/README.mdpackages/nemo_evaluator_sdk/examples/run_agent_eval/aut_agent.workspace.example.ymlpackages/nemo_evaluator_sdk/examples/run_agent_eval/aut_runtime.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/build_spec.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/gating.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/layout.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/mini_agent.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/platform_runtime.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/usage.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/verify.pypackages/nemo_evaluator_sdk/examples/run_agent_eval/workflow_runtime.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/metrics.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/environment.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_environment.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_metrics.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_profbench.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_trials.py
💤 Files with no reviewable changes (1)
- packages/nemo_evaluator_sdk/tests/agent_eval/test_profbench.py
✅ Files skipped from review due to trivial changes (3)
- packages/nemo_evaluator_sdk/examples/run_agent_eval/.gitignore
- packages/nemo_evaluator_sdk/examples/run_agent_eval/README.md
- packages/nemo_evaluator_sdk/examples/run_agent_eval/verify.py
🚧 Files skipped from review as they are similar to previous changes (7)
- packages/nemo_evaluator_sdk/tests/agent_eval/test_import_hygiene.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/mini_agent.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/pipeline.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/aut_agent.workspace.example.yml
- packages/nemo_evaluator_sdk/examples/run_agent_eval/run_agent_eval.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/layout.py
- packages/nemo_evaluator_sdk/examples/run_agent_eval/usage.py
arpitsardhana
force-pushed
the
aalgo-258-nat-runner/arpsingh
branch
from
June 23, 2026 05:39
592f1e5 to
fe07759
Compare
SandyChapman
approved these changes
Jun 23, 2026
SandyChapman
left a comment
SandyChapman
left a comment
Contributor
There was a problem hiding this comment.
lgtm! thanks for the edits
…xample Add a lean, trials-based agent-evaluation core under nemo_evaluator_sdk.agent_eval and a self-contained run_agent_eval example that drives it. SDK core owns the invariant spine only: AgentEvaluator.run generates or imports trials, scores each trial x metric, builds a summary, and persists the run bundle (trials.jsonl, scores, report.html). Supporting modules cover trial/evidence/target primitives (trials.py), reusable metrics and trial measurements (metrics.py), results/scores models, a pluggable agent execution environment (runtimes/environment.py), persistence, and the dashboard. The example owns task definitions, the runtime/target adapters (toy host subprocess, Docker-backed nat run workflow, and a deployed agent-under-test), evidence layout, the deterministic gate policy, build specs, and CLI/printing. It supports an online path, a real tests/agentic-use BUILD -> AGENT -> VERIFY path, and offline rescoring of a prior run's trials. Includes import-hygiene guardrail tests and aligned unit tests covering the SDK modules. Signed-off-by: Arpit Singh (SW-CLOUD) <arpsingh@nvidia.com>
arpitsardhana
force-pushed
the
aalgo-258-nat-runner/arpsingh
branch
from
June 23, 2026 19:45
fe07759 to
f2082b8
Compare
arpitsardhana
enabled auto-merge
June 23, 2026 19:55
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Promotes a small set of generic agent-evaluation primitives into the Agent-Eval SDK, and ships a runnable
run_agent_evalexample that drives the full build env → run agent → verify → score → gate → persist flow end to end. The SDK stays deliberately lean — it owns generating trials and scoring them; CI policy (the pass/fail gate), the pipeline wrapper, the Harbor-style verifier, the run layout, and all Docker/build glue live in the example.Agent-Eval SDK (
nemo_evaluator_sdk/agent_eval/)Three focused additions, no speculative surface:
runtimes/environment.py— a pluggable agent-execution seam that sits belowAgentTaskRunner:AgentEnvironmentProvider/AgentEnvironmentHandleprotocols, anAbstractEnvironmentHandlethat routesrun_agent/run_verifierthrough a singlerun(spec, role), theEnvRunSpec/EnvCommandResultvalue types, and a Docker reference implementation (DockerEnvironmentProvider/DockerEnvironmentHandle, stdlib-subprocessdocker runwith secret redaction).metrics.py— reusable scorers plus the measurement contract:AgentPhaseSuccessMetric(reads the agent-phase outcome) andEvidencePresenceMetric(scores overcandidate.evidenceon disk rather than a stamped reward — the SDK's metric-over-evidence value-add), andTrialMeasurements, the typed projection of the token/runtime/reward keys producers write ontotrial.metadata.trials.py(extended) — adds theAgentTrialSerdeprotocol (the offline counterpart toAgentTaskRunner, so prior runs can be re-scored), and the runtime-agnostic trial-shaping helpersresolve_trial_statusandstandard_evidence_descriptors(the documented evidence-key builder).beta/evaluator/agent_eval/) regenerated to match.run_agent_evalexample (nemo_evaluator_sdk/examples/run_agent_eval/)A self-contained example that drives agent-eval tasks through the SDK with two paths:
mini_agent.pyruns as a host subprocess viaworkflow_runtime.py(anAgentTaskRunner+TrialJsonSerde+ example tasks/metric), so it runs end to end with no external infrastructure.--agentic-task <name>) — runs an actualtests/agentic-usetask through Docker with two backends:platform_runtime.py) — task-localnat runofworkflow.yml, with BUILD (image) / AGENT (nat run) / VERIFY (pytest) phases.aut_runtime.py) — create → seed inference providers → deploy → health-check → invoke a deployed agent-under-test.build_spec.py(environment.yaml→BuildPlan, with a Dockerfile escape hatch),layout.py(run-dir scaffold),verify.py(Harborreward.txtconvention),gating.py(evaluate_gate/GateThresholds→gate.json),pipeline.py(online run + offline rescore + gate wrapper), andusage.py(token/runtime extraction onto each trial).run_agent_eval.pyis the CLI harness (online run, offline rescore, gate); ships a task-capableaut_agent.workspace.example.ymlmatched toworkspace-basic-mcp(text-based ReAct so it works across whichever backend the gateway routes to).Tests (
tests/agent_eval/mirrors the SDK package 1:1)test_environment.py,test_metrics.py, andtest_trials.py(incl. the trial-shaping helpers) cover the new SDK code;test_import_hygiene.pykeepsagent_eval/free of NeMo-Platform imports.test_profbench.py.Test plan
uv run pytest packages/nemo_evaluator_sdk/tests/agent_eval/→ green (55 passed)uv run ruff check/ruff format --check/uv run ty checkon the new code → cleanmake vendor→ SDK mirror in sync (no drift)run_agent_evaltoy path (defaultmini_agent, no external infra) end to end, plus offline rescore of the same bundle:run_agent_evalreal path (AUT backend) verified end to end onworkspace-basic-mcp: