feat(experimentalist): add a fast, diagnostic smoke agent fixture

[schuellc-nvidia]

Summary

Adds a deterministic smoke-agent fixture for the Experimentalist. Unlike the
existing long-running examples, it has deliberate, known failures and
deterministic task results, so a run can verify that the Experimentalist repaired
the measured problem rather than merely completed.

The fixture agent makes no model calls. It uses regex handlers over a small local
records file; the Experimentalist is the only stochastic component.

Changes

• Adds five deliberately weak capability groups: aggregation, name matching,
  long inputs, dispatch ordering, and edge cases.
• Adds host-side Mode 2 E2E coverage for the four repair groups plus G4's
  generalization case, each from an isolated fixture copy and experiment
  directory.
• Adds Mode 1 E2E coverage for the repair groups. Each test records its failing
  train traces, uploads them to Intake, writes a deterministic mocked Insight
  containing only those trace references, then exercises Eval Author and the
  insight-driven Experimentalist loop.
• Replaces the separate artifact gate test with direct assertions in the Mode 1
  and Mode 2 E2E tests.
• Stops the short fixture configuration once reward and shape_ok reach their
  known target of 1.0.
• Keeps full-suite test collection independent of pytest's global conftest
  import order and removes stale Eval Author test dependencies.

Scenario expectations

Scenario	Healthy result
G1, G2, G3, G5 repair	a changed winner passes the committed validation tasks
G4 generalization	the baseline remains selected because a narrow train fix fails validation

Validation

• Targeted Ruff check and format check pass.
• Targeted ty check passes for the repaired Experimentalist tests.
• Eval Author and Experimentalist regression tests pass: 12 passed.
• Both smoke-agent E2E modules collect successfully together under CI=1.
• Full pre-commit's Ruff, format, type, config-doc, lock-drift, and copyright
  hooks pass. The local full gate is otherwise blocked by missing helm-docs
  and yq, and by local uv 0.12.1 differing from the repository-pinned 0.9.14.
• Mode 1 and Mode 2 E2E tests require Docker, a running local Platform, and
  configured real model entities; they are opt-in via SMOKE_AGENT_E2E=1.

Known Mode 1 finding

Mode 1 currently exposes an Eval Author defect: it writes the authored metric
fragment successfully, but emits a malformed Python merge block in the generated
verifier. Harbor therefore records only reward and shape_ok. The E2E suite
keeps this visible rather than treating the run as a healthy optimization.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The pull request adds a deterministic, offline smoke-agent fixture with strict output contracts, five weakness-focused dataset groups, Docker and trace tooling, synchronized verifiers, scenario profiles, and extensive baseline, asset, repair, generalization, and trace tests.

Changes

Smoke-agent benchmark fixture

Layer / File(s)	Summary
Agent runtime and execution contracts examples/smoke-agent/agent/*, examples/smoke-agent/AGENT-SPEC.md, examples/smoke-agent/dataset/_shared/*	Adds deterministic record lookup, list, and count handling; Harbor execution; exact artifact output; OTLP tracing; and offline runtime constraints.
Task templates and benchmark datasets examples/smoke-agent/dataset/task-template/*, examples/smoke-agent/dataset/groups/*	Adds task templates and five groups covering aggregation, name patterns, long inputs, dispatch order, and edge cases.
Verifier and asset integrity checks examples/smoke-agent/dataset/_shared/test.sh, examples/smoke-agent/dataset/groups/*/tests/test.sh, tests/experimentalist/test_smoke_agent_assets.py	Adds fail-closed scoring, CRLF-normalized exact comparison, shape metrics, symlink protection, synchronized verifiers, Docker pin checks, and combined-dataset consistency checks.
Dataset and trace tooling examples/smoke-agent/scripts/*, examples/smoke-agent/.gitignore	Adds combined-group assembly, content-addressed image stamping, verifier synchronization, trace upload and retrieval, and generated-directory handling.
Profiles, documentation, and behavioral gates examples/smoke-agent/README.md, docs/smoke-agent-weaknesses.md, examples/smoke-agent/configs/*, examples/smoke-agent/optimizer*.yaml, tests/experimentalist/test_smoke_agent*.py	Documents fixture constraints and scenarios. Adds baseline, behavioral, trace, repair, and generalization checks.৮

Suggested reviewers: aleckhoury, sandychapman, arpitsardhana

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the addition of the diagnostic Experimentalist smoke-agent fixture.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch experimentalist-smoke-agent-proposal/cschueller

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 20

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md (1)

1-10: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Add required SPDX metadata to all fixture files.

These files lack both required SPDX identifiers. Update the raw-fixture format and verifier handling so metadata does not become part of the expected agent output.

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md#L1-L10: add SPDX metadata.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt#L1-L1: make the expected-output format SPDX-compatible.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md#L1-L10: add SPDX metadata.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt#L1-L1: make the expected-output format SPDX-compatible.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md#L1-L10: add SPDX metadata.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt#L1-L1: make the expected-output format SPDX-compatible.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md#L1-L10: add SPDX metadata.

As per coding guidelines, “Every file must contain the NVIDIA SPDX copyright header and Apache-2.0 license identifier.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md`
around lines 1 - 10, Add the standard NVIDIA SPDX copyright header and
Apache-2.0 license identifier to every listed instruction.md and expected.txt
fixture, updating the raw-fixture format and verifier handling so these metadata
lines are excluded from expected agent output. Apply the changes to all seven
specified files, preserving each instruction and expected result content.

Source: Coding guidelines

🧹 Nitpick comments (1)

plugins/nemo-experimentalist/examples/smoke-agent/README.md (1)

15-140: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Split the documentation by Diataxis type.

This README combines explanation, reference, how-to steps, and timings. Keep this page as a HOW-TO, then move design, layout, and timings to linked EXPLANATION or REFERENCE pages.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md` around lines 15
- 140, The smoke-agent README currently mixes how-to guidance with
explanation/reference content and timings. Keep the README focused on the
run/check workflow, and move the Design, Layout, Scenarios, and Timings sections
into separate linked pages classified as EXPLANATION or REFERENCE. Update the
existing section anchors and links so the README still points to the relocated
content using the visible README sections as the split points.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md`:
- Around line 4-16: Update the documentation page’s front matter or opening
metadata to identify it as an EXPLANATION, then add a prerequisites section
before the existing overview. Append a Next Steps section at the end linking to
the smoke-agent implementation and its guard tests, preserving the current
weakness documentation.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yaml`:
- Around line 17-25: Select a distinct held-out dataset for the generalization
scenario in the configuration around max_train_batch_tasks, using a separate
profile or supported dataset/config override rather than inheriting
optimizer.yaml’s g1-aggregation settings. Ensure the documented command resolves
to this generalization dataset instead of the repair scenario, and update the
corresponding README scenario description to match.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile`:
- Around line 16-17: Replace the pip-based installation in the Dockerfile
dependency layer with the project’s uv-based dependency-install flow, preserving
the existing nooa[tracing] Git dependency and pinned revision. Do not introduce
or retain pip usage.

In `@plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh`:
- Around line 31-46: Reject symlinked output artifacts before scoring in both
plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh lines
31-46 and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
lines 31-46. Update the scoring flow around the OUTPUT file check so validation
uses an isolated finalized artifact or race-safe no-follow access, preventing [
-f ], head, sed, and cmp from following an agent-controlled symlink; preserve
the existing shape and byte-for-byte comparison behavior for regular files.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.md`:
- Around line 1-9: Every listed instruction.md and tests/expected.txt file must
include the NVIDIA SPDX copyright header and Apache-2.0 license identifier. Add
headers to the six specified files while preserving each instruction’s task
contract and expected output; update the expected-output parsing/verifier as
needed so the added metadata is ignored and comparisons remain unchanged.
Affected files:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.md
(lines 1-9), train/lookup-ada/tests/expected.txt (line 1),
validation/total-hours-analysts/instruction.md (lines 1-9),
validation/total-hours-analysts/tests/expected.txt (line 1),
validation/total-hours-ops/instruction.md (lines 1-9), and
validation/total-hours-ops/tests/expected.txt (line 1).

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.sh`:
- Around line 35-36: Update the shape_ok assignment checks in all three listed
test.sh files at lines 35-36 to require at least one non-whitespace character
after the equals sign, so entries such as total= or dept= are rejected while
populated values remain accepted.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/instruction.md`:
- Around line 1-9: Resolve SPDX compliance across all six listed fixture files:
add the NVIDIA copyright header and Apache-2.0 SPDX identifier to the three
instruction.md files, while preserving their task content. For each listed
expected.txt file, replace the raw byte-exact format with the verifier’s
supported header-bearing fixture format, or add and enforce an explicit
exemption without inserting header text into the existing payload.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.md`:
- Around line 1-9: Resolve SPDX compliance through the approved fixture-policy
mechanism rather than modifying fixture contents: apply the prompt-fixture
exception to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.md
lines 1-9,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/instruction.md
lines 1-9, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/instruction.md
lines 1-9. Apply the approved literal-artifact exception to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
line 1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
line 1, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
line 1; preserve all prompt and exact expected-output text unchanged while
ensuring the policy recognizes the NVIDIA copyright and Apache-2.0 licensing
requirement.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.md`:
- Line 1: Resolve missing SPDX metadata across the listed fixtures: add the
NVIDIA copyright and Apache-2.0 identifiers to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.md#L1-L1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/instruction.md#L1-L1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.md#L1-L1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/instruction.md#L1-L1;
apply the approved non-semantic exact-output-fixture metadata mechanism to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt#L1-L1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt#L1-L1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt#L1-L1
without inserting literal headers or changing verifier bytes.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt`:
- Line 1: Add the required NVIDIA SPDX copyright and Apache-2.0 license metadata
to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt:1-1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt:1-1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt:1-1
without altering the compared payload; update the exact-output fixture format or
verifier so these headers are ignored during comparison. Add the same required
SPDX header to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md:1-9
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md:1-9.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt`:
- Line 1: Document an approved SPDX licensing exception or metadata rule for all
affected content-sensitive dataset fixtures, without modifying their bytes or
task behavior. Apply this to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt:1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md:1-9
and tests/expected.txt:1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md:1-9
and tests/expected.txt:1, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md:1-9;
do not prepend SPDX headers or otherwise alter fixture contents.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.md`:
- Line 1: Add the standard NVIDIA SPDX copyright and Apache-2.0 license
identifiers to the Markdown instruction files at
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.md:1-1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/instruction.md:1-1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md:1-1.
Update the expected-output fixtures at
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt:1-1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt:1-1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt:1-1
to use the licensed fixture representation, and update the canonical verifier to
ignore the added metadata while preserving expected-output semantics.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh`:
- Around line 29-49: Add symlink and same-inode rejection before the output
scoring branch in each test.sh:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
lines 29-49,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.sh
lines 29-49, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
lines 29-49. Before the existing [ -f "$OUTPUT" ] check, exit successfully after
reporting a symbolic-link OUTPUT or when stat confirms OUTPUT and EXPECTED_FILE
share an inode, while guarding inode comparison on EXPECTED_FILE readability.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt`:
- Line 1: Update the affected fixtures and instructions to include the required
NVIDIA SPDX/Apache-2.0 header while keeping the expected payloads unchanged, and
adjust each listed verifier to strip or ignore license-header lines before
comparing output. In the g4-dispatch-order lookup-ada test flow, update
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
and the matching tests/test.sh comparison logic; do the same for the
g5-edge-cases missing-person-hours and missing-person-role test fixtures and
test.sh scripts. Also add Markdown-compatible SPDX identifiers to the two
instruction.md files so the task text remains valid while satisfying the
file-header requirement.

In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 61-79: Update the README commands around scripts/build_image.py
and the scenario execution to use the required uv environment: invoke the build
script through uv run, and remove the UV_PROJECT_ENVIRONMENT override or set it
to the repository root .venv. Keep the existing package and plugin arguments
unchanged.

In
`@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py`:
- Around line 41-74: Strengthen test_verifier_emits_exactly_the_two_metric_keys
and test_verifier_keeps_its_reward_hacking_guards to validate complete
invariants rather than substring fragments: parse the verifier’s emitted metric
keys and require exactly reward and shape_ok, and require a set command
containing both -u and pipefail while still rejecting -e. Use _verifier_code to
ignore comments and preserve the existing checks for tr -d, cmp -s, and refusing
to score.

In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py`:
- Around line 105-112: Update the winner validation in the smoke test around
_winner_label() and _agent_source() so it no longer treats the presence of
“sum(” as aggregation evidence. Instead, verify a changed execution path that
sums the relevant record field for total-hours support, or replay a total-hours
input against the winning agent and assert the expected result.
- Around line 195-220: Update test_g_2_a_candidate_actually_fixed_train to
retain the best train-improving candidate’s identity, then compare that
candidate’s validation reward from _validation_aggregate against agent-0’s
validation reward. Assert the candidate does not beat the baseline while
preserving the existing requirement that its train reward is higher.

In
`@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py`:
- Around line 30-41: Update test_trace_carries_the_method_call_graph to select a
recorded trace containing the G4 call path, rather than assuming _traces()[0] is
suitable. Search available traces for one whose method counts include
ReportAgent.handle_list, or use a dedicated known G4 trace, while preserving the
existing solve and handle_list assertions.

In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py`:
- Around line 106-132: Update _LEAK_SCAN_SKIP_DIRS in
test_example_directory_does_not_leak_the_weaknesses to match
_AGENT_COPY_EXCLUDE_NAMES exactly, including traces and the other
copier-excluded directories such as .git, .claude, .uv, eval-and-optimize, and
scratch; remove any exclusions not present in that copier set so the leak scan
covers precisely the files available to the Coder.

---

Outside diff comments:
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md`:
- Around line 1-10: Add the standard NVIDIA SPDX copyright header and Apache-2.0
license identifier to every listed instruction.md and expected.txt fixture,
updating the raw-fixture format and verifier handling so these metadata lines
are excluded from expected agent output. Apply the changes to all seven
specified files, preserving each instruction and expected result content.

---

Nitpick comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 15-140: The smoke-agent README currently mixes how-to guidance
with explanation/reference content and timings. Keep the README focused on the
run/check workflow, and move the Design, Layout, Scenarios, and Timings sections
into separate linked pages classified as EXPLANATION or REFERENCE. Update the
existing section anchors and links so the README still points to the relocated
content using the visible README sections as the split points.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7c11a54a-71cb-4ce9-923e-2a94bfc083e7

📥 Commits

Reviewing files that changed from the base of the PR and between 1fa5b1c and 3ba1d3e.

📒 Files selected for processing (177)

• plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md
• plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md
• plugins/nemo-experimentalist/examples/smoke-agent/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/agent.py
• plugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/configs/repair.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/harbor_wrapper.py
• plugins/nemo-experimentalist/examples/smoke-agent/main.py
• plugins/nemo-experimentalist/examples/smoke-agent/optimizer.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_image.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	33342/42105	79.2%	64.1%
Integration Tests	N/A	N/A	N/A

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[schuellc-nvidia]

Review status

Nine threads answered and resolved individually. The remaining twelve are
deliberately still open — grouping them here rather than posting twelve
near-identical replies.

Deferred, needing a decision

SPDX headers on task fixtures (8 threads). These need a call rather than a
mechanical fix, and the suggestions themselves range across "add the identifiers",
"resolve the policy conflict", and "document an exception" — which reflects a real
conflict:

• tests/expected.txt is compared byte-for-byte by the verifier, so a header
  becomes part of the expected answer
• instruction.md is the prompt handed to the agent, so a header becomes part of
  what it reads

Either the fixture format grows a metadata section the verifier and loader strip,
or these files get a documented exception. Both are more than a header insertion,
and getting it wrong silently changes what every task measures.

Symlink handling, CWE-59 (2 threads). harbor_wrapper.py already refuses to
upload a symlink, or a directory containing one at any depth, precisely because
upload_dir follows them. Whether that covers the paths flagged in test.sh
needs checking properly rather than asserting, so it stays open.

generalization.yaml as a distinct scenario (1 thread). Fair. The two
single-round configs differ in one line today. The split design is what actually
decides which question a run answers, and the config could express that more
plainly.

Entry/exit sections in the weaknesses doc (1 thread). Straightforward; not
done yet.

Also worth noting

This branch was force-pushed after a rebase onto current main and a squash, and
the agent source moved into an agent/ subdirectory, so most line anchors have
shifted. Threads resolved above were resolved against the current head.

The two Experimentalist fixes this fixture depends on now live in #1163, and this
PR is based on that branch — it will retarget to main automatically once #1163
merges.

[coderabbitai]

Actionable comments posted: 13

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/.gitignore`:
- Around line 1-7: Add the required NVIDIA SPDX copyright header and Apache-2.0
license identifier at the top of the .gitignore file, before the existing
generated-dataset comments, while preserving the ignore rule and explanatory
text.

In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py`:
- Line 60: Define a concrete TypedDict for records with name, dept, role, and
hours fields, then update the _records annotation in the agent initialization to
use list of that TypedDict instead of list[dict].

In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py`:
- Around line 87-101: The upload flow around _reject_symlinks and the
selected-entry loop must apply EXCLUDE and EXCLUDE_GLOB recursively, not only to
AGENT_DIR’s top-level children. Build an allowlisted staging tree or otherwise
ensure every descendant is filtered before environment.upload_dir or
environment.upload_file, preventing nested credentials and excluded directories
such as .env and .venv from reaching the container.
- Around line 112-115: Update the command constructed in the environment.exec
call to invoke main.py through uv using “uv run python main.py”, while
preserving the existing quoted instruction and session_id arguments. Do not add
--no-project or --frozen.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md`:
- Around line 1-9: Define a non-semantic SPDX metadata mechanism for all five
instruction.md fixtures without adding literal headers to prompt content:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
(lines 1-9), lookup-grace/instruction.md (lines 1-9),
lookup-role-obrien/instruction.md (lines 1-9),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
(lines 1-9), and validation/trailing-prose/instruction.md (lines 1-9). Store the
required SPDX lines outside the prompts or implement a defined metadata-block
stripping path in the instruction loader, and add tests confirming the loaded
prompt content and baseline behavior remain unchanged.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md`:
- Line 1: Add Markdown SPDX copyright and Apache-2.0 license comments before the
existing prompt in each listed instruction.md file:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md
(lines 1-1), and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md
(lines 1-1). Preserve each instruction’s prompt content exactly and validate
task behavior after adding the metadata.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md`:
- Around line 1-9: Add non-rendered NVIDIA copyright and Apache-2.0 SPDX
identifier comments to each affected instruction.md fixture, preserving every
existing task prompt and required output format:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md
lines 1-9; and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md
lines 1-9.

In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 199-202: Update the gate description near the existing “Five
assertions” text to state that it contains eight assertions, including rejection
of candidates that fail generalization on held-out instances. Ensure the listed
assertion categories accurately cover all eight checks.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py`:
- Around line 64-68: Update assemble so removal of the generated target
directory does not suppress deletion errors: replace the ignore_errors behavior
in the shutil.rmtree call with error propagation. Keep the existing target path
and rebuild flow unchanged so assembly stops instead of continuing with stale
contents when deletion is incomplete.

In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py`:
- Around line 163-170: Update the proposer prompt construction in
_run_with_context so available_types are presented as a preference rather than a
requirement, allowing types from tried_types to be proposed when appropriate.
Preserve the all_types allowance used by _filter_improvements, and add a prompt
contract test verifying reused types remain permissible while untried types are
favored.

In
`@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py`:
- Around line 39-41: Update `_task_tomls()` to assert that `dataset/groups`
exists and that its recursive `task.toml` result is non-empty, failing when task
fixtures are missing. Remove early returns in the image, verifier, and
environment tests that skip execution for an empty task list, while preserving
`_all` as optional in its separate tests.

In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py`:
- Around line 108-127: Update _agent_class so candidate agent.py files are never
imported or executed in the pytest process. Remove the direct importlib
execution path and replay each candidate through the existing isolated Harbor or
container environment, returning its ReportAgent-compatible result via that
boundary; do not use a host subprocess as a substitute for isolation.

In
`@plugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py`:
- Around line 43-53: Strengthen test_contract_offers_an_alternative_grounding by
asserting the contract includes “name the methods that ran”, “the order they ran
in”, “their status”, and “do not retry” in addition to the existing call-graph
requirement. Keep the zero-turn grounding and forbidden-ID assertions unchanged.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 12efb76a-62cb-4d47-8575-f0aa26940538

📥 Commits

Reviewing files that changed from the base of the PR and between 648152c and 01ddc0e.

📒 Files selected for processing (184)

• plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md
• plugins/nemo-experimentalist/examples/smoke-agent/.gitignore
• plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md
• plugins/nemo-experimentalist/examples/smoke-agent/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py
• plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py
• plugins/nemo-experimentalist/examples/smoke-agent/agent/main.py
• plugins/nemo-experimentalist/examples/smoke-agent/configs/full.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/configs/repair.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/optimizer.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_image.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py
• plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_scorer.py
• plugins/nemo-experimentalist/tests/experimentalist/test_proposer_validation.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py
• plugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py

🚧 Files skipped from review as they are similar to previous changes (99)

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/optimizer.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.toml
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yaml
• plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.toml
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/configs/repair.yaml

[coderabbitai]

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

🛑 Comments failed to post (13)

plugins/nemo-experimentalist/examples/smoke-agent/.gitignore (1)

1-7: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add the required SPDX header.

This file lacks the NVIDIA SPDX copyright header and Apache-2.0 license identifier.

As per coding guidelines: “Every file must include the required NVIDIA SPDX copyright header and Apache-2.0 license identifier.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/.gitignore` around lines 1
- 7, Add the required NVIDIA SPDX copyright header and Apache-2.0 license
identifier at the top of the .gitignore file, before the existing
generated-dataset comments, while preserving the ignore rule and explanatory
text.

Source: Coding guidelines

plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py (1)

60-60: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu
file=$(fd -t f 'agent\.py$' plugins/nemo-experimentalist/examples/smoke-agent)
printf '%s\n' "$file"
sed -n '1,130p' "$file"
printf '\n--- record access and related types ---\n'
rg -n --glob '*.py' '(_records|RECORDS_PATH|TypedDict|Record)' plugins/nemo-experimentalist/examples/smoke-agent plugins/nemo-experimentalist

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 22120

---

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- smoke-agent files ---'
git ls-files plugins/nemo-experimentalist/examples/smoke-agent
printf '%s\n' '--- record fixtures ---'
fd -t f -i 'records.*' plugins/nemo-experimentalist
printf '%s\n' '--- baseline fixture and configuration ---'
sed -n '1,95p' plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
rg -n --glob 'pyproject.toml' --glob '*.toml' 'ruff|ty|mypy|disallow_any|ann' plugins/nemo-experimentalist .
printf '%s\n' '--- JSON record shape probe ---'
python3 - <<'PY'
import json
from pathlib import Path

for path in Path("plugins/nemo-experimentalist").rglob("*.json"):
    if "records" not in path.name.lower():
        continue
    try:
        value = json.loads(path.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError):
        continue
    print(path)
    print("root_type:", type(value).__name__, "length:", len(value) if hasattr(value, "__len__") else None)
    if isinstance(value, list):
        for index, record in enumerate(value):
            print(index, type(record).__name__, sorted(record) if isinstance(record, dict) else None,
                  {key: type(item).__name__ for key, item in record.items()} if isinstance(record, dict) else None)
PY

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 50381

---

🏁 Script executed:

#!/bin/bash
set -eu
record_file=plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json
printf '%s\n' '--- records.json ---'
cat "$record_file"
printf '%s\n' '--- targeted type-check configuration ---'
sed -n '560,690p' pyproject.toml
printf '%s\n' '--- root Ruff rules ---'
sed -n '1,130p' ruff.toml
printf '%s\n' '--- record-shape verifier ---'
python3 - <<'PY'
import json
from pathlib import Path

path = Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json")
records = json.loads(path.read_text(encoding="utf-8"))
assert isinstance(records, list)
expected = {"name": str, "dept": str, "role": str, "hours": int}
print("records:", len(records))
for index, record in enumerate(records):
    assert isinstance(record, dict), (index, type(record).__name__)
    print(index, {key: type(value).__name__ for key, value in record.items()})
    assert set(record) == set(expected), (index, set(record))
    assert all(isinstance(record[key], expected[key]) for key in expected), index
print("shape: list[TypedDict-like records] with keys name, dept, role, hours")
PY

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 8357

---

Use a concrete record type for _records.

Define a TypedDict with name, dept, role, and hours, then annotate _records with it instead of list[dict].

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py` at line 60,
Define a concrete TypedDict for records with name, dept, role, and hours fields,
then update the _records annotation in the agent initialization to use list of
that TypedDict instead of list[dict].

Source: Coding guidelines

plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py (2)

87-101: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Filter excluded paths recursively.

Line 87 filters only top-level entries. Lines 100-101 upload nested .env, .venv, and other excluded paths inside a selected directory. A nested credential file can reach the task container.

Build an allowlisted staging directory, or apply the exclusions to every descendant before upload.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py`
around lines 87 - 101, The upload flow around _reject_symlinks and the
selected-entry loop must apply EXCLUDE and EXCLUDE_GLOB recursively, not only to
AGENT_DIR’s top-level children. Build an allowlisted staging tree or otherwise
ensure every descendant is filtered before environment.upload_dir or
environment.upload_file, preventing nested credentials and excluded directories
such as .env and .venv from reaching the container.

---

112-115: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- target context ---'
sed -n '90,130p' plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py

printf '%s\n' '--- relevant uv and Python execution guidance ---'
rg -n -i --glob 'README*' --glob '*.md' --glob '*.py' --glob '*.yaml' --glob '*.yml' --glob '*.toml' \
  'uv run|python main\.py|nooa|smoke-agent|task image' \
  plugins/nemo-experimentalist . 2>/dev/null | head -n 240

printf '%s\n' '--- smoke-agent files ---'
git ls-files plugins/nemo-experimentalist/examples/smoke-agent

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 50381

---

🏁 Script executed:

#!/bin/bash
set -eu

for f in \
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile \
  plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md \
  plugins/nemo-experimentalist/examples/smoke-agent/README.md \
  plugins/nemo-experimentalist/examples/smoke-agent/agent/main.py \
  plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py \
  plugins/nemo-experimentalist/examples/tau3-nooa-agent/harbor_wrapper.py \
  plugins/nemo-experimentalist/examples/tau3-nooa-agent/README.md
do
  printf '\n--- %s ---\n' "$f"
  wc -l "$f"
  case "$f" in
    *Dockerfile) cat -n "$f" ;;
    *AGENT-SPEC.md) cat -n "$f" ;;
    *smoke-agent/README.md) sed -n '1,115p' "$f"; sed -n '180,215p' "$f" ;;
    *smoke-agent/agent/main.py) cat -n "$f" ;;
    *smoke-agent/agent/harbor_wrapper.py) cat -n "$f" ;;
    *tau3-nooa-agent/harbor_wrapper.py) sed -n '55,155p' "$f" ;;
    *tau3-nooa-agent/README.md) cat -n "$f" ;;
  esac
done

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 25677

---

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- smoke-agent tests concerning the image and wrapper ---'
for f in \
  plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py \
  plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py \
  plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py \
  plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py
do
  printf '\n--- %s ---\n' "$f"
  rg -n -C 5 'Dockerfile|uv|python main|harbor|NOOA|nooa|image|wrapper|exec' "$f" || true
done

printf '%s\n' '--- image and dependency declarations ---'
rg -n -C 4 'FROM |COPY --from|uv |nooa|pyproject|uv.lock|docker_image' \
  plugins/nemo-experimentalist/examples/smoke-agent \
  plugins/nemo-experimentalist/pyproject.toml \
  pyproject.toml

printf '%s\n' '--- available uv executable and run semantics ---'
if command -v uv >/dev/null 2>&1; then
  uv --version
  uv run --help | sed -n '1,180p'
else
  echo 'uv is not available in the verification sandbox'
fi

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 50383

---

🌐 Web query:

uv run behavior without pyproject.toml project-less directory system site-packages

💡 Result:

When you use uv run in a directory without a pyproject.toml file, uv operates in a project-less context [1]. Its behavior is determined by the following rules: 1. Project Discovery: Without a pyproject.toml, uv does not treat the directory as a project [1][2]. If you have a.venv directory in your current or parent directory, uv will detect and use that virtual environment [2]. If no virtual environment is found, uv will fall back to using the discovered Python interpreter [1][2]. 2. Handling System Site-Packages: uv creates isolated environments and does not natively support mixing system-installed packages with packages installed inside a uv-managed environment [3][4]. While you can create a virtual environment with the --system-site-packages flag (e.g., uv venv --system-site-packages), this only grants the environment access to system packages at runtime [5][4]. Crucially, uv does not consider these system packages when calculating dependencies, installing packages, or resolving environments [5][4]. This means tools like uv pip list or uv add will remain unaware of the system-installed packages [5][4]. 3. Common Pitfalls: If you have a pyproject.toml that lacks a [project] table, uv may still attempt to treat it as a project and fail with an error [6]. In such cases, or if you simply wish to avoid project discovery entirely, you can use the --no-project flag to force uv to ignore any existing project configuration and run the command in an isolated or environment-agnostic context [1][7][6]. In summary, uv is designed to maintain strict isolation [3]. If you have a requirement to use system-level packages alongside uv-managed dependencies, the recommended approach is to have uv manage all dependencies directly, as it does not support "inheriting" or mixing in system site-packages for its dependency management workflows [3][5].

Citations:

• 1: https://pydevtools.com/handbook/explanation/what-happens-when-you-run-uv-run/
• 2: https://docs.astral.sh/uv/reference/cli/
• 3: https://stackoverflow.com/questions/79839386/use-system-packages-inside-a-python-script-with-uv
• 4: astral-sh/uv#4466
• 5: astral-sh/uv#17215
• 6: astral-sh/uv#7068
• 7: https://github.com/astral-sh/uv/blob/262a50bb/docs/guides/scripts.md

---

🌐 Web query:

site:docs.astral.sh/uv uv run --no-project --frozen no pyproject.toml behavior

💡 Result:

When you use uv run with the --no-project and --frozen flags, the behavior is as follows: --no-project: This flag prevents uv from discovering or using any project or workspace in the current directory or its parents [1][2]. Instead of using the project environment (the.venv created by the pyproject.toml), uv will create an isolated, ephemeral environment [1][2]. This means it will not install your current project, nor will it incorporate dependencies defined in your pyproject.toml [1][3][2]. --frozen: When used in conjunction with other flags or within a project, this tells uv to use the lockfile (if present) as the source of truth without attempting to update it [1][4]. However, when combined with --no-project, you are typically creating an isolated environment from scratch (e.g., using --with to specify dependencies) [1][2]. If you are not providing any dependencies via --with, and there is no project environment being used, uv will operate in a minimal environment [1][2]. If the command you run relies on packages, they must be specified via --with, or they must already be present in the environment (which is generally not the case for ephemeral --no-project environments unless explicitly handled) [1][2]. In summary, --no-project tells uv to ignore the pyproject.toml and project-related environment management entirely, opting for an isolated, ephemeral environment, while --frozen restricts uv from modifying or attempting to update any lockfiles [1][4][2]. If you use both, you are effectively running a command in a highly restricted, non-project-aware state that will not touch your project files [1][2].

Citations:

• 1: https://docs.astral.sh/uv/reference/cli/
• 2: https://docs.astral.sh/uv/reference/cli/index.md
• 3: https://docs.astral.sh/uv/guides/scripts/
• 4: https://docs.astral.sh/uv/concepts/projects/sync/

---

Run main.py through uv.

Use uv run python main.py .... The task image includes uv and installs NOOA into its system interpreter. Do not use --no-project or --frozen, which would not reuse that installation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py`
around lines 112 - 115, Update the command constructed in the environment.exec
call to invoke main.py through uv using “uv run python main.py”, while
preserving the existing quoted instruction and session_id arguments. Do not add
--no-project or --frozen.

Source: Coding guidelines

plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md (1)

1-9: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

files=(
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md
)

for file in "${files[@]}"; do
  grep -q 'SPDX-License-Identifier: Apache-2.0' "$file" || {
    echo "Missing Apache-2.0 identifier: $file"
    exit 1
  }
done

rg -n 'SPDX|Apache-2.0|fixture|license' plugins/nemo-experimentalist

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 319

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

files=(
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md
)

printf '%s\n' '--- target file metadata and contents ---'
for file in "${files[@]}"; do
  printf '\n### %s\n' "$file"
  wc -l "$file"
  cat -n "$file"
done

printf '%s\n' '--- repository policy and fixture metadata references ---'
rg -n -i --glob '!**/.venv/**' \
  'SPDX|Apache-2\.0|fixture|metadata|smoke-agent|instruction\.md|license' \
  plugins/nemo-experimentalist .github README.md 2>/dev/null | head -n 400

printf '%s\n' '--- nearby and representative files ---'
find plugins/nemo-experimentalist -type f \( -name '*.md' -o -name '*.yaml' -o -name '*.yml' -o -name '*.json' -o -name '*.py' -o -name '*.toml' \) \
  | sort | head -n 250

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 50383

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- scoped policy ---'
cat -n plugins/nemo-experimentalist/AGENTS.md | sed -n '1,100p'

printf '%s\n' '--- smoke-agent files and references ---'
find plugins/nemo-experimentalist/examples/smoke-agent -type f -print | sort
rg -n -F 'smoke-agent' plugins/nemo-experimentalist --glob '*.py' --glob '*.yaml' --glob '*.yml' --glob '*.toml' --glob '*.md'
rg -n -F 'instruction.md' plugins/nemo-experimentalist --glob '*.py' --glob '*.yaml' --glob '*.yml' --glob '*.toml'
rg -n -i 'read_text|instruction|prompt|dataset' plugins/nemo-experimentalist/src plugins/nemo-experimentalist/benchmarks --glob '*.py' | head -n 300

printf '%s\n' '--- SPDX coverage under the example tree ---'
python3 - <<'PY'
from pathlib import Path

root = Path("plugins/nemo-experimentalist/examples/smoke-agent")
for path in sorted(p for p in root.rglob("*") if p.is_file()):
    text = path.read_text(encoding="utf-8", errors="replace")
    if "SPDX-License-Identifier: Apache-2.0" not in text:
        print(path)
PY

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 50381

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

files=(
  plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md
  plugins/nemo-experimentalist/examples/smoke-agent/README.md
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml
  plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml
  plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py
  plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py
  plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
)

for file in "${files[@]}"; do
  printf '\n--- %s ---\n' "$file"
  wc -l "$file"
  case "$file" in
    *.py) rg -n -C 8 'instruction\.md|read_text|SPDX|digest|hash|task\.toml|asset' "$file" ;;
    *) cat -n "$file" | sed -n '1,220p' ;;
  esac
done

printf '\n--- all metadata-like files in smoke-agent ---\n'
find plugins/nemo-experimentalist/examples/smoke-agent -type f \
  \( -iname '*meta*' -o -iname '*license*' -o -iname '*.json' -o -iname '*.toml' -o -iname '*.yaml' -o -iname '*.yml' \) \
  -print | sort

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 40237

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path

targets = [
    Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md"),
    Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md"),
    Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md"),
    Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md"),
    Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md"),
]

assert len(targets) == 5
for path in targets:
    text = path.read_text(encoding="utf-8")
    assert "SPDX-License-Identifier: Apache-2.0" not in text, path
    assert "SPDX-FileCopyrightText:" not in text, path
    assert text == text.strip() + "\n", path
    assert (path.parent / "task.toml").is_file(), path

gate = Path("plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py").read_text(encoding="utf-8")
baseline = Path("plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py").read_text(encoding="utf-8")
assets = Path("plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py").read_text(encoding="utf-8")

assert '(task_dir / "instruction.md").read_text(encoding="utf-8").strip()' in gate
assert '(task / "instruction.md").read_text(encoding="utf-8").strip()' in baseline
assert 'for name in ("instruction.md", "task.toml", "tests/expected.txt")' in assets
assert "fixture exception" not in gate.lower() + baseline.lower() + assets.lower()
assert "metadata mechanism" not in gate.lower() + baseline.lower() + assets.lower()

header = (
    "<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION "
    "& AFFILIATES. All rights reserved. -->\n"
    "<!-- SPDX-License-Identifier: Apache-2.0 -->\n\n"
)
for path in targets:
    original = path.read_text(encoding="utf-8")
    assert (header + original).strip() != original.strip(), path
    assert (header + original).startswith(header), path

print("targets_missing_both_headers=5")
print("approved_fixture_exception_or_metadata_path=absent")
print("instruction_content_is_read_directly= true")
print("literal_header_changes_prompt_bytes= true")

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 417

---

Define a non-semantic SPDX mechanism for the five prompt fixtures.

All five files lack both required SPDX lines. Do not prepend literal headers: the loader reads instruction.md directly, so headers change prompt content and baseline behavior. Store metadata outside the prompt or strip a defined metadata block before loading, with tests.

🧰 Tools

🪛 markdownlint-cli2 (0.23.2)

[warning] 1-1: First line in a file should be a top-level heading

(MD041, first-line-heading, first-line-h1)

---

[warning] 5-5: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

📍 Affects 5 files

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md#L1-L9 (this comment)
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md#L1-L9

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md`
around lines 1 - 9, Define a non-semantic SPDX metadata mechanism for all five
instruction.md fixtures without adding literal headers to prompt content:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
(lines 1-9), lookup-grace/instruction.md (lines 1-9),
lookup-role-obrien/instruction.md (lines 1-9),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
(lines 1-9), and validation/trailing-prose/instruction.md (lines 1-9). Store the
required SPDX lines outside the prompts or implement a defined metadata-block
stripping path in the instruction loader, and add tests confirming the loaded
prompt content and baseline behavior remain unchanged.

Source: Coding guidelines

plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md (1)

1-1: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add required SPDX metadata to every task instruction.

These files lack the required NVIDIA copyright and Apache-2.0 identifiers. Use Markdown comments. Preserve the prompt content and validate task behavior after the change.

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md#L1-L1: add SPDX comments before the instruction.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md#L1-L1: add SPDX comments before the instruction.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md#L1-L1: add SPDX comments before the instruction.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md#L1-L1: add SPDX comments before the instruction.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md#L1-L1: add SPDX comments before the instruction.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md#L1-L1: add SPDX comments before the instruction.

🧰 Tools

🪛 markdownlint-cli2 (0.23.2)

[warning] 1-1: First line in a file should be a top-level heading

(MD041, first-line-heading, first-line-h1)

📍 Affects 6 files

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md#L1-L1 (this comment)
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md#L1-L1
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md#L1-L1
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md#L1-L1
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md#L1-L1
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md#L1-L1

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md`
at line 1, Add Markdown SPDX copyright and Apache-2.0 license comments before
the existing prompt in each listed instruction.md file:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md
(lines 1-1), and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md
(lines 1-1). Preserve each instruction’s prompt content exactly and validate
task behavior after adding the metadata.

Source: Coding guidelines

plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md (1)

1-9: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Add the required SPDX identifiers to each task fixture.

Use non-rendered Markdown comments. Preserve the task prompt and output format.

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.

🧰 Tools

🪛 markdownlint-cli2 (0.23.2)

[warning] 1-1: First line in a file should be a top-level heading

(MD041, first-line-heading, first-line-h1)

---

[warning] 5-5: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

📍 Affects 7 files

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md#L1-L9 (this comment)
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md#L1-L9
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md#L1-L9

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md`
around lines 1 - 9, Add non-rendered NVIDIA copyright and Apache-2.0 SPDX
identifier comments to each affected instruction.md fixture, preserving every
existing task prompt and required output format:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md
lines 1-9; and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md
lines 1-9.

Source: Coding guidelines

plugins/nemo-experimentalist/examples/smoke-agent/README.md (1)

199-202: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Correct the gate assertion count.

The PR adds an eight-assertion gate. This section says five assertions and omits failed-generalization candidate rejection.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md` around lines 199
- 202, Update the gate description near the existing “Five assertions” text to
state that it contains eight assertions, including rejection of candidates that
fail generalization on held-out instances. Ensure the listed assertion
categories accurately cover all eight checks.

plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py (1)

64-68: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Do not suppress generated-directory deletion errors.

Line 67 ignores a failed deletion. A partial deletion can retain tasks from a removed or excluded group, then mix them with current tasks in _all.

Proposed fix

-    shutil.rmtree(target, ignore_errors=True)
+    if target.exists():
+        shutil.rmtree(target)

Based on learnings: the combined-group test requires generated tasks to exactly match their sources.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

def assemble(dataset_dir: Path) -> list[Path]:
    """Rebuild the combined group from its sources; return the task directories written."""
    target = dataset_dir / "groups" / COMBINED
    if target.exists():
        shutil.rmtree(target)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py`
around lines 64 - 68, Update assemble so removal of the generated target
directory does not suppress deletion errors: replace the ignore_errors behavior
in the shutil.rmtree call with error propagation. Keep the existing target path
and rebuild flow unchanged so assembly stops instead of continuing with stale
contents when deletion is incomplete.

plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py (1)

163-170: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make repeated types permissible in the proposer prompt.

Lines 163-170 allow reused types. _run_with_context still requires available_types and avoids tried_types. The model will not propose a reused type while an untried type exists. Change the prompt to prefer available_types, not require them. Add a prompt contract test.

Proposed fix

-        - tried_types (list[str]): optimization_types already attempted — AVOID these
-        - available_types (list[str]): types not yet tried — PICK FROM THESE
+        - tried_types (list[str]): optimization_types already attempted — reuse when analysis supports it
+        - available_types (list[str]): types not yet tried — prefer these for novelty
...
-        3. Load the matching card with `doc(self.optimize.<name>)` and pick ONE
-           optimization_type from `available_types` that the card covers.
+        3. Load the matching card with `doc(self.optimize.<name>)` and pick ONE
+           optimization_type from `available_types` or `tried_types` that the card covers.
...
-        - Each Improvement.optimization_type MUST be in `available_types`. If
-          `available_types` is empty, pick the least-tried type from the tried set.
+        - Prefer an Improvement.optimization_type in `available_types`. Reuse a
+          type from `tried_types` when it is the best match for the root cause.

Based on supplied change details, novelty is intended to be a preference, not a requirement.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py`
around lines 163 - 170, Update the proposer prompt construction in
_run_with_context so available_types are presented as a preference rather than a
requirement, allowing types from tried_types to be proposed when appropriate.
Preserve the all_types allowance used by _filter_improvements, and add a prompt
contract test verifying reused types remain permissible while untried types are
favored.

plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py (1)

39-41: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Fail when the fixture has no tasks.

An absent dataset/groups directory returns an empty list. The image and verifier tests then return, and the environment test iterates zero tasks. Deleting all task assets can pass this gate.

Assert that the groups directory and task list exist in _task_tomls(). Remove the early returns that depend on an empty task list. Keep _all optional in its separate tests.

Proposed fix

 def _task_tomls() -> list[Path]:
     groups = _EXAMPLE_DIR / "dataset" / "groups"
-    return sorted(groups.rglob("task.toml")) if groups.is_dir() else []
+    assert groups.is_dir(), f"missing smoke-agent task groups: {groups}"
+    tasks = sorted(groups.rglob("task.toml"))
+    assert tasks, f"no smoke-agent tasks found under {groups}"
+    return tasks

Also applies to: 145-147, 159-161, 179-186

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py`
around lines 39 - 41, Update `_task_tomls()` to assert that `dataset/groups`
exists and that its recursive `task.toml` result is non-empty, failing when task
fixtures are missing. Remove early returns in the image, verifier, and
environment tests that skip execution for an empty task list, while preserving
`_all` as optional in its separate tests.

plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py (1)

108-127: 🔒 Security & Privacy | 🔴 Critical | 🏗️ Heavy lift

Do not execute candidate code in the pytest process.

spec.loader.exec_module(module) executes agent.py from SMOKE_EXPERIMENT_DIR. Candidate code is generated during optimization and can run arbitrary host commands, read CI credentials, or modify the checkout during this test.

Replay candidates only in the existing isolated Harbor or container environment. A host subprocess is not an isolation boundary.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py`
around lines 108 - 127, Update _agent_class so candidate agent.py files are
never imported or executed in the pytest process. Remove the direct importlib
execution path and replay each candidate through the existing isolated Harbor or
container environment, returning its ReportAgent-compatible result via that
boundary; do not use a host subprocess as a substitute for isolation.

plugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py (1)

43-53: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the complete zero-turn grounding rule.

The test accepts a contract that only says “call graph.” It does not require methods, execution order, status, or the prohibition on retrying turn lookups. A regression that removes those requirements still passes.

Assert "name the methods that ran", "the order they ran in", "their status", and "do not retry".

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py`
around lines 43 - 53, Strengthen test_contract_offers_an_alternative_grounding
by asserting the contract includes “name the methods that ran”, “the order they
ran in”, “their status”, and “do not retry” in addition to the existing
call-graph requirement. Keep the zero-turn grounding and forbidden-ID assertions
unchanged.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 8

🧹 Nitpick comments (1)

plugins/nemo-experimentalist/examples/smoke-agent/README.md (1)

4-20: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Keep README.md in one Diátaxis quadrant.

This page mixes a HOW-TO, architecture explanation, reference tables, troubleshooting, and a timing report. Keep it as a short HOW-TO. Move design, group rationale, and timing details to linked explanation or reference pages.

As per coding guidelines, each documentation page must fit one Diátaxis quadrant and use progressive disclosure.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md` around lines 4 -
20, Refocus the smoke-agent README on a concise HOW-TO covering setup and usage
only. Move architecture, design rationale, group details, reference information,
troubleshooting, and timing results into appropriate linked explanation or
reference documentation, preserving the existing guidance about agent and
AGENT-SPEC.md constraints through progressive disclosure.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md`:
- Around line 4-6: Add the required top-level Prerequisites section before Job
in plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md (lines 4-6)
and before Design in plugins/nemo-experimentalist/examples/smoke-agent/README.md
(lines 4-6); add a Next Steps section at the end of AGENT-SPEC.md after line 67
and README.md after line 262, preserving each page’s existing content and
navigation structure.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile`:
- Around line 10-25: Create a dedicated non-root user in the Dockerfile, grant
ownership or write access to /app, /app/artifacts, and /app/traces, then set the
image’s USER to that account after setup. Ensure Harbor can still read
/app/data/records.json and run the upload and artifact-write smoke tests under
the configured user.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.md`:
- Line 1: Shorten the preamble in the instruction so the “What is the hours of
Grace Hopper?” question begins within the 240-character scan limit used by
solve() and remains matchable by LOOKUP_RE; preserve the existing “what is”
question form.

In
`@plugins/nemo-experimentalist/examples/smoke-agent/optimizer-generalization.yaml`:
- Around line 16-19: Update the explanatory comment in
optimizer-generalization.yaml to reference the actual fixture paths,
optimizer-generalization.yaml and configs/single-round.yaml, instead of the
nonexistent generalization.yaml and optimizer.yaml names.

In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 204-210: Update the README’s standalone g5-edge-cases run
instructions to use a trajectory-enabled configuration, either by referencing an
appropriate config or explicitly overriding disable_trajectory_scoring to false.
State the required trajectory-scoring setting so the documented command can
exercise g5-edge-cases.
- Around line 235-238: Update the gate description near the “Five assertions”
text to accurately reflect the current eight-assertion gate, including the
complete assertion list, or replace the summary with a direct reference to
test_smoke_agent_gate.py. Ensure the README no longer implies incomplete
coverage.
- Around line 171-182: Define an explicit SPDX exception in the governing policy
for task payload files under dataset/groups/, including instruction.md and
tests/expected.txt, without adding headers to those payloads. Update the “Why
the task files carry no licence header” section to reference the new policy
exception while preserving the byte-for-byte and prompt-content constraints.

In `@plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_image.py`:
- Around line 50-54: Harden the task discovery and .gitkeep creation flow around
the loop over `dataset_dir / "groups"`: reject symlinked `task.toml` files and
environment paths, resolve candidate paths, and verify they remain under the
intended `dataset_dir` before creating directories or writing files. Apply the
same validation to the related lines 63-66 path handling, and use no-follow file
operations if the implementation addresses concurrent filesystem changes.

---

Nitpick comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 4-20: Refocus the smoke-agent README on a concise HOW-TO covering
setup and usage only. Move architecture, design rationale, group details,
reference information, troubleshooting, and timing results into appropriate
linked explanation or reference documentation, preserving the existing guidance
about agent and AGENT-SPEC.md constraints through progressive disclosure.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a6c6177d-70c2-4850-a866-8179d6e4f362

📥 Commits

Reviewing files that changed from the base of the PR and between 0c4dc81 and 8e6c77d.

📒 Files selected for processing (180)

• plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md
• plugins/nemo-experimentalist/examples/smoke-agent/.gitignore
• plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md
• plugins/nemo-experimentalist/examples/smoke-agent/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py
• plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py
• plugins/nemo-experimentalist/examples/smoke-agent/agent/main.py
• plugins/nemo-experimentalist/examples/smoke-agent/configs/full.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/configs/single-round.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/environment/.gitkeep
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/optimizer-generalization.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/optimizer.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_image.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.py
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py

🚧 Files skipped from review as they are similar to previous changes (110)

• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/.gitignore
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/agent/main.py
• plugins/nemo-experimentalist/examples/smoke-agent/configs/full.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.sh
• plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.py
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/optimizer.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.sh
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.toml
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.toml
• plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.sh
• plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txt
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
• plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.sh

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

plugins/nemo-experimentalist/examples/smoke-agent/README.md (1)

251-263: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add a Next Steps section.

The README ends with timings. Add cross-links to plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md, the scenario profiles, and the gate tests.

As per coding guidelines, each documentation page must end with a Next Steps section with cross-links.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md` around lines 251
- 263, Add a final “Next Steps” section to the README after the Timings section,
including cross-links to the smoke-agent weaknesses document, scenario profiles,
and gate tests. Ensure this section is the last content in the documentation
page and uses the repository’s relative-link conventions.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 4-6: Add a Prerequisites section near the top of the smoke-agent
README, before the Design section, listing the required Docker sandbox, sbx, uv,
and platform model configuration. Keep the existing setup and usage content
unchanged.

---

Outside diff comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 251-263: Add a final “Next Steps” section to the README after the
Timings section, including cross-links to the smoke-agent weaknesses document,
scenario profiles, and gate tests. Ensure this section is the last content in
the documentation page and uses the repository’s relative-link conventions.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 031a5290-e0b1-47e5-a927-ef0ae5f5a137

📥 Commits

Reviewing files that changed from the base of the PR and between 874698d and fbabb9d.

📒 Files selected for processing (4)

• plugins/nemo-experimentalist/examples/smoke-agent/README.md
• plugins/nemo-experimentalist/examples/smoke-agent/configs/full.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/configs/short.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/optimizer-generalization.yaml

🚧 Files skipped from review as they are similar to previous changes (2)

• plugins/nemo-experimentalist/examples/smoke-agent/optimizer-generalization.yaml
• plugins/nemo-experimentalist/examples/smoke-agent/configs/full.yaml

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-1089-experimentalist-smoke-agent-proposal-cs.docs.buildwithfern.com/nemo-platform