Skip to content

refactor(advisor): make review turns ledger-driven - #6547

Merged
cv merged 9 commits into
mainfrom
codex/advisor-ledger-flow
Jul 9, 2026
Merged

refactor(advisor): make review turns ledger-driven#6547
cv merged 9 commits into
mainfrom
codex/advisor-ledger-flow

Conversation

@cv

@cv cv commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Refactors PR Review Advisor so each stage starts with its instruction, invokes real turn-scoped context tools, emits visible analysis, and then commits findings to a canonical ledger. Final synthesis is read-only, and the runner projects canonical ledger findings into the published result so drift cannot silently change the review.

Related Issue

Related issue #6446. This PR is a follow-up.

Changes

  • Replace pre-prompt synthetic messages with one canonical turn-scoped context-tool contract and explicit ordering validation; no compatibility aliases are retained.
  • Add a shared finding ledger with stable IDs, strict atomic operation batches, rollback, and evidence-backed transitions.
  • Make synthesis and retry ledger-backed, expose only open findings to the model, preserve the full audit ledger, preserve deterministic runtime-test requirements against model downgrades, and fail closed only on persistent ledger drift.
  • Confine read, grep, find, and ls to SDK-normalized real paths inside the checkout, and require exactly one successful terminal ledger mutation per analysis turn.
  • Document the revised conversation flow and add regression coverage for PR and E2E advisor callers.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Internal maintainer automation only; its internal README was updated and no user-facing NemoClaw behavior changed.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent boundary review reproduced and then verified fixes for absolute, parent, alias, symlink, and post-realpath Unicode-normalization escapes; final review found no remaining actionable defect.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project integration test/advisor-repo-read-only-tools.test.ts test/advisor-session-runner.test.ts test/advisor-session-context-tools.test.ts test/pr-review-advisor-ledger-tools.test.ts test/pr-review-advisor-test-depth.test.ts test/pr-review-advisor.test.ts test/pr-review-advisor-turns.test.ts test/e2e-advisor.test.ts test/e2e-advisor-targets.test.ts test/pr-review-advisor-workflow-boundary.test.ts test/test-title-style.test.ts (147 passed)
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — umask 022; env -u SSH_CLIENT -u SSH_CONNECTION -u SSH_TTY npm test (15,328 passed; 39 skipped; 1 todo)
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

@cv cv added the v0.0.79 label Jul 9, 2026
@cv cv self-assigned this Jul 9, 2026
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds a review finding ledger, routes PR review advisor flows through ledger-backed prompts and persistence, replaces synthetic tool-result injection with per-turn context tools, and updates tests and documentation to match the new contracts.

Changes

Ledger and context-tool advisor flow

Layer / File(s) Summary
Review finding ledger implementation
tools/pr-review-advisor/review-ledger.mts
Defines ledger types, state transitions, batch validation, snapshotting, normalization helpers, and update/read tool controller wiring.
Advisor session context-tool runtime
tools/advisors/session.mts
Adds context tool result types, prompt-turn ordering fields, the inert context tool runtime, turn-tool resolution, and sequencing validation helpers.
Analyze.mts ledger-backed advisor flow
tools/pr-review-advisor/analyze.mts
Creates and persists the shared finding ledger, canonicalizes first-pass and retry output from ledger snapshots, rewrites stage prompts to use the ledger protocol, and bounds serialized changedFiles metadata.
Ledger tool test suite
test/pr-review-advisor-ledger-tools.test.ts, test/pr-review-advisor.test.ts
Covers ledger tool behavior, atomic batch semantics, synthesis separation, drift canonicalization, conclusion update rules, retry preservation, and stage-contract assertions.
Context tool test suite
test/advisor-session-context-tools.test.ts, test/e2e-advisor*.test.ts, test/pr-review-advisor-turns.test.ts
Updates context-tool assertions across advisor and e2e tests, including prompt construction, artifact naming, and turn-context field usage.
README documentation updates
tools/advisors/README.md, tools/e2e-advisor/README.md, tools/pr-review-advisor/README.md
Updates the documented advisor/session and PR-review advisor contracts, tool terminology, and artifact descriptions.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Analyze as tools/pr-review-advisor/analyze.mts
  participant Session as tools/advisors/session.mts
  participant Ledger as tools/pr-review-advisor/review-ledger.mts
  Analyze->>Ledger: createReviewFindingLedger()
  Analyze->>Session: runReadOnlyAdvisor(..., customTools)
  Session->>Ledger: pr_review_update_ledger / pr_review_read_ledger
  Session->>Session: validate required tools and turn ordering
  Analyze->>Ledger: snapshot canonical findings for output
Loading

Possibly related PRs

  • NVIDIA/NemoClaw#5170: Also changes tools/pr-review-advisor/analyze.mts prompt construction and advisor guidance, overlapping with the stage-protocol rewrite here.

Suggested labels: refactor

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 1.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: advisor review turns are now ledger-driven.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/advisor-ledger-flow

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No NemoClaw E2E is required. The deterministic risk plan lists no required jobs, and the PR is limited to CI advisor tooling, advisor docs, and unit-test coverage. It does not change installer/onboarding, sandbox lifecycle, product credentials, network policy, inference routing, deployment, or real assistant user flows.

Optional E2E

  • None.

New E2E recommendations

  • None.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: None
Optional E2E targets: None

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • None. No NemoClaw E2E target dispatch is required. The PR changes advisor tooling, advisor documentation, and unit tests outside test/e2e/; it does not modify .github/workflows/e2e.yaml, the E2E target registry/runtime support, live E2E files, shared E2E fixtures/support, onboarding runtime code, or other surfaces that affect E2E target behavior.

Optional E2E targets

  • None.

Relevant changed files

  • None.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — Changes requested

Merge posture: Do not merge yet
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 2 suggestions · 4 test follow-ups
Since last review: 0 prior items resolved · 0 still apply · 2 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Add runtime validation test for repo-confined tools against real Pi SDK
  • PRA-T4 Add or justify test follow-up: Add runtime validation test for context-tool ordering with real provider
  • PRA-1 In-scope improvement: Add runtime validation test for repo-confined tools against real Pi SDK in test/advisor-repo-read-only-tools.test.ts:1
  • PRA-2 In-scope improvement: Add runtime validation test for context-tool ordering with real provider in test/advisor-session-runner.test.ts:1

Findings index

ID Severity Category Location Required action
PRA-1 Improvement tests test/advisor-repo-read-only-tools.test.ts:1 Add a test in test/advisor-repo-read-only-tools.test.ts that spins up a real Pi SDK session with createRepoConfinedReadOnlyTools() and attempts symlink escapes, Unicode-space collisions, and @/~ aliases — verifying the SDK never receives an outside path.
PRA-2 Improvement tests test/advisor-session-runner.test.ts:1 Add a test in test/advisor-session-runner.test.ts (or new integration test file) that runs a single-turn advisor with the real provider, omits a required context tool call, and verifies the turn fails with 'omitted required tool result(s)'.
Review findings by urgency: 0 required fixes, 0 items to resolve/justify, 2 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

  • None.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-1 Improvement — Add runtime validation test for repo-confined tools against real Pi SDK

  • Location: test/advisor-repo-read-only-tools.test.ts:1
  • Category: tests
  • Problem: Unit tests verify RepoPathGuard logic but do not exercise the actual Pi SDK read/grep/find/ls implementations with the confined wrappers. A real SDK integration test would verify the full confinement chain end-to-end.
  • Impact: Without real SDK integration test, a Pi SDK behavior change (e.g., new path alias, normalization difference) could bypass confinement without detection.
  • Suggested action: Add a test in test/advisor-repo-read-only-tools.test.ts that spins up a real Pi SDK session with createRepoConfinedReadOnlyTools() and attempts symlink escapes, Unicode-space collisions, and @/~ aliases — verifying the SDK never receives an outside path.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the new test and confirm it exercises Pi SDK's actual read/grep/find/ls code paths, not just the guard logic.
  • Missing regression test: Add test: 'confines real Pi SDK read/grep/find/ls via repo-confined wrappers' — create temp workspace with symlink to outside, real Pi SDK session, attempt escapes, assert all rejected with 'outside the workspace' or 'not stable under Pi SDK normalization'.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: testDepth verdict: runtime_validation_recommended; repo-read-only-tools.mts:137 lines new security-critical confinement layer; existing 184-line test file uses mocked SDK only

PRA-2 Improvement — Add runtime validation test for context-tool ordering with real provider

  • Location: test/advisor-session-runner.test.ts:1
  • Category: tests
  • Problem: Mocked SDK tests verify turn-tool resolution and flow errors, but do not exercise real model interaction with the required context-tool protocol (model must call each context tool before emitting text, exactly one terminal ledger mutation after).
  • Impact: Without real provider test, a model behavior change or SDK update could allow the model to skip required context tools or call them out of order without the runner detecting it.
  • Suggested action: Add a test in test/advisor-session-runner.test.ts (or new integration test file) that runs a single-turn advisor with the real provider, omits a required context tool call, and verifies the turn fails with 'omitted required tool result(s)'.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the new test against the real inference API and confirm the turn settles with status=failed and error containing 'omitted required tool result(s)'.
  • Missing regression test: Add test: 'real provider omits required context tool -> turn fails closed' — configure real provider, single turn with required context tool, prompt model to answer without calling it, assert settlement error matches omission.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: testDepth verdict: runtime_validation_recommended; session.mts:880 lines new context-tool runtime; existing 202-line runner test mocks Pi SDK entirely
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Add runtime validation test for repo-confined tools against real Pi SDK (symlink/Unicode escapes).. Runtime/sandbox/infrastructure paths need behavioral runtime validation: tools/advisors/README.md, tools/advisors/repo-read-only-tools.mts, tools/advisors/session.mts, tools/e2e-advisor/README.md, tools/e2e-advisor/analyze.mts, tools/e2e-advisor/targets.mts, tools/pr-review-advisor/README.md, tools/pr-review-advisor/analyze.mts.
  • PRA-T2 Runtime validation — Add runtime validation test for context-tool ordering with real provider (omitted tool -> turn fails).. Runtime/sandbox/infrastructure paths need behavioral runtime validation: tools/advisors/README.md, tools/advisors/repo-read-only-tools.mts, tools/advisors/session.mts, tools/e2e-advisor/README.md, tools/e2e-advisor/analyze.mts, tools/e2e-advisor/targets.mts, tools/pr-review-advisor/README.md, tools/pr-review-advisor/analyze.mts.
  • PRA-T3 Add runtime validation test for repo-confined tools against real Pi SDK — Add a test in test/advisor-repo-read-only-tools.test.ts that spins up a real Pi SDK session with createRepoConfinedReadOnlyTools() and attempts symlink escapes, Unicode-space collisions, and @/~ aliases — verifying the SDK never receives an outside path.
  • PRA-T4 Add runtime validation test for context-tool ordering with real provider — Add a test in test/advisor-session-runner.test.ts (or new integration test file) that runs a single-turn advisor with the real provider, omits a required context tool call, and verifies the turn fails with 'omitted required tool result(s)'.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Improvement — Add runtime validation test for repo-confined tools against real Pi SDK

  • Location: test/advisor-repo-read-only-tools.test.ts:1
  • Category: tests
  • Problem: Unit tests verify RepoPathGuard logic but do not exercise the actual Pi SDK read/grep/find/ls implementations with the confined wrappers. A real SDK integration test would verify the full confinement chain end-to-end.
  • Impact: Without real SDK integration test, a Pi SDK behavior change (e.g., new path alias, normalization difference) could bypass confinement without detection.
  • Suggested action: Add a test in test/advisor-repo-read-only-tools.test.ts that spins up a real Pi SDK session with createRepoConfinedReadOnlyTools() and attempts symlink escapes, Unicode-space collisions, and @/~ aliases — verifying the SDK never receives an outside path.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the new test and confirm it exercises Pi SDK's actual read/grep/find/ls code paths, not just the guard logic.
  • Missing regression test: Add test: 'confines real Pi SDK read/grep/find/ls via repo-confined wrappers' — create temp workspace with symlink to outside, real Pi SDK session, attempt escapes, assert all rejected with 'outside the workspace' or 'not stable under Pi SDK normalization'.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: testDepth verdict: runtime_validation_recommended; repo-read-only-tools.mts:137 lines new security-critical confinement layer; existing 184-line test file uses mocked SDK only

PRA-2 Improvement — Add runtime validation test for context-tool ordering with real provider

  • Location: test/advisor-session-runner.test.ts:1
  • Category: tests
  • Problem: Mocked SDK tests verify turn-tool resolution and flow errors, but do not exercise real model interaction with the required context-tool protocol (model must call each context tool before emitting text, exactly one terminal ledger mutation after).
  • Impact: Without real provider test, a model behavior change or SDK update could allow the model to skip required context tools or call them out of order without the runner detecting it.
  • Suggested action: Add a test in test/advisor-session-runner.test.ts (or new integration test file) that runs a single-turn advisor with the real provider, omits a required context tool call, and verifies the turn fails with 'omitted required tool result(s)'.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the new test against the real inference API and confirm the turn settles with status=failed and error containing 'omitted required tool result(s)'.
  • Missing regression test: Add test: 'real provider omits required context tool -> turn fails closed' — configure real provider, single turn with required context tool, prompt model to answer without calling it, assert settlement error matches omission.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: testDepth verdict: runtime_validation_recommended; session.mts:880 lines new context-tool runtime; existing 202-line runner test mocks Pi SDK entirely

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-code-quality

github-code-quality Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the main branch.

TypeScript / code-coverage/cli

The overall coverage in the codex/advisor-ledger... branch is 77%. The coverage in the main branch is 76%.

Show a code coverage summary of the most impacted files.
File main b1e35df codex/advisor-ledger... 854b898 +/-
src/lib/state/config-io.ts 96% 93% -3%
src/lib/securit...ntial-filter.ts 98% 95% -3%
src/lib/onboard/providers.ts 94% 92% -2%
src/lib/onboard.ts 28% 28% 0%
src/lib/inferen...board-probes.ts 86% 87% +1%
src/lib/inference/local.ts 69% 74% +5%
src/lib/sandbox/config.ts 54% 61% +7%
src/lib/sandbox...vileged-exec.ts 82% 89% +7%
src/lib/inferen...time-context.ts 91% 100% +9%
src/lib/inferen...del-registry.ts 66% 100% +34%

Updated July 09, 2026 08:58 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 0 suggestions · 1 test follow-up
Since last review: 0 prior items resolved · 0 still apply · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Add or identify a focused local integration test using the real Pi SDK session/tool-dispatch machinery, without contacting the model provider, that proves per-turn context tools and pr_review_update_ledger/pr_review_read_ledger are activated only for their turn and reset to read/grep/find/ls afterward.. Focused unit and mocked-boundary coverage is strong, and the current deterministic risk plan has no required E2E jobs. The remaining confidence gap is the shared advisor runner's integration with the real Pi SDK tool-dispatch path, because test/advisor-session-runner.test.ts mocks createAgentSession and setActiveToolsByName behavior.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
tools/pr-review-advisor/review-ledger.mts (1)

267-298: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

pr_review_read_ledger exposes resolved/superseded findings, relying on prompt compliance to filter.

ledgerResult serializes snapshot.findings unfiltered (all statuses) for the read tool. The synthesis/retry prompts instruct the model to "Include only status=open findings," but the tool itself doesn't enforce that — a model that forgets to filter will get caught downstream only by analyze.mts's reviewLedgerConsistencyIssues check, which then triggers a retry or hard failure. Filtering to status === "open" (already in insertion order) here would remove one class of avoidable retries/failures.

♻️ Proposed fix
   const read = defineTool({
     name: REVIEW_LEDGER_READ_TOOL,
     label: "Read review finding ledger",
     description: "Read the canonical finding ledger for final synthesis.",
     parameters: Type.Object({}),
     executionMode: "sequential",
-    execute: async () => ledgerResult(ledger.snapshot()),
+    execute: async () => ledgerResult(openOnlySnapshot(ledger.snapshot())),
   });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/pr-review-advisor/review-ledger.mts` around lines 267 - 298,
`pr_review_read_ledger` is returning all ledger findings, including resolved and
superseded entries, instead of only open ones. Update `ledgerResult` in the
review ledger tool flow to filter `snapshot.findings` to `status === "open"`
before serializing the JSON, so `REVIEW_LEDGER_READ_TOOL` only exposes current
findings and doesn’t depend on prompt-side filtering.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/pr-review-advisor-ledger-tools.test.ts`:
- Line 57: The test titles in PR review ledger tools are missing the required
local issue-reference suffix. Update the `describe` block in
`pr-review-advisor-ledger-tools.test.ts` and any related `it` titles to keep
them behavior-oriented while appending the appropriate `(`#1234`)` suffix at the
end of each title, following the existing test naming conventions.

In `@tools/pr-review-advisor/analyze.mts`:
- Around line 417-424: The fail-closed path in analyze.mts is too broad because
the retry decision in the parsed/result flow treats any retry failure as fatal
even when the first pass already produced a canonicalized result. Update the
logic around parseAdvisorResult, reviewLedgerConsistencyIssues,
withCanonicalReviewLedgerFindings, and the retry handling so process.exit(1)
only happens when the retry error itself indicates a ledger mismatch; for
unrelated retry errors, preserve the first-pass result and attach a
retry-failure limitation instead of discarding result.

---

Nitpick comments:
In `@tools/pr-review-advisor/review-ledger.mts`:
- Around line 267-298: `pr_review_read_ledger` is returning all ledger findings,
including resolved and superseded entries, instead of only open ones. Update
`ledgerResult` in the review ledger tool flow to filter `snapshot.findings` to
`status === "open"` before serializing the JSON, so `REVIEW_LEDGER_READ_TOOL`
only exposes current findings and doesn’t depend on prompt-side filtering.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: aa20f72b-a639-43d5-88eb-fc65898dbeb8

📥 Commits

Reviewing files that changed from the base of the PR and between 26bfdfb and de21f79.

📒 Files selected for processing (7)
  • test/advisor-session-context-tools.test.ts
  • test/pr-review-advisor-ledger-tools.test.ts
  • test/pr-review-advisor.test.ts
  • tools/advisors/session.mts
  • tools/pr-review-advisor/README.md
  • tools/pr-review-advisor/analyze.mts
  • tools/pr-review-advisor/review-ledger.mts

};
}

describe("PR review ledger tools", () => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Test titles are missing the (#1234) issue-reference suffix.

Titles are behavior-oriented, which is good, but neither the describe block nor any it includes a local issue reference. As per coding guidelines, "**/*.test.ts: Write behavior-oriented test titles, and put local issue references in a final (#1234) suffix."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/pr-review-advisor-ledger-tools.test.ts` at line 57, The test titles in
PR review ledger tools are missing the required local issue-reference suffix.
Update the `describe` block in `pr-review-advisor-ledger-tools.test.ts` and any
related `it` titles to keep them behavior-oriented while appending the
appropriate `(`#1234`)` suffix at the end of each title, following the existing
test naming conventions.

Source: Coding guidelines

Comment thread tools/pr-review-advisor/analyze.mts Outdated
Signed-off-by: Carlos Villela <cvillela@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
tools/e2e-advisor/analyze.mts (2)

292-334: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Tool-name list is duplicated between the array and the prompt string — drift risk.

The contextToolResults array (lines 293-328) and the hard-coded tool-name list in the prompt template (line 332) must be kept in sync manually. Adding/removing/reordering a context tool here requires editing both places; a missed edit would silently desync the instructed tool-call list from what's actually attached.

Consider deriving the prompt's tool-call list from the contextToolResults array's toolName fields (e.g., contextToolResults.map((r) => r.toolName).join(", ")) so the two can't drift.

As per path instructions, "Derive inventories and limits from a canonical source where possible; flag duplicated lists that can silently drift."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e-advisor/analyze.mts` around lines 292 - 334, The tool-name list in
the prompt is duplicated from the contextToolResults array, creating drift risk
if tools change. Update analyze.mts so the prompt’s required tool-call list is
derived from the same contextToolResults source (using the toolName values from
that array) rather than being hard-coded. Keep the prompt and attached tools in
sync through a single canonical list, especially around contextToolResult and
the prompt template in the E2E advisor builder.

Source: Path instructions


336-342: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

contextToolResult() helper duplicated across three files.

An identical contextToolResult(toolName, content, contentType, label) implementation exists here, in tools/e2e-advisor/targets.mts (lines 397-403), and in tools/pr-review-advisor/analyze.mts (lines 397-404, per graph context). Since AdvisorContextToolResult is already exported from tools/advisors/session.mts, this trivial constructor is a good candidate to canonicalize there once, rather than reimplementing it three times where each copy can silently diverge (e.g., one adding isError support and the others not).

As per path instructions for tools/{pr-review-advisor,e2e-advisor}/**, "Derive inventories and limits from a canonical source where possible; flag duplicated lists that can silently drift."

♻️ Proposed consolidation
-function contextToolResult(
-  toolName: string,
-  content: string,
-  contentType: AdvisorContextToolResult["contentType"],
-  label?: string,
-): AdvisorContextToolResult {
-  return { toolName, content, contentType, label };
-}
+// moved to tools/advisors/session.mts and exported, then imported here:
+// export function contextToolResult(...): AdvisorContextToolResult { ... }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e-advisor/analyze.mts` around lines 336 - 342, The contextToolResult
helper is duplicated in multiple analyzer files and should be centralized to
avoid silent drift. Move the shared constructor logic into the canonical
AdvisorContextToolResult utility in tools/advisors/session.mts, then update the
contextToolResult call sites in analyze.mts and targets.mts (including the
pr-review-advisor and e2e-advisor copies) to use that shared implementation
instead of maintaining local duplicates.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tools/e2e-advisor/analyze.mts`:
- Around line 292-334: The tool-name list in the prompt is duplicated from the
contextToolResults array, creating drift risk if tools change. Update
analyze.mts so the prompt’s required tool-call list is derived from the same
contextToolResults source (using the toolName values from that array) rather
than being hard-coded. Keep the prompt and attached tools in sync through a
single canonical list, especially around contextToolResult and the prompt
template in the E2E advisor builder.
- Around line 336-342: The contextToolResult helper is duplicated in multiple
analyzer files and should be centralized to avoid silent drift. Move the shared
constructor logic into the canonical AdvisorContextToolResult utility in
tools/advisors/session.mts, then update the contextToolResult call sites in
analyze.mts and targets.mts (including the pr-review-advisor and e2e-advisor
copies) to use that shared implementation instead of maintaining local
duplicates.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5402b943-e218-4475-8ad8-6c4aa22c7df7

📥 Commits

Reviewing files that changed from the base of the PR and between 4a421b1 and 628f0a3.

📒 Files selected for processing (14)
  • test/advisor-session-context-tools.test.ts
  • test/e2e-advisor-targets.test.ts
  • test/e2e-advisor.test.ts
  • test/pr-review-advisor-ledger-tools.test.ts
  • test/pr-review-advisor-turns.test.ts
  • test/pr-review-advisor.test.ts
  • tools/advisors/README.md
  • tools/advisors/session.mts
  • tools/e2e-advisor/README.md
  • tools/e2e-advisor/analyze.mts
  • tools/e2e-advisor/targets.mts
  • tools/pr-review-advisor/README.md
  • tools/pr-review-advisor/analyze.mts
  • tools/pr-review-advisor/review-ledger.mts
✅ Files skipped from review due to trivial changes (3)
  • tools/advisors/README.md
  • tools/e2e-advisor/README.md
  • tools/pr-review-advisor/README.md
🚧 Files skipped from review as they are similar to previous changes (5)
  • test/pr-review-advisor.test.ts
  • test/advisor-session-context-tools.test.ts
  • tools/pr-review-advisor/review-ledger.mts
  • tools/pr-review-advisor/analyze.mts
  • tools/advisors/session.mts

cv added 6 commits July 9, 2026 01:02
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv
cv merged commit 5ff7664 into main Jul 9, 2026
49 checks passed
@cv
cv deleted the codex/advisor-ledger-flow branch July 9, 2026 09:00
@cv

cv commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

Automated review follow-up

  • Linked issue context: resolved. Advisor attempt 2 collected feat(ci): add deterministic PR risk plans #6446, including its body and comments, and both reconciliation contexts contain issue 6446. GPT-5.5 now reports merge_as_is with no findings; Nemotron has no required items or warnings.
  • Pi SDK confinement runtime coverage: test/advisor-repo-read-only-tools.test.ts constructs read, grep, find, and ls through the real Pi SDK definition factories used in production and invokes their actual execute methods. It covers normal operation plus absolute paths, parent traversal, @/~ aliases, file and directory symlinks, Unicode-space normalization, and post-realpath normalization. Adding a session solely to dispatch those same definitions would primarily retest upstream SDK dispatch rather than the confinement boundary owned here.
  • Turn-scoped activation coverage: test/advisor-session-runner.test.ts mocks the external session boundary so it can deterministically emit omissions and failures, while executing the real generated context-tool closures and recording every setActiveToolsByName transition. It asserts per-turn rebinding, fail-closed omission/error handling, reset to read/grep/find/ls, and post-turn/session tool inertness. A real-provider omission test would be nondeterministic, contact an external API, and conflict with the repository rule to mock external dependencies. A provider-free Pi session by itself cannot exercise this runner's transitions without replacing the same prompt boundary or adding production injection solely for the test.

The optional runtime-test follow-ups are therefore covered at the narrow deterministic boundaries that this PR owns; no additional compatibility layer or live-provider test is being added.

@cv cv mentioned this pull request Jul 9, 2026
21 tasks
cv added a commit that referenced this pull request Jul 9, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Make the PR Review Advisor's six analysis stages observable and its
ledger writes atomic. This follows a census of every advisor run after
#6547: useful findings were being hidden by protocol failures,
especially when models omitted, retried, or mixed prose with ledger
calls.

## Related Issue

Follow-up to #6547 and #6446.

## Changes

- Split every stage into a visible analysis turn and a strict tool-only
commit turn, with one bounded repair attempt when no ledger mutation
settles.
- Add a reusable turn-protocol abstraction so stage ordering,
validation, and repair behavior are defined once.
- Preserve valid committed findings when a later stage fails, require
explicit source-of-truth finding IDs, and remove prose-derived synthetic
findings.
- Recognize multi-issue `Refs`/`References`/`Follow-up to` relations and
provide `rg` from a trusted runner binary or an exact package pin.
- Add regression coverage for omitted, failed, retried, malformed,
empty, and late-failure model behavior; update the advisor maintainer
documentation.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: this changes internal
PR-review automation; the two maintainer READMEs were updated, with no
end-user behavior change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — [GPT advisor
review](#6566 (comment))
identified the package provenance gap; the runner/package are now pinned
and enforced by a semantic workflow-boundary mutation test.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 131/131 focused advisor tests passed;
the final workflow boundary suite passed 8/8.
- [x] Applicable broad gate passed — the clean-checkout CI matrix and
aggregate `checks` gate passed. The earlier local `npm test` run
completed with 15,432 passed, 39 skipped, 1 todo, and 27
environment-only failures caused by the borrowed dependency symlink and
ambient local `umask`/`SSH_AUTH_SOCK`; all advisor tests passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with zero errors and only pre-existing warnings.
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: J. Yaunches <jyaunches@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Refactors PR Review Advisor so each stage starts with its instruction,
invokes real turn-scoped context tools, emits visible analysis, and then
commits findings to a canonical ledger. Final synthesis is read-only,
and the runner projects canonical ledger findings into the published
result so drift cannot silently change the review.

## Related Issue

Related issue NVIDIA#6446. This PR is a follow-up.

## Changes

- Replace pre-prompt synthetic messages with one canonical turn-scoped
context-tool contract and explicit ordering validation; no compatibility
aliases are retained.
- Add a shared finding ledger with stable IDs, strict atomic operation
batches, rollback, and evidence-backed transitions.
- Make synthesis and retry ledger-backed, expose only open findings to
the model, preserve the full audit ledger, preserve deterministic
runtime-test requirements against model downgrades, and fail closed only
on persistent ledger drift.
- Confine `read`, `grep`, `find`, and `ls` to SDK-normalized real paths
inside the checkout, and require exactly one successful terminal ledger
mutation per analysis turn.
- Document the revised conversation flow and add regression coverage for
PR and E2E advisor callers.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Internal maintainer
automation only; its internal README was updated and no user-facing
NemoClaw behavior changed.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent boundary
review reproduced and then verified fixes for absolute, parent, alias,
symlink, and post-realpath Unicode-normalization escapes; final review
found no remaining actionable defect.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project integration
test/advisor-repo-read-only-tools.test.ts
test/advisor-session-runner.test.ts
test/advisor-session-context-tools.test.ts
test/pr-review-advisor-ledger-tools.test.ts
test/pr-review-advisor-test-depth.test.ts test/pr-review-advisor.test.ts
test/pr-review-advisor-turns.test.ts test/e2e-advisor.test.ts
test/e2e-advisor-targets.test.ts
test/pr-review-advisor-workflow-boundary.test.ts
test/test-title-style.test.ts` (147 passed)
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — `umask 022; env -u SSH_CLIENT -u
SSH_CONNECTION -u SSH_TTY npm test` (15,328 passed; 39 skipped; 1 todo)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Make the PR Review Advisor's six analysis stages observable and its
ledger writes atomic. This follows a census of every advisor run after
NVIDIA#6547: useful findings were being hidden by protocol failures,
especially when models omitted, retried, or mixed prose with ledger
calls.

## Related Issue

Follow-up to NVIDIA#6547 and NVIDIA#6446.

## Changes

- Split every stage into a visible analysis turn and a strict tool-only
commit turn, with one bounded repair attempt when no ledger mutation
settles.
- Add a reusable turn-protocol abstraction so stage ordering,
validation, and repair behavior are defined once.
- Preserve valid committed findings when a later stage fails, require
explicit source-of-truth finding IDs, and remove prose-derived synthetic
findings.
- Recognize multi-issue `Refs`/`References`/`Follow-up to` relations and
provide `rg` from a trusted runner binary or an exact package pin.
- Add regression coverage for omitted, failed, retried, malformed,
empty, and late-failure model behavior; update the advisor maintainer
documentation.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: this changes internal
PR-review automation; the two maintainer READMEs were updated, with no
end-user behavior change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — [GPT advisor
review](NVIDIA#6566 (comment))
identified the package provenance gap; the runner/package are now pinned
and enforced by a semantic workflow-boundary mutation test.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 131/131 focused advisor tests passed;
the final workflow boundary suite passed 8/8.
- [x] Applicable broad gate passed — the clean-checkout CI matrix and
aggregate `checks` gate passed. The earlier local `npm test` run
completed with 15,432 passed, 39 skipped, 1 todo, and 27
environment-only failures caused by the borrowed dependency symlink and
ambient local `umask`/`SSH_AUTH_SOCK`; all advisor tests passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with zero errors and only pre-existing warnings.
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: J. Yaunches <jyaunches@nvidia.com>
@wscurran wscurran added the refactor PR restructures code without intended behavior change label Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactor PR restructures code without intended behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants