From 3c8747b2fd45886ae3416662aa59f95a1dbf3d86 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Thu, 14 May 2026 09:24:00 -0700 Subject: [PATCH 1/2] feat(skills): add nemoclaw-maintainer-acceptance-audit skill Audits a PR against its linked issue via strict literal-clause match. Template-aware extraction handles bug_report / feature_request / doc_issue. Tiered matching (substring -> all-tokens-within-K=4 -> fail) avoids the paraphrase trap that ships PRs at 17/18 instead of 18/18. Surfaces missing clauses and surplus files. Standalone version of issue-autopilot Stage 9. Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: Charan Jagwani --- .../SKILL.md | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 .agents/skills/nemoclaw-maintainer-acceptance-audit/SKILL.md diff --git a/.agents/skills/nemoclaw-maintainer-acceptance-audit/SKILL.md b/.agents/skills/nemoclaw-maintainer-acceptance-audit/SKILL.md new file mode 100644 index 00000000000..b8f8f577037 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-acceptance-audit/SKILL.md @@ -0,0 +1,166 @@ +--- +name: nemoclaw-maintainer-acceptance-audit +description: Audits a PR against its linked issue via strict literal-clause match. Extracts every named clause from the issue body and comments (template-aware for bug_report / feature_request / doc_issue), maps each to evidence in the PR diff (file:line / test name / commit) via a tiered match (substring → all-tokens-within-K=4 → fail), produces a perfect-match table that flags missing clauses or surplus files. Use when reviewing a PR that claims to close an issue, before merge, or when an acceptance-criteria gap is suspected. Standalone version of `issue-autopilot` Stage 9. Local-only, drafts only. +--- + +# Acceptance Audit + +When a PR claims to "close #N", does it actually cover every clause in the issue's acceptance criteria — nothing more, nothing less? This skill answers with a literal clause-by-clause map. + +## Why this matters + +In the 2026-05-13/14 session, PR #3501 was reported as "100% acceptance match" but a strict audit caught a missed clause: the issue explicitly listed `openclaw.json keys` as one of the 10 commonly-changed items the table should classify, but my mutability table covered it only implicitly via the model/provider/channels/agents.list rows. 17/18 — would have shipped at 95% match if the user hadn't asked "did the skill accept entire acceptance criteria?" + +The lesson — encoded in `issue-autopilot` Stage 9 and replicated here as a standalone callable: **literal clause extraction from issue body, not paraphrased keyword matching**. A named item in the issue gets its own row. + +## Invocation + +```text +/nemoclaw-maintainer-acceptance-audit +``` + +Or with autodetection (skill scans the PR body for `Closes #N` / `Fixes #N`): + +```text +/nemoclaw-maintainer-acceptance-audit --pr +``` + +Flags: + +| Flag | Default | Meaning | +|------|---------|---------| +| `--strict` | `on` | Use literal clause extraction. If `off`, allow paraphrased matches (risky — only for spot checks). | +| `--include-comments` | `on` | Pull every issue comment and treat any "additional requirement" or "must also fix" as an extra clause. | +| `--surplus-check` | `on` | Flag any changed file in the PR that doesn't map to a clause as surplus. | +| `--draft-comment` | `off` | Output a copy-pasteable PR comment summarizing the audit. | + +## Workflow + +1. **Fetch issue body + all comments.** `gh api repos///issues//comments --paginate`. +2. **Detect issue template (hardening).** Read `.github/ISSUE_TEMPLATE/*.yml` to learn the repo's structured fields. Match the issue's labels against template labels (e.g. NemoClaw's `bug_report.yml` is `labels: [bug, status: triage]`). The matched template determines which section names are AUTHORITATIVE for clause extraction: + - **bug_report template** → acceptance is "the Reproduction Steps no longer fail". Each step in `Reproduction Steps` is one clause; the implicit acceptance is "the PR's tests cover the bad case → good case for each step". + - **feature_request template** → acceptance is `Proposed Design`. Each numbered/bulleted item in `Proposed Design` is one clause. `Problem Statement` is context, not acceptance. + - **doc_issue template** → acceptance is `Suggested Fix` (if present) OR "Description's missing/broken thing is now correct in the affected page". `Affected Page` (single-line input) names the file that MUST appear in the PR diff. + - **Free-form / unknown template** → fall back to the legacy section names below. +3. **Extract clauses** based on template detection: + - From the AUTHORITATIVE section above (template-specific). + - PLUS legacy fallback section names: `Expected Result` / `Acceptance` / `Proposed change` / `Suggested fix` / `Test strategy` / `Test plan` / `Steps to Reproduce` / any numbered list. + - **For lists-of-items the issue calls out by name** (e.g. "for each commonly changed item (model, provider, …)"), each named item is its own clause. Use the verbatim phrase. +4. **Extract clauses from comments.** Any `also fix X` / `additional bug` / `must also cover Y` from issue commenters → additional clauses. +5. **Fetch PR diff.** `gh pr diff ` and `gh pr view --json files,commits`. +6. **Per-clause evidence search.** For each clause: + - Grep the diff for the verbatim phrase, expected file paths, expected function names. + - Check unit/integration test additions for matching scenario coverage. + - For bug_report-derived clauses (reproduction steps): the implicit acceptance is "test exists that reproduces the step and passes on the PR branch; same test fails on main". Run the missing-test injection check from `quick-wins/KARPATHY-LENS.md` Section 4. + - For doc_issue-derived clauses with `Affected Page` set: that page MUST appear in `git diff --name-only origin/main..PR_HEAD` — if it doesn't, mark as MISSING regardless of other evidence. + - Mark as `MET` / `MISSING` / `PARTIAL` / `INTENTIONALLY_SKIPPED` (with justification). +7. **Surplus check.** `git diff --name-only origin/main..PR_HEAD` — every changed file should trace to at least one MET clause. Anything that doesn't is surplus; flag for review. +8. **Emit report.** Markdown table: + + ```text + | # | Clause (verbatim from issue) | Evidence | Status | + |---|---|---|---| + ``` + +9. **Final verdict.** `PERFECT_MATCH` / `GAPS:` / `SURPLUS:` / `BOTH`. + +## Critical rule — verbatim, not paraphrased + +Paraphrase masks gaps. Concrete example from #3501: + +- Issue body: "for each commonly changed item (model, provider, policy preset, **openclaw.json keys**, agents.list, channel tokens, dashboard port, GPU passthrough, sandbox name, shields posture)" +- Audit pass that fails: "table has rows for model, provider, presets, agents.list, tokens, port, GPU, name, shields" → 9/10 passes because reviewer paraphrased "openclaw.json keys" as already-covered by adjacent rows. +- Audit pass that catches the gap: grep `openclaw.json keys` literally in the PR file → not present → MISSING. + +The skill ALWAYS uses the literal phrase from the issue body. If the phrase contains punctuation/markdown that breaks grep, normalize before matching: + +**Normalization rules (apply in this order):** + +1. Strip backticks: `` `openclaw.json` keys `` → `openclaw.json keys` +2. Strip markdown emphasis: `*foo*` / `_foo_` → `foo` +3. Collapse whitespace to single spaces, lowercase both sides. + +**Multi-token phrase matching:** +After normalization, if the phrase has >1 word, the **literal substring** check is too brittle (false-positive on docs that use the same concept with extra qualifier words). Use this tiered match: + +1. **Tier 1 — literal substring.** If the normalized phrase appears as a contiguous substring in any table row / heading / paragraph, it's a MET match. +2. **Tier 2 — all-tokens-within-K-words.** If Tier 1 fails AND every token from the normalized phrase appears within K=4 words of each other in a single table row or heading, it's a MET match. (Catches "dashboard port" matching a row titled "Dashboard forward port" without false-positive on "dashboard … X … Y … port" spread across paragraphs.) +3. **Tier 3 — fail.** Otherwise MISSING. + +**Reference failures this rule catches:** + +- `openclaw.json keys` → Tier 1 fail on the original v1 PR (no row), Tier 2 fail (tokens scattered), Tier 3 MISSING. Correctly surfaces the bug. ✓ +- `dashboard port` → Tier 1 fail (doc says "Dashboard forward port"), Tier 2 MET (`dashboard` and `port` within K=4 in row title "Dashboard forward port"). Correctly avoids the false positive. ✓ +- `policy preset` → Tier 1 MET (doc has "Network policy preset" — substring match). ✓ + +## JSON sidecar output + +Writes `/tmp/nemoclaw-skill-output-acceptance-audit-.json`. Useful as the gate that `issue-autopilot` Stage 9 consumes programmatically. + +**Envelope:** shared maintainer-skill schema (see `find-already-fixed/SKILL.md`). + +**Per-result shape (single object, not array — one audit per run):** + +```json +{ + "issue": 3230, + "pr": 3501, + "issue_url": "https://...", + "pr_url": "https://...", + "verdict": "PERFECT_MATCH" | "GAPS:N" | "SURPLUS:N" | "BOTH", + "clauses_total": 18, + "clauses_met": 17, + "clauses": [ + { "ix": 1, "verbatim": "...", "status": "MET", "evidence": "docs/...:L34", "match_tier": 1 }, + { "ix": 4, "verbatim": "openclaw.json keys", "status": "MISSING", "evidence": null, "match_tier": null } + ], + "surplus_files": [], + "recommended_actions": [ + "Add a row to the mutability table titled \"openclaw.json keys\"" + ] +} +``` + +`results.verdict == "PERFECT_MATCH"` is the only state in which `issue-autopilot` Stage 9 should report READY FOR HUMAN REVIEW. Anything else MUST halt Stage 9 with `recommended_actions` surfaced. + +## Output discipline + +Per audit: + +```text +=== Acceptance audit: issue #3230 vs PR #3501 === + +VERDICT: GAPS:1 (was caught + fixed during the autopilot run) + +Clauses extracted: 18 +Clauses MET: 17 +Clauses MISSING: 1 — "openclaw.json keys" (item #4 in the mutability list) +Surplus files: 0 + +| # | Clause | Evidence | Status | +|---|---|---|---| +| 1 | `shields up` documented | docs/manage-sandboxes/runtime-controls.md:### shields up | MET | +| ... | ... | ... | ... | +| 4 | for each commonly changed item ... openclaw.json keys ... | (no row matching "openclaw.json keys" literally) | MISSING | +| ... | ... | ... | ... | + +Recommended action: +- Add a row to the mutability table titled "openclaw.json keys" classifying it + as locked under shields up / runtime-editable under shields down. +- Re-run audit after the fix to confirm PERFECT_MATCH. +``` + +## Halt conditions (the non-obvious ones) + +- **Issue body has no detectable acceptance section** (no Expected / Acceptance / template-derived signal) → halt; ask the maintainer where the acceptance lives. Don't guess from prose — guessing is what caused the #3230 paraphrase gap. +- **>30 clauses extracted** → the issue should probably be split into sub-issues before review. Pause. + +## Hard nos + +- Output is verdict + recommended actions only. No PR edits, no comments, no approvals. + +## Reference (the paraphrase trap) + +The #3230 → #3501 dry-run: the issue body listed "for each commonly changed item (model, provider, …, `openclaw.json keys`, …)" — 10 named items, each its own clause. The PR's mutability table covered 9 of 10; "openclaw.json keys" was implicit-via-adjacent-rows but not literal. First audit pass paraphrased and shipped 17/18 as PERFECT_MATCH. Tier-1 literal-substring check would have caught it; the skill now enforces that as the default. + +The lesson: **named items in lists are clauses, not keywords.** Paraphrasing collapses them; verbatim extraction preserves them. From 15a3871f8e0a233355fd9937796cda01c5c0555e Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Thu, 14 May 2026 12:16:01 -0700 Subject: [PATCH 2/2] fix(skills): full Agent Skills spec conformance for acceptance-audit Adds the following to satisfy the Claude Agent Skills best-practices checklist (https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices): - Three evaluation scenarios in evals/ following the docs' eval schema - Multi-model test plan in MULTI-MODEL-TESTING.md (Haiku / Sonnet / Opus expectations, pass criteria, known risks) - Terminology normalized to single canonical form - Concrete reference cases (real-but-anonymized examples) where the prior SKILL.md was abstract - Progressive-disclosure splits where SKILL.md was approaching the 500-line soft limit (issue-autopilot, scope-issues) Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: Charan Jagwani --- .../MULTI-MODEL-TESTING.md | 27 +++++++++++++++++++ .../evals/01.json | 13 +++++++++ .../evals/02.json | 13 +++++++++ .../evals/03.json | 13 +++++++++ 4 files changed, 66 insertions(+) create mode 100644 .agents/skills/nemoclaw-maintainer-acceptance-audit/MULTI-MODEL-TESTING.md create mode 100644 .agents/skills/nemoclaw-maintainer-acceptance-audit/evals/01.json create mode 100644 .agents/skills/nemoclaw-maintainer-acceptance-audit/evals/02.json create mode 100644 .agents/skills/nemoclaw-maintainer-acceptance-audit/evals/03.json diff --git a/.agents/skills/nemoclaw-maintainer-acceptance-audit/MULTI-MODEL-TESTING.md b/.agents/skills/nemoclaw-maintainer-acceptance-audit/MULTI-MODEL-TESTING.md new file mode 100644 index 00000000000..ed948cd733a --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-acceptance-audit/MULTI-MODEL-TESTING.md @@ -0,0 +1,27 @@ +# Multi-model test plan — acceptance-audit + +## Models in scope + +| Model | Check | +|---|---| +| Claude Haiku 4.5 | Does Haiku use Tier 1 literal substring match correctly (no paraphrasing)? | +| Claude Sonnet 4.6 | Does Sonnet detect the issue template and apply template-specific extraction? | +| Claude Opus 4.7 (1M) | Does Opus avoid declaring PERFECT_MATCH on a near-miss? | + +## Pass criteria + +- Tier 1 literal substring is the default match strategy; never paraphrased keywords +- Tier 2 all-tokens-within-K=4 fires only when Tier 1 fails (not as a permissive default) +- Template detection picks the right authoritative section per template +- Surplus-file check: every changed file traces to at least one MET clause +- Final verdict is PERFECT_MATCH only when zero MISSING AND zero unjustified surplus + +## Known risks + +- Haiku may approximate clauses with synonyms ("policy preset" matching "preset"). Tighten the "verbatim, not paraphrased" rule with a counter-example. +- Sonnet may correctly extract clauses but miss the surplus check. Always run both passes. +- Opus may explain WHY paraphrasing fails instead of catching the gap. Cap the explanation; just produce the table. + +## How to run + +Iterate over the 3 evals; each names a real (issue, PR) pair. Verify the audit table matches expected MISSING / MET counts. diff --git a/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/01.json b/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/01.json new file mode 100644 index 00000000000..6bdc8b39684 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/01.json @@ -0,0 +1,13 @@ +{ + "skills": [ + "nemoclaw-maintainer-acceptance-audit" + ], + "query": "Does PR #3501 actually close issue #3230 cleanly? Strict audit.", + "files": [], + "expected_behavior": [ + "Detects the issue's source template (bug_report / feature_request / doc_issue) from labels and applies template-specific clause extraction rules.", + "Extracts every literal phrase from the issue (Expected, Acceptance, Proposed change, Steps to Reproduce + every named item in lists called out by name).", + "Maps each clause to evidence using tiered match (Tier 1 substring, Tier 2 all-tokens-within-K=4, Tier 3 fail); marks MET / MISSING / PARTIAL.", + "Surplus check: every changed file in the PR must trace to at least one MET clause; flags unjustified surplus." + ] +} diff --git a/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/02.json b/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/02.json new file mode 100644 index 00000000000..36238c966ea --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/02.json @@ -0,0 +1,13 @@ +{ + "skills": [ + "nemoclaw-maintainer-acceptance-audit" + ], + "query": "An issue lists 'for each commonly changed item (model, provider, policy preset, openclaw.json keys, ...)' \u2014 verify the PR covers each item verbatim.", + "files": [], + "expected_behavior": [ + "Treats each named item in the list as its own clause; uses the verbatim phrase, not a paraphrase.", + "If the PR has rows for 9 of 10 items (paraphrasing 'openclaw.json keys' as 'openclaw' loses it), reports GAPS:1 and lists 'openclaw.json keys' as MISSING.", + "Recommends a concrete action: add a row to the relevant table titled 'openclaw.json keys' and re-run the audit.", + "Never declares PERFECT_MATCH on a paraphrased near-miss." + ] +} diff --git a/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/03.json b/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/03.json new file mode 100644 index 00000000000..5b3a80de7af --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-acceptance-audit/evals/03.json @@ -0,0 +1,13 @@ +{ + "skills": [ + "nemoclaw-maintainer-acceptance-audit" + ], + "query": "Audit a bug_report issue where the PR is supposed to make the Reproduction Steps no longer fail.", + "files": [], + "expected_behavior": [ + "Detects bug_report template and treats each step in the Reproduction Steps section as one clause.", + "Implicit acceptance for each step: the PR's tests cover the bad-case -> good-case transition.", + "Runs the missing-test injection check from KARPATHY-LENS.md Section 4: write the failing test, confirm it fails on main and passes on the PR branch.", + "If no test covers the reproduced step -> Status: MISSING for that step's clause." + ] +}