Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions .agents/skills/nemoclaw-maintainer-cross-issue-sweep/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
---
name: nemoclaw-maintainer-cross-issue-sweep
description: Scans other open issues to find ones a given PR may also fix or accidentally break. Outputs adjacent-fix opportunities and contradiction risks with file:line evidence. Use when reviewing a PR to discover bundling opportunities or downstream impact across the issue queue.
user_invocable: true
---

# Cross-Issue Regression Sweep

Surfaces the issues a single PR may also fix or accidentally break beyond the one it claims to address. Two outputs:

- **Adjacent fixes** — "PR may also close #X" → bundling intel (ship one PR, close multiple issues)
- **Contradicting risks** — "PR may break what #Y wants" → coordination needed before merge

## Prerequisites

- `gh` CLI authenticated
- A target repository with open issues
- An open PR to scan

## Repo policy

Defaults assume NemoClaw conventions. Edit `repo-policy.md` to override per-repo (bot logins, candidate caps, language regex).

## Workflow

Copy this checklist into your response and check off each step:

```text
Cross-issue sweep progress:
- [ ] Step 1: Extract fingerprint (files, symbols, error strings, primary issue)
- [ ] Step 2: Search candidate issues (capped at 30, primary excluded)
- [ ] Step 3: Classify each candidate (4-class with evidence)
- [ ] Step 4: Apply reverse-link boost
- [ ] Step 5: Filter (drop UNRELATED, SAME_ISSUE_DIFF, low-confidence)
- [ ] Step 6: Render report using templates/report.md
```

### Step 1: Extract fingerprint

```bash
scripts/extract-fingerprint.sh <pr-number>
```

Pulls four dimensions: touched files, touched symbols (per-language regex), error-string tokens, and the PR's primary linked issue (for exclusion). See `checks/fingerprint-extraction.md`.

### Step 2: Search candidate issues

```bash
scripts/search-candidate-issues.sh <fingerprint-json>
```

Three search dimensions, capped at 30 total candidates:

- Per symbol: top 10 by recency
- Per file path: top 5 by recency
- Per error string: top 5 by recency

Dedupes; excludes the PR's primary linked issue.

### Step 3: Classify each candidate

For each candidate, the LLM classifies as one of four classes per `checks/relationship-judgment.md`:

- **ADJACENT_FIX** — PR's changes likely also resolve this issue
- **CONTRADICTING** — PR's approach blocks what this issue wants
- **SAME_ISSUE_DIFF** — same root bug as PR's primary issue (dedup filter)
- **UNRELATED** — no meaningful relationship

Required for ADJACENT_FIX or CONTRADICTING:

- Cite specific PR diff line
- Cite specific issue symptom
- Confidence: high / medium / low

If no specific evidence can be cited, the LLM must answer UNRELATED. This floors hallucination.

### Step 4: Reverse-link boost

If the candidate issue's body or comments already mention this PR's number, the relationship is already in someone's mental model. Boost confidence by one tier (low → medium, medium → high).

### Step 5: Filter

- Suppress UNRELATED + SAME_ISSUE_DIFF
- Drop low-confidence judgments
- Keep ADJACENT_FIX and CONTRADICTING with high or medium confidence

### Step 6: Render report

```bash
scripts/render-report.py < classifications.json
```

See `templates/report.md` for the format.

## Reference files

- `repo-policy.md` — configurable per-repo defaults
- `relationship-rules.md` — 4-class definitions with worked examples
- `checks/fingerprint-extraction.md` — what to pull from the diff, per language
- `checks/relationship-judgment.md` — LLM judgment criteria + evidence requirement
- `templates/report.md` — output template
- `validation/backtest.md` — backtest the skill against historical PRs

## Scripts (execute, do not read)

- `scripts/extract-fingerprint.sh` — symbols + paths + error strings, deterministic
- `scripts/search-candidate-issues.sh` — GitHub Search wrapper, dedupe, cap
- `scripts/render-report.py` — report renderer

## Composition with other skills

The pr-comparator (`nemoclaw-maintainer-pr-comparator`) calls this skill as a sub-step when comparing competing PRs. Adjacent-fix counts feed Tier 3 tiebreakers; contradicting hits factor into Tier 2 quality scoring.

## What this skill does NOT do (deferred)

These would raise the ceiling but require infrastructure beyond GitHub API + LLM:

- Run PR code against adversarial inputs (sandboxed)
- Static-analyzer dataflow tracing (CodeQL, Semgrep)
- ML-based symbol disambiguation across codebases
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Fingerprint Extraction

What to pull from the PR diff and how. Run `scripts/extract-fingerprint.sh <pr>` to do this mechanically.

## Contents

- Touched files (paths)
- Touched symbols (per-language)
- Error-string tokens
- Primary linked issue

## Touched files

From `gh pr view <pr> --json files`. Filter to source paths (drop test fixtures, generated files, lockfiles).

**Why baseline:** Some issues mention a file path but no symbol. Without file matching, those go uncaught.

## Touched symbols

Per-language regex against added/modified lines in the diff. Defaults live in `repo-policy.md`.

**Why this is the killer angle:** Most matchers stop at file paths. A user issue saying "validateInput rejects empty strings" pinpoints a function — file-path matching alone misses it if the function is in a different module than expected.

**Filtering rules:**

- Only extract symbols from added/modified lines, not deleted lines (those are going away)
- Drop common short names (`do`, `if`, `as`) — too noisy
- Drop language keywords
- Drop test-helper names (`describe`, `it`, `test`) — they match too many issues

## Error-string tokens

Strings inside:

- `throw new Error("...")` / `throw Error("...")`
- `console.error("...")`
- `print(f"...")` / Python f-strings flagged with error-shape (`Error:`, `Failed`)
- Distinctive flag/option names (`--no-color`, `--verbose`)

**Why this catches symptoms:** When a user files an issue, they often paste the error message they saw. That string is high-info and rarely false-matches.

**Filtering rules:**

- Skip strings <8 chars (too generic)
- Skip strings with no alpha chars
- Strip placeholders (`%s`, `${var}`, `{0}`) before searching

## Primary linked issue

From the PR body, parse:

- `closes #N` / `fixes #N` / `resolves #N`
- `Linked Issue: #N` block

Captured for **exclusion** during search. Without this, every PR self-matches its own issue.

## Output

Fingerprint JSON shape:

```json
{
"pr": 2851,
"files": ["src/lib/shields.ts", "Dockerfile.base"],
"symbols": ["normalize_mutable_config_perms", "applyStateDirLockMode"],
"error_strings": ["EACCES on .openclaw"],
"primary_issue": 2681
}
```

Consumed by `scripts/search-candidate-issues.sh`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Relationship Judgment

How the LLM classifies each candidate issue. The judgment is the only non-deterministic step in the pipeline; everything else is mechanical search.

## Contents

- Inputs to the LLM
- The prompt
- Evidence requirement
- Confidence levels
- Reverse-link boost

## Inputs to the LLM (per candidate)

- PR diff (truncated to 3000 chars if larger)
- PR description body
- PR's primary linked issue number (for context — used by SAME_ISSUE_DIFF check)
- Candidate issue number, title, body (truncated to 2000 chars)
- Candidate issue's first ~5 comments (for symptom context)

## The prompt

```text
Judge whether this PR's changes affect the open issue.

PR #{pr_number}: {pr_title}
PR description: {pr_body}
PR diff (relevant slice): {diff}

Candidate issue #{issue_number}: {issue_title}
Issue body: {issue_body}

PR's primary linked issue: #{primary_issue}

Classify the relationship:

- ADJACENT_FIX: PR's changes resolve this issue OR open a clear follow-on path
on the same code the PR just touched
- CONTRADICTING: PR's approach makes this issue's desired behavior impossible,
OR the PR's scope is incomplete and the issue reports the leftover gap
- SAME_ISSUE_DIFF: same root bug as #{primary_issue} (dedupe filter)
- UNRELATED: no meaningful relationship

For ADJACENT_FIX or CONTRADICTING, REQUIRED — cite ONE of these evidence shapes:

(a) DIRECT: cite specific PR diff line(s) (file:line) AND specific issue
symptom(s) that map to those lines

(b) BY-OMISSION (partial-fix detection): cite the PR's diff *scope* — what
class of bug it addressed — AND the issue's symptom showing the same
class but a different instance the PR did NOT touch. Required: name the
bug class, name the instances PR fixed, name the instances issue
reports as still broken.

(c) FOLLOW-ON: cite the symbol/file the PR introduced or modified AND the
issue's request to harden the same symbol/file (e.g., "PR introduced
rcf_patch.py; issue requests rcf_patch.py be hardened against X").

Confidence: high / medium / low

If you cannot cite specific evidence under any of (a), (b), (c), answer UNRELATED.
```

## Evidence requirement (anti-hallucination)

For any ADJACENT_FIX or CONTRADICTING verdict, the LLM must cite evidence under one of three shapes:

- **Direct**: specific PR diff line + specific issue symptom that map to each other
- **By-omission**: PR's diff *scope* (the bug class it addressed) + issue symptom showing the same class but a different instance the PR did NOT touch (catches partial-fix patterns)
- **Follow-on**: the symbol/file the PR introduced + the issue's request to harden the same symbol/file (catches "PR introduced X, now harden X" follow-up patterns)

Without one of these citations, the answer must be UNRELATED.

This rule is the single most important defense against hallucinated matches. The three shapes give the LLM legitimate paths to flag genuine relationships without lowering the bar to vague "they touch the same area" matches.

## Confidence levels

- **high**: clear semantic match between cited PR change and cited issue symptom
- **medium**: plausible match but partial evidence (e.g., the change touches the right area but doesn't directly fix the cited symptom)
- **low**: weak inference; below the default `confidence_floor` from `repo-policy.md` and gets dropped

## Reverse-link boost

If the candidate issue's body or comments already mention this PR's number (e.g., "fixed by PR #2851"), the relationship is already in someone's mental model. Boost confidence one tier:

- low → medium (rescues a borderline match)
- medium → high (cements a likely match)
- high → unchanged (already at ceiling)

Implementation: after the LLM's classification, the orchestrator checks the candidate issue body and comments for the PR number. If found, applies the boost.

## Why this beats naive token-overlap

Naive token-overlap finds candidates but produces high false-positive rates. Two filters separate signal from noise:

1. **LLM judgment** distinguishes "function name appears in issue" from "function's behavior is what the issue describes"
2. **Evidence requirement** forces the LLM to commit to specific lines, not vague hand-waving

The reverse-link boost handles the case where humans have already noticed the relationship — that's strong prior signal the skill should respect.
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Relationship Classification Rules

The four classes the LLM assigns to each candidate issue, with worked examples.

## Contents

- ADJACENT_FIX
- CONTRADICTING
- SAME_ISSUE_DIFF
- UNRELATED

## ADJACENT_FIX

The PR's changes resolve this issue OR open a clear follow-on path on the same code the PR just touched.

### Example A — incidental closure (direct evidence)

PR description: "fix EACCES when shields-down user writes config"
PR diff: adds `chmod g+w` to `.openclaw` directory at startup
Candidate issue #2810: "Telegram preset writes fail intermittently after sandbox rebuild"
Issue body cites: "EPERM on `.openclaw/credentials/telegram.json`"

**Classification:** ADJACENT_FIX, high confidence
**Evidence cited (direct):** PR diff `Dockerfile.base:97` (chmod g+w on .openclaw); issue body line 14 ("EPERM on .openclaw/credentials/telegram.json"). Same root cause, same fix.

### Example B — follow-on hardening on PR-introduced code

PR #2696 introduced `scripts/rcf_patch.py` with regex-based property matching.
Candidate issue #2875: "Harden rcf_patch.py against property-order drift" — issue says PR #2696 is "a real improvement... one follow-on hardening gap: the regex still assumes `snapshot` before `nextConfig`."

**Classification:** ADJACENT_FIX, high confidence (boosted by reverse-link)
**Evidence cited (follow-on):** PR introduced `scripts/rcf_patch.py`; issue requests hardening the same file's regex against a specific drift case. The PR's code is the subject of the issue's hardening request, not a separate concern.

## CONTRADICTING

The PR's approach makes this issue's desired behavior impossible, OR the PR's scope is incomplete and the issue reports the leftover gap.

### Example A — direct contradiction

PR description: "remove silent EACCES swallow from Patch 4b"
PR diff: deletes try/catch around `mutateConfigFile`
Candidate issue #4187: "Allow opt-in error suppression for sandbox config writes during shutdown"

**Classification:** CONTRADICTING, medium confidence
**Evidence cited (direct):** PR diff removes `try { ... } catch { /* swallow */ }` at `Dockerfile:142`; issue body line 8 explicitly requests "opt-in suppression for shutdown-time write failures." PR strictly rejects what issue requests.

### Example B — partial-fix gap (evidence by omission)

PR #2700 changed 5 env-var validations from `return 1` to `return 0` in `scripts/nemoclaw-start.sh`.
Candidate issue #2762: "PR #2700 changed validations... However... NEMOCLAW_CONTEXT_WINDOW and NEMOCLAW_MAX_TOKENS with invalid values still cause the container to exit with code 1."

**Classification:** CONTRADICTING, high confidence (boosted by reverse-link)
**Evidence cited (by-omission):** Bug class — env-var validation hard-exits under `set -euo pipefail`. PR fixed instances `NEMOCLAW_MODEL_OVERRIDE`, `NEMOCLAW_INFERENCE_API_OVERRIDE`, `NEMOCLAW_REASONING`, `NEMOCLAW_CORS_ORIGIN`, plus one more. Instances PR did NOT touch: `NEMOCLAW_CONTEXT_WINDOW`, `NEMOCLAW_MAX_TOKENS`. Issue reports same hard-exit class on those exact untouched instances. The PR's incomplete scope is the contradiction with the issue's expectation of a class-level fix.

## SAME_ISSUE_DIFF

The candidate issue describes the same root bug as the PR's primary linked issue. Suppress to avoid double-counting.

**Example:**

PR's primary issue: #2681 ("Enable Dreaming permission error")
Candidate issue #2895: "Toggle in OpenClaw UI fails with EACCES"

Both describe the same EACCES failure on the same toggle. The candidate is a duplicate of the primary issue. **Classification:** SAME_ISSUE_DIFF (suppressed from output).

## UNRELATED

No meaningful relationship. The candidate showed up in search because of token overlap but doesn't align with the PR's actual changes.

**Example:**

PR description: "extract sandbox-gateway-state helpers"
Candidate issue #4523: "Sandbox gateway timeout on first connect"

Search matched on "gateway." But the PR is a pure refactor (no behavior change), and the issue is about timing. **Classification:** UNRELATED.

## Decision rule

If the LLM cannot cite a specific PR diff line **and** a specific issue symptom that map to each other, the answer must be UNRELATED. This prevents hallucinated matches.
Loading
Loading