Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .changeset/pr-review-feedback-detection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
"@blacklite/crew-cli": minor
"@blacklite/crew-sdk": minor
---

PR review feedback detection in Ralph's work-check cycle, and thread-resolution rules for agents.

Ralph's PR scan reads `reviewDecision`, which reports `CHANGES_REQUESTED` and `APPROVED` — but a `COMMENTED` review leaves `reviewDecision: null`, and so does an inline review thread. A reviewer could ask for a change on a specific line and Ralph would report the PR as clean, so requested changes sat unread on work that was otherwise ready to merge.

- **New Ralph category (`ralph-reference.md` → "PR Review Feedback Detection").** Keyed on `reviewThreads(first:N) { isResolved }`, plus `COMMENTED` reviews and PR conversation comments. One GraphQL request covers every configured repo via aliases; repos come from `commentWatch.pullRequestRepos` in `.crew/config.json`, since review feedback spans the repos the crew opens PRs in rather than the single issue tracker. Sorts second in the priority order — after human comments, but ahead of approved PRs, so Ralph cannot merge past outstanding feedback.
- **No `seen=` on review threads.** The issue-side machinery exists because GitHub tracks nothing about whether a comment was answered. Review threads carry `isResolved`, which is exactly that, so they need no marker-based high-water mark, no adoption cutoff, and have no retroactive problem. Adding `seen=` there would create a second mark that can disagree with GitHub's. `seen=` still applies to top-level `COMMENTED` review bodies and PR conversation comments, which have no resolution state.
- **Asymmetric bot filter.** Quality and CI bots post a top-level review or PR comment on effectively every PR, but open an inline thread on roughly one PR in sixty. Inline threads therefore keep every author (a bot pointing at a real line is real feedback); top-level review bodies and PR comments drop `author.__typename == "Bot"`, or the category is permanently lit and becomes noise. Filtering on `__typename` rather than a login allowlist needs no per-installation maintenance.
- **Answered-but-unresolved (`issue-lifecycle.md`).** The `<!-- crew:agent={member} -->` marker now also goes on inline thread replies, where it does a different job: it lets Ralph demote a thread whose newest comment is a signed agent reply to `awaiting confirmation` instead of re-routing it as new work. A further reviewer reply returns the thread to the queue automatically.
- **Agents reply to review threads; agents do not resolve them.** New "Resolving review threads" section. Resolving is the natural acknowledgement and keeps the queue clean, but an agent resolving a thread it only partly addressed makes that feedback disappear silently — the exact failure this mechanism exists to prevent — while leaving it open merely nags. The queue-cleanliness argument is answered by the `awaiting confirmation` state instead, so the reviewer keeps the resolve button. Explicit operator instruction remains an exception.
25 changes: 23 additions & 2 deletions .crew-templates/crew.agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -872,8 +872,20 @@ of the newest human comment read — this is the high-water mark Ralph uses:
<!-- crew:agent=link seen=2026-07-29T03:06:23Z -->
```

The marker also goes on **replies to inline PR review threads**. `seen=` does **not** — a
review thread has `isResolved`, which is already the high-water mark, and a second one
would compete with it. `seen=` stays for issue comments, top-level `COMMENTED` review
bodies and PR conversation comments, none of which have resolution state.

**Agents reply to review threads; agents do not resolve them.** Resolving is the
reviewer's call. An agent that resolves a thread it only partly addressed makes that
feedback disappear silently, which is exactly the failure this whole mechanism exists to
prevent; leaving it open merely nags. Reply — including when the agent disagrees, saying
so plainly — and let the reviewer close it.

Include this instruction in every spawn prompt for an agent that may comment on an issue
or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing".
or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing" and
"Resolving review threads".

---

Expand All @@ -891,7 +903,16 @@ human reply can supersede a recommendation the crew is already acting on. Ralph
reviewer must state explicitly whether the reply supersedes an earlier recommendation, and
must close the loop with a `seen=` acknowledgement.

**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, watch mode, state model, board format, and follow-up integration.
Ralph's scan also covers **PR review feedback** across every repo listed in
`commentWatch.pullRequestRepos` — inline review threads with `isResolved: false`, plus
`COMMENTED` reviews and PR conversation comments from non-`Bot` authors with no
`<!-- crew:agent= -->` marker. `reviewDecision` reports none of these, so nothing else in
the scan catches them. These sort **second**, ahead of approved PRs, so Ralph cannot merge
past outstanding feedback. Ralph surfaces (repo, PR, `file:line`, gist) and routes to the
PR author agent to *address* — the point is that requested changes get actioned, not just
noticed.

**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, PR review feedback detection, watch mode, state model, board format, and follow-up integration.

### Connecting to a Repo

Expand Down
50 changes: 47 additions & 3 deletions .crew-templates/issue-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,9 +343,16 @@ issue or a PR:
- **Anything without the marker is, by definition, a human comment.** That inverse is
the point: the marker is what makes human replies detectable at all.

Applies to `gh issue comment`, `gh pr comment`, `gh pr review --body`, the equivalent
GitHub MCP tools, and any workflow that posts on an agent's behalf. It does **not**
apply to commit messages, PR bodies, or issue bodies — comments only.
Applies to `gh issue comment`, `gh pr comment`, `gh pr review --body`, **replies to
inline PR review threads** (`gh api .../comments`, `add_reply_to_pull_request_comment`),
the equivalent GitHub MCP tools, and any workflow that posts on an agent's behalf. It
does **not** apply to commit messages, PR bodies, or issue bodies — comments only.

The marker is unchanged on review threads, but it does a different job there. On an issue
it makes human comments detectable. On a review thread — where GitHub's `isResolved`
already tracks outstanding-ness — it is what lets Ralph tell "an agent has replied here"
from "nobody has", so an addressed thread stops being re-routed as new work while still
staying open for the reviewer. See `ralph-reference.md` → "Answered-but-unresolved".

### Acknowledging a human reply — the `seen=` field

Expand All @@ -365,13 +372,50 @@ that question.
Acknowledge honestly. `seen=` asserts "an agent read this and responded to it". It is
not a dismissal button.

**`seen=` does not apply to inline review threads.** Those carry `isResolved`, so a
`seen=` field there would be a second high-water mark competing with GitHub's own. Use
the bare `<!-- crew:agent={member} -->` marker on thread replies. `seen=` still applies
to top-level `COMMENTED` review bodies and PR conversation comments, which have no
resolution state of their own.

### Resolving review threads

**Agents reply to review threads. Agents do not resolve them.** Resolution stays with
the reviewer.

The tempting argument for the other choice is real: resolving is the natural
acknowledgement, and it keeps the queue clean. But compare how the two options fail.

- An agent resolves a thread it only partly addressed → the feedback disappears from
every view the reviewer uses. The failure is **silent**, and nothing surfaces it later.
- An agent leaves an addressed thread open → the thread stays visible until someone
confirms. The failure is **noisy**, and the reviewer sees it immediately.

For a mechanism whose entire purpose is that feedback stops sitting unread, a
silent-drop failure mode defeats the thing being built. A noisy one merely annoys.

That asymmetry only decides the question because the queue-cleanliness argument has
another answer: Ralph demotes a thread whose newest comment is a signed agent reply to
`awaiting confirmation` rather than re-reporting it as new work. The queue stays clean
without the agent taking the reviewer's button. And if the reviewer replies again, the
thread returns to the queue on its own.

There is a second reason, smaller but not nothing: resolving another person's review
thread is theirs to do by convention on GitHub. An agent doing it at scale reads as the
crew closing its own homework.

**Exception:** if the operator explicitly asks an agent to resolve threads — in the
session, not inferred from a PR comment — that is a direct instruction and it holds.

### Anti-patterns

- ❌ Omitting the marker "just this once" — one unmarked agent comment is a false human alert.
- ❌ Putting the marker anywhere but the last line.
- ❌ Setting `seen=` to the current time — the value is the timestamp of the comment
being acknowledged, not the time of acknowledgement.
- ❌ Setting `seen=` on a comment that did not actually address the human input.
- ❌ Resolving a review thread to clear it from Ralph's queue.
- ❌ Silently skipping a review thread the agent disagrees with — reply and say so.
- ❌ Back-filling markers onto historical comments — use an adoption cutoff instead
(see `ralph-reference.md` → "Adoption cutoff").

Expand Down
Loading
Loading