Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .changeset/agent-comment-marker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
"@blacklite/crew-cli": minor
"@blacklite/crew-sdk": minor
---

Agent comment signing, and human-comment detection in Ralph's work-check cycle.

Crew agents post through `gh` using the operator's credentials, so agent comments and human comments carry the same `author.login`. Author cannot distinguish them and style heuristics are fragile, which meant a human reply on an issue was the one work signal Ralph could not see — an answer could sit in a thread indefinitely while the crew kept acting on advice that answer had overturned.

- **Marker convention (`crew.agent.md`, `issue-lifecycle.md`).** Every agent-posted issue or PR comment ends with `<!-- crew:agent={member} -->`. Anything unmarked is a human comment by definition. Mandatory for every member and every posting path (`gh issue comment`, `gh pr comment`, `gh pr review --body`, GitHub MCP equivalents); not applied to commit messages, issue bodies or PR bodies.
- **`seen=` acknowledgement field.** A reply to human input extends the marker with the ISO-8601 timestamp of the newest human comment read (`<!-- crew:agent=link seen=2026-07-29T03:06:23Z -->`). This is the durable, per-issue high-water mark, and it lives in the thread rather than on disk — agents run in worktrees, in CI and on multiple machines, where a local state file is wrong. A marked comment *without* `seen=` does not advance the mark, so a status update cannot silence an outstanding question.
- **Ralph's scan (`ralph-reference.md`).** New "Human Comment Detection" section: a single GraphQL query for the whole tracker (replacing the O(open issues) `gh issue view` loop), the two-level high-water mark (session set for within-session dedupe, `seen=` for across sessions), and a `commentWatch.since` adoption cutoff in `.crew/config.json` so pre-convention comments are not reported as a back-catalogue of false human alerts. Human comments sort first in the priority order, because a human reply can supersede work already in flight; the routed reviewer must state explicitly whether it does.
36 changes: 35 additions & 1 deletion .crew-templates/crew.agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -849,6 +849,32 @@ Before connecting to a GitHub repository, verify that the `gh` CLI is available
| "work on issue #N" / "pick up #N" | Route issue to appropriate agent |
| "work on all issues" / "start the backlog" | Route all open issues (batched) |

### Agent Comment Signing — MANDATORY

Agents post through `gh`, authenticated as the operator, so **every agent comment is
authored by the human's GitHub account**. Author cannot distinguish agent output from a
genuine human reply, and without a distinction human replies on issues go undetected.

**Append `<!-- crew:agent={member} -->` as the last line of every comment any agent posts
to an issue or a PR.** No exceptions — this applies to every member including Ralph, Rai,
Scribe and the Fact Checker, and to `gh issue comment`, `gh pr comment`,
`gh pr review --body`, and the equivalent GitHub MCP tools. It does not apply to commit
messages, issue bodies or PR bodies.

Anything unmarked is a human comment **by definition**. One unsigned agent comment is a
false human alert; one unsigned agent comment in the other direction is a human reply that
never gets read.

When the comment answers human input on that thread, extend the marker with the timestamp
of the newest human comment read — this is the high-water mark Ralph uses:

```
<!-- crew:agent=link seen=2026-07-29T03:06:23Z -->
```

Include this instruction in every spawn prompt for an agent that may comment on an issue
or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing".

---

## Ralph — Work Monitor
Expand All @@ -857,7 +883,15 @@ Ralph is the always-on work monitor. When active, Ralph runs a continuous scan

Do not pause for permission between work items when Ralph is active.

**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, watch mode, state model, board format, and follow-up integration.
Ralph's scan also covers **unreviewed human comments** on open issues — comments with no
`<!-- crew:agent= -->` marker, newer than the issue's `seen=` high-water mark and newer than
the `commentWatch.since` adoption cutoff in `.crew/config.json`. These sort **first**: a
human reply can supersede a recommendation the crew is already acting on. Ralph surfaces
(issue, owner, gist) and routes to the issue's `crew:{member}`; it does not interpret. The
reviewer must state explicitly whether the reply supersedes an earlier recommendation, and
must close the loop with a `seen=` acknowledgement.

**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, watch mode, state model, board format, and follow-up integration.

### Connecting to a Repo

Expand Down
57 changes: 57 additions & 0 deletions .crew-templates/issue-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,61 @@ When spawning an agent to work on an issue, include this context block:
4. Report PR URL to coordinator
```

## Agent Comment Signing

Crew agents post through `gh`, which authenticates as the operator. **Every agent
comment is therefore authored by the human's GitHub account.** Author is useless as a
discriminator, and style heuristics ("agents write long comments with `##` headers")
break the moment either party writes atypically.

Sign agent comments instead.

### The marker

Append this as the **last line** of the body of every comment an agent posts to an
issue or a PR:

```
<!-- crew:agent={member} -->
```

- `{member}` is the crew member's roster name, lowercased (`link`, `sparks`, `ralph`).
- It renders as nothing on GitHub, survives comment edits, and is one `grep` away.
- **Anything without the marker is, by definition, a human comment.** That inverse is
the point: the marker is what makes human replies detectable at all.

Applies to `gh issue comment`, `gh pr comment`, `gh pr review --body`, the equivalent
GitHub MCP tools, and any workflow that posts on an agent's behalf. It does **not**
apply to commit messages, PR bodies, or issue bodies — comments only.

### Acknowledging a human reply — the `seen=` field

When an agent comment is a **response to human input on that thread**, add a `seen=`
field carrying the ISO-8601 timestamp of the newest human comment the agent read:

```
<!-- crew:agent=link seen=2026-07-29T03:06:23Z -->
```

`seen=` is the per-issue high-water mark for human comments. Ralph treats every
unmarked comment newer than the highest `seen=` on that issue as unreviewed and keeps
surfacing it. A marked comment **without** `seen=` does not advance the mark — a
status update posted while a human question is still outstanding must not silence
that question.

Acknowledge honestly. `seen=` asserts "an agent read this and responded to it". It is
not a dismissal button.

### Anti-patterns

- ❌ Omitting the marker "just this once" — one unmarked agent comment is a false human alert.
- ❌ Putting the marker anywhere but the last line.
- ❌ Setting `seen=` to the current time — the value is the timestamp of the comment
being acknowledged, not the time of acknowledgement.
- ❌ Setting `seen=` on a comment that did not actually address the human input.
- ❌ Back-filling markers onto historical comments — use an adoption cutoff instead
(see `ralph-reference.md` → "Adoption cutoff").

## Ralph's Role in Issue Lifecycle

Ralph (the work monitor) continuously checks issue and PR state:
Expand Down Expand Up @@ -402,6 +457,8 @@ All PRs reviewed → All PRs merged → Epic closed
- ❌ Using `checkout -b` when parallel agents are active (causes working directory conflicts)
- ❌ Manually transitioning issue states — let the platform and Crew automation handle it
- ❌ Skipping the branch naming convention — breaks Ralph's tracking logic
- ❌ Posting an agent comment without the `<!-- crew:agent={member} -->` marker — it will
be misread as a human reply and re-routed for review

## Migration Notes

Expand Down
141 changes: 140 additions & 1 deletion .crew-templates/ralph-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,16 @@ gh pr list --state open --json number,title,author,labels,isDraft,reviewDecision

# Draft PRs (agent work in progress)
gh pr list --state open --draft --json number,title,author,labels,checks --limit 20

# Unreviewed human comments on open issues (see "Human Comment Detection" below)
```

**Step 2 — Categorize findings:**

| Category | Signal | Action |
|----------|--------|--------|
| **Untriaged issues** | `crew` label, no `crew:{member}` label | Lead triages: reads issue, assigns `crew:{member}` label |
| **Human comments** | Comment on an open issue with no `<!-- crew:agent= -->` marker, newer than that issue's high-water mark | Report issue + gist, route to the issue's `crew:{member}` for review |
| **Assigned but unstarted** | `crew:{member}` label, no assignee or no PR | Spawn the assigned agent to pick it up |
| **Draft PRs** | PR in draft from crew member | Check if agent needs to continue; if stalled, nudge |
| **Review feedback** | PR has `CHANGES_REQUESTED` review | Route feedback to PR author agent to address |
Expand All @@ -59,7 +62,10 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit
| **No work found** | All clear | Report: "📋 Board is clear. Ralph is idling." Suggest `npx @blacklite/crew-cli watch` for persistent polling. |

**Step 3 — Act on highest-priority item:**
- Process one category at a time, highest priority first (untriaged > assigned > CI failures > review feedback > approved PRs)
- Process one category at a time, highest priority first (human comments > untriaged > assigned > CI failures > review feedback > approved PRs)
- **Human comments sort first on purpose.** A human reply can supersede a recommendation the
crew is already acting on; surfacing it before spawning more work is what stops the crew
building on advice that has been overturned.
- Spawn agents as needed, collect results
- **⚡ CRITICAL: After results are collected, DO NOT stop. DO NOT wait for user input. IMMEDIATELY go back to Step 1 and scan again.** This is a loop — Ralph keeps cycling until the board is clear or the user says "idle". Each cycle is one "round".
- If multiple items exist in the same category, process them in parallel (spawn multiple agents)
Expand All @@ -77,6 +83,132 @@ After every 3-5 rounds, pause and report before continuing:

**Do NOT ask for permission to continue.** Just report and keep going. The user must explicitly say "idle" or "stop" to break the loop. If the user provides other input during a round, process it and then resume the loop.

### Human Comment Detection

A reply from the operator on an issue is a work signal, and it is the one signal Ralph
historically missed: the scan covers issues, PRs, labels and checks, but not comments. A
human answer could sit in a thread indefinitely while the crew kept acting on advice that
answer had already overturned.

**Why author filtering does not work.** Crew agents post through `gh`, authenticated as the
operator, so agent comments and human comments carry the *same* `author.login`. Filtering on
author returns everything. Style heuristics are worse — they misfire the first time either
party writes atypically.

Detection depends on the marker convention in `issue-lifecycle.md` → "Agent Comment Signing":
every agent comment ends with `<!-- crew:agent={member} -->`, so **an unmarked comment is a
human comment by definition**. Ralph does not need to recognise humans; it only needs to
recognise agents, which it can do exactly.

#### High-water mark

Two levels, deliberately:

| Level | Lives in | Suppresses |
|-------|----------|------------|
| **Session** | Ralph's in-session state (`commentsReported`, a set of `{issue, createdAt}`) | Re-reporting the same comment on every round of the same session |
| **Durable** | The `seen=` field on agent comments **in the thread itself** | Re-reporting across sessions, machines and worktrees |

The durable mark lives on GitHub, not on disk, and that is the design choice worth
defending. A file under `.crew/` would be a mutable, per-checkout, merge-conflicting
record of something GitHub already stores; agents run in worktrees, in CI and on more
than one machine, and a local file is wrong in all of those the moment two of them run.
Putting the mark in the thread means the thread is self-describing: anyone — Ralph, a
member, a human reading the issue — can tell what has been acknowledged from the issue
alone.

Per-issue mark = the highest `seen=` value across that issue's marked comments. A marked
comment with no `seen=` does **not** advance it: an agent posting a status update while a
human question is outstanding must not silence the question. An unacknowledged human
comment therefore keeps being reported every session until someone actually answers it —
that nagging is the feature.

#### Adoption cutoff

Comments posted **before** the signing convention was adopted carry no marker, so a naive
first run classifies the entire back-catalogue of agent reports as new human comments.

Set a floor once, at adoption, in `.crew/config.json`:

```json
{
"commentWatch": {
"repo": "{owner}/{repo}",
"since": "{ISO-8601 timestamp of adoption}"
}
}
```

Comments at or before `since` are never reported, marked or not. This is the only honest
option: the marker cannot retroactively classify comments that predate it, and back-filling
markers means editing historical comments in someone else's thread. Do a **one-time manual
sweep** of pre-cutoff threads when adopting, act on anything genuinely unanswered, and then
let the cutoff hold the line.

`.crew/config.json` is operator config — `crew upgrade` reads it and never rewrites it — so
the cutoff survives upgrades.

#### The scan

One GraphQL call for the whole tracker, not one REST call per issue. The per-issue loop
(`gh issue list` → `gh issue view` per number) works but is O(open issues) requests per
cycle, and Ralph cycles continuously.

```bash
CUTOFF=$(jq -r '.commentWatch.since // "1970-01-01T00:00:00Z"' .crew/config.json)
REPO=$(jq -r '.commentWatch.repo' .crew/config.json)

gh api graphql -F owner="${REPO%%/*}" -F repo="${REPO##*/}" -F issues=50 -F comments=30 -f query='
query($owner:String!, $repo:String!, $issues:Int!, $comments:Int!) {
repository(owner:$owner, name:$repo) {
issues(states:OPEN, first:$issues, orderBy:{field:UPDATED_AT, direction:DESC}) {
nodes {
number title
labels(first:20) { nodes { name } }
comments(last:$comments) { nodes { createdAt author { login } body } }
}
}
}
}' | jq -r --arg cutoff "$CUTOFF" '
def marked: (.body // "") | test("<!--[ \t]*crew:agent=");
def seenAt: [ ((.body // "") | scan("<!--[ \t]*crew:agent=[^ \t>]+[^>]*[ \t]seen=([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:]+Z)")) ] | flatten | max;

.data.repository.issues.nodes[]
| . as $i
| ( [ $i.comments.nodes[] | select(marked) | seenAt | select(. != null) ] | max ) as $ack
| ( [ $cutoff, ($ack // $cutoff) ] | max ) as $mark
| $i.comments.nodes[]
| select(marked | not)
| select(.createdAt > $mark)
| "\(.createdAt)\t#\($i.number)\t\([ $i.labels.nodes[].name | select(startswith("crew:")) ] | first // "unassigned")\t\(.body | gsub("[\r\n]+"; " ") | .[0:120])"
' | sort -r
```

`first: 50` / `last: 30` cover a tracker of this size in a single request. If either bound
is hit, page with `pageInfo { hasNextPage endCursor }` rather than raising the limits — the
GraphQL node budget is a product of the two.

#### What Ralph does with a hit

1. **Surface, do not interpret.** Report the issue number, the assigned `crew:{member}`, the
timestamp, and the first line or two. Ralph detects work; it does not decide what a reply
means.
2. **Route to the owner.** Spawn the issue's assigned `crew:{member}`. If the issue has no
`crew:{member}` label, it goes to the Lead as untriaged.
3. **Require a supersession check.** The reviewer's brief must include: *does this reply
overturn a recommendation the crew has already made or is already acting on?* If it does,
say so explicitly in the reply — name the superseded recommendation. Do not quietly
rewrite the plan.
4. **Acknowledge.** The reviewer's reply closes the loop with
`<!-- crew:agent={member} seen={timestamp of the comment being answered} -->`, which
advances the durable mark.

Board line:

```
💬 Human comments: 2 unreviewed (#84 → link, #81 → sparks)
```

### Watch Mode (`crew watch`)

Ralph's in-session loop processes work while it exists, then idles. For **persistent polling** between sessions or when you're away from the keyboard, use the `crew watch` CLI command:
Expand Down Expand Up @@ -108,6 +240,12 @@ Ralph's state is session-scoped (not persisted to disk):
- **Round count** — how many check cycles completed
- **Scope** — what categories to monitor (default: all)
- **Stats** — issues closed, PRs merged, items processed this session
- **Comments reported** — `{issue, createdAt}` pairs surfaced this session, so a comment is
reported once per session rather than once per round

Session state deliberately does **not** hold the comment high-water mark. A session-scoped
mark re-reports every old comment at the start of every new session; the durable mark lives
in the thread as `seen=` (see [Human Comment Detection](#human-comment-detection)).

### Ralph on the Board

Expand All @@ -117,6 +255,7 @@ When Ralph reports status, use this format:
🔄 Ralph — Work Monitor
━━━━━━━━━━━━━━━━━━━━━━
📊 Board Status:
💬 Human input: 1 unreviewed comment (#84 → link)
🔴 Untriaged: 2 issues need triage
🟡 In Progress: 3 issues assigned, 1 draft PR
🟢 Ready: 1 PR approved, awaiting merge
Expand Down
Loading
Loading