diff --git a/.changeset/agent-comment-marker.md b/.changeset/agent-comment-marker.md new file mode 100644 index 000000000..d0e9ab2a2 --- /dev/null +++ b/.changeset/agent-comment-marker.md @@ -0,0 +1,12 @@ +--- +"@blacklite/crew-cli": minor +"@blacklite/crew-sdk": minor +--- + +Agent comment signing, and human-comment detection in Ralph's work-check cycle. + +Crew agents post through `gh` using the operator's credentials, so agent comments and human comments carry the same `author.login`. Author cannot distinguish them and style heuristics are fragile, which meant a human reply on an issue was the one work signal Ralph could not see — an answer could sit in a thread indefinitely while the crew kept acting on advice that answer had overturned. + +- **Marker convention (`crew.agent.md`, `issue-lifecycle.md`).** Every agent-posted issue or PR comment ends with ``. Anything unmarked is a human comment by definition. Mandatory for every member and every posting path (`gh issue comment`, `gh pr comment`, `gh pr review --body`, GitHub MCP equivalents); not applied to commit messages, issue bodies or PR bodies. +- **`seen=` acknowledgement field.** A reply to human input extends the marker with the ISO-8601 timestamp of the newest human comment read (``). This is the durable, per-issue high-water mark, and it lives in the thread rather than on disk — agents run in worktrees, in CI and on multiple machines, where a local state file is wrong. A marked comment *without* `seen=` does not advance the mark, so a status update cannot silence an outstanding question. +- **Ralph's scan (`ralph-reference.md`).** New "Human Comment Detection" section: a single GraphQL query for the whole tracker (replacing the O(open issues) `gh issue view` loop), the two-level high-water mark (session set for within-session dedupe, `seen=` for across sessions), and a `commentWatch.since` adoption cutoff in `.crew/config.json` so pre-convention comments are not reported as a back-catalogue of false human alerts. Human comments sort first in the priority order, because a human reply can supersede work already in flight; the routed reviewer must state explicitly whether it does. diff --git a/.crew-templates/crew.agent.md b/.crew-templates/crew.agent.md index 277195f79..5a19eb48f 100644 --- a/.crew-templates/crew.agent.md +++ b/.crew-templates/crew.agent.md @@ -849,6 +849,32 @@ Before connecting to a GitHub repository, verify that the `gh` CLI is available | "work on issue #N" / "pick up #N" | Route issue to appropriate agent | | "work on all issues" / "start the backlog" | Route all open issues (batched) | +### Agent Comment Signing — MANDATORY + +Agents post through `gh`, authenticated as the operator, so **every agent comment is +authored by the human's GitHub account**. Author cannot distinguish agent output from a +genuine human reply, and without a distinction human replies on issues go undetected. + +**Append `` as the last line of every comment any agent posts +to an issue or a PR.** No exceptions — this applies to every member including Ralph, Rai, +Scribe and the Fact Checker, and to `gh issue comment`, `gh pr comment`, +`gh pr review --body`, and the equivalent GitHub MCP tools. It does not apply to commit +messages, issue bodies or PR bodies. + +Anything unmarked is a human comment **by definition**. One unsigned agent comment is a +false human alert; one unsigned agent comment in the other direction is a human reply that +never gets read. + +When the comment answers human input on that thread, extend the marker with the timestamp +of the newest human comment read — this is the high-water mark Ralph uses: + +``` + +``` + +Include this instruction in every spawn prompt for an agent that may comment on an issue +or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing". + --- ## Ralph — Work Monitor @@ -857,7 +883,15 @@ Ralph is the always-on work monitor. When active, Ralph runs a continuous scan Do not pause for permission between work items when Ralph is active. -**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, watch mode, state model, board format, and follow-up integration. +Ralph's scan also covers **unreviewed human comments** on open issues — comments with no +`` marker, newer than the issue's `seen=` high-water mark and newer than +the `commentWatch.since` adoption cutoff in `.crew/config.json`. These sort **first**: a +human reply can supersede a recommendation the crew is already acting on. Ralph surfaces +(issue, owner, gist) and routes to the issue's `crew:{member}`; it does not interpret. The +reviewer must state explicitly whether the reply supersedes an earlier recommendation, and +must close the loop with a `seen=` acknowledgement. + +**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, watch mode, state model, board format, and follow-up integration. ### Connecting to a Repo diff --git a/.crew-templates/issue-lifecycle.md b/.crew-templates/issue-lifecycle.md index c5dacf291..4c2e16b40 100644 --- a/.crew-templates/issue-lifecycle.md +++ b/.crew-templates/issue-lifecycle.md @@ -320,6 +320,61 @@ When spawning an agent to work on an issue, include this context block: 4. Report PR URL to coordinator ``` +## Agent Comment Signing + +Crew agents post through `gh`, which authenticates as the operator. **Every agent +comment is therefore authored by the human's GitHub account.** Author is useless as a +discriminator, and style heuristics ("agents write long comments with `##` headers") +break the moment either party writes atypically. + +Sign agent comments instead. + +### The marker + +Append this as the **last line** of the body of every comment an agent posts to an +issue or a PR: + +``` + +``` + +- `{member}` is the crew member's roster name, lowercased (`link`, `sparks`, `ralph`). +- It renders as nothing on GitHub, survives comment edits, and is one `grep` away. +- **Anything without the marker is, by definition, a human comment.** That inverse is + the point: the marker is what makes human replies detectable at all. + +Applies to `gh issue comment`, `gh pr comment`, `gh pr review --body`, the equivalent +GitHub MCP tools, and any workflow that posts on an agent's behalf. It does **not** +apply to commit messages, PR bodies, or issue bodies — comments only. + +### Acknowledging a human reply — the `seen=` field + +When an agent comment is a **response to human input on that thread**, add a `seen=` +field carrying the ISO-8601 timestamp of the newest human comment the agent read: + +``` + +``` + +`seen=` is the per-issue high-water mark for human comments. Ralph treats every +unmarked comment newer than the highest `seen=` on that issue as unreviewed and keeps +surfacing it. A marked comment **without** `seen=` does not advance the mark — a +status update posted while a human question is still outstanding must not silence +that question. + +Acknowledge honestly. `seen=` asserts "an agent read this and responded to it". It is +not a dismissal button. + +### Anti-patterns + +- ❌ Omitting the marker "just this once" — one unmarked agent comment is a false human alert. +- ❌ Putting the marker anywhere but the last line. +- ❌ Setting `seen=` to the current time — the value is the timestamp of the comment + being acknowledged, not the time of acknowledgement. +- ❌ Setting `seen=` on a comment that did not actually address the human input. +- ❌ Back-filling markers onto historical comments — use an adoption cutoff instead + (see `ralph-reference.md` → "Adoption cutoff"). + ## Ralph's Role in Issue Lifecycle Ralph (the work monitor) continuously checks issue and PR state: @@ -402,6 +457,8 @@ All PRs reviewed → All PRs merged → Epic closed - ❌ Using `checkout -b` when parallel agents are active (causes working directory conflicts) - ❌ Manually transitioning issue states — let the platform and Crew automation handle it - ❌ Skipping the branch naming convention — breaks Ralph's tracking logic +- ❌ Posting an agent comment without the `` marker — it will + be misread as a human reply and re-routed for review ## Migration Notes diff --git a/.crew-templates/ralph-reference.md b/.crew-templates/ralph-reference.md index d1c3ce047..1dc83598d 100644 --- a/.crew-templates/ralph-reference.md +++ b/.crew-templates/ralph-reference.md @@ -44,6 +44,8 @@ gh pr list --state open --json number,title,author,labels,isDraft,reviewDecision # Draft PRs (agent work in progress) gh pr list --state open --draft --json number,title,author,labels,checks --limit 20 + +# Unreviewed human comments on open issues (see "Human Comment Detection" below) ``` **Step 2 — Categorize findings:** @@ -51,6 +53,7 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | Category | Signal | Action | |----------|--------|--------| | **Untriaged issues** | `crew` label, no `crew:{member}` label | Lead triages: reads issue, assigns `crew:{member}` label | +| **Human comments** | Comment on an open issue with no `` marker, newer than that issue's high-water mark | Report issue + gist, route to the issue's `crew:{member}` for review | | **Assigned but unstarted** | `crew:{member}` label, no assignee or no PR | Spawn the assigned agent to pick it up | | **Draft PRs** | PR in draft from crew member | Check if agent needs to continue; if stalled, nudge | | **Review feedback** | PR has `CHANGES_REQUESTED` review | Route feedback to PR author agent to address | @@ -59,7 +62,10 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | **No work found** | All clear | Report: "📋 Board is clear. Ralph is idling." Suggest `npx @blacklite/crew-cli watch` for persistent polling. | **Step 3 — Act on highest-priority item:** -- Process one category at a time, highest priority first (untriaged > assigned > CI failures > review feedback > approved PRs) +- Process one category at a time, highest priority first (human comments > untriaged > assigned > CI failures > review feedback > approved PRs) +- **Human comments sort first on purpose.** A human reply can supersede a recommendation the + crew is already acting on; surfacing it before spawning more work is what stops the crew + building on advice that has been overturned. - Spawn agents as needed, collect results - **⚡ CRITICAL: After results are collected, DO NOT stop. DO NOT wait for user input. IMMEDIATELY go back to Step 1 and scan again.** This is a loop — Ralph keeps cycling until the board is clear or the user says "idle". Each cycle is one "round". - If multiple items exist in the same category, process them in parallel (spawn multiple agents) @@ -77,6 +83,132 @@ After every 3-5 rounds, pause and report before continuing: **Do NOT ask for permission to continue.** Just report and keep going. The user must explicitly say "idle" or "stop" to break the loop. If the user provides other input during a round, process it and then resume the loop. +### Human Comment Detection + +A reply from the operator on an issue is a work signal, and it is the one signal Ralph +historically missed: the scan covers issues, PRs, labels and checks, but not comments. A +human answer could sit in a thread indefinitely while the crew kept acting on advice that +answer had already overturned. + +**Why author filtering does not work.** Crew agents post through `gh`, authenticated as the +operator, so agent comments and human comments carry the *same* `author.login`. Filtering on +author returns everything. Style heuristics are worse — they misfire the first time either +party writes atypically. + +Detection depends on the marker convention in `issue-lifecycle.md` → "Agent Comment Signing": +every agent comment ends with ``, so **an unmarked comment is a +human comment by definition**. Ralph does not need to recognise humans; it only needs to +recognise agents, which it can do exactly. + +#### High-water mark + +Two levels, deliberately: + +| Level | Lives in | Suppresses | +|-------|----------|------------| +| **Session** | Ralph's in-session state (`commentsReported`, a set of `{issue, createdAt}`) | Re-reporting the same comment on every round of the same session | +| **Durable** | The `seen=` field on agent comments **in the thread itself** | Re-reporting across sessions, machines and worktrees | + +The durable mark lives on GitHub, not on disk, and that is the design choice worth +defending. A file under `.crew/` would be a mutable, per-checkout, merge-conflicting +record of something GitHub already stores; agents run in worktrees, in CI and on more +than one machine, and a local file is wrong in all of those the moment two of them run. +Putting the mark in the thread means the thread is self-describing: anyone — Ralph, a +member, a human reading the issue — can tell what has been acknowledged from the issue +alone. + +Per-issue mark = the highest `seen=` value across that issue's marked comments. A marked +comment with no `seen=` does **not** advance it: an agent posting a status update while a +human question is outstanding must not silence the question. An unacknowledged human +comment therefore keeps being reported every session until someone actually answers it — +that nagging is the feature. + +#### Adoption cutoff + +Comments posted **before** the signing convention was adopted carry no marker, so a naive +first run classifies the entire back-catalogue of agent reports as new human comments. + +Set a floor once, at adoption, in `.crew/config.json`: + +```json +{ + "commentWatch": { + "repo": "{owner}/{repo}", + "since": "{ISO-8601 timestamp of adoption}" + } +} +``` + +Comments at or before `since` are never reported, marked or not. This is the only honest +option: the marker cannot retroactively classify comments that predate it, and back-filling +markers means editing historical comments in someone else's thread. Do a **one-time manual +sweep** of pre-cutoff threads when adopting, act on anything genuinely unanswered, and then +let the cutoff hold the line. + +`.crew/config.json` is operator config — `crew upgrade` reads it and never rewrites it — so +the cutoff survives upgrades. + +#### The scan + +One GraphQL call for the whole tracker, not one REST call per issue. The per-issue loop +(`gh issue list` → `gh issue view` per number) works but is O(open issues) requests per +cycle, and Ralph cycles continuously. + +```bash +CUTOFF=$(jq -r '.commentWatch.since // "1970-01-01T00:00:00Z"' .crew/config.json) +REPO=$(jq -r '.commentWatch.repo' .crew/config.json) + +gh api graphql -F owner="${REPO%%/*}" -F repo="${REPO##*/}" -F issues=50 -F comments=30 -f query=' + query($owner:String!, $repo:String!, $issues:Int!, $comments:Int!) { + repository(owner:$owner, name:$repo) { + issues(states:OPEN, first:$issues, orderBy:{field:UPDATED_AT, direction:DESC}) { + nodes { + number title + labels(first:20) { nodes { name } } + comments(last:$comments) { nodes { createdAt author { login } body } } + } + } + } + }' | jq -r --arg cutoff "$CUTOFF" ' + def marked: (.body // "") | test("`, which + advances the durable mark. + +Board line: + +``` +💬 Human comments: 2 unreviewed (#84 → link, #81 → sparks) +``` + ### Watch Mode (`crew watch`) Ralph's in-session loop processes work while it exists, then idles. For **persistent polling** between sessions or when you're away from the keyboard, use the `crew watch` CLI command: @@ -108,6 +240,12 @@ Ralph's state is session-scoped (not persisted to disk): - **Round count** — how many check cycles completed - **Scope** — what categories to monitor (default: all) - **Stats** — issues closed, PRs merged, items processed this session +- **Comments reported** — `{issue, createdAt}` pairs surfaced this session, so a comment is + reported once per session rather than once per round + +Session state deliberately does **not** hold the comment high-water mark. A session-scoped +mark re-reports every old comment at the start of every new session; the durable mark lives +in the thread as `seen=` (see [Human Comment Detection](#human-comment-detection)). ### Ralph on the Board @@ -117,6 +255,7 @@ When Ralph reports status, use this format: 🔄 Ralph — Work Monitor ━━━━━━━━━━━━━━━━━━━━━━ 📊 Board Status: + 💬 Human input: 1 unreviewed comment (#84 → link) 🔴 Untriaged: 2 issues need triage 🟡 In Progress: 3 issues assigned, 1 draft PR 🟢 Ready: 1 PR approved, awaiting merge diff --git a/.github/agents/crew.agent.md b/.github/agents/crew.agent.md index 277195f79..5a19eb48f 100644 --- a/.github/agents/crew.agent.md +++ b/.github/agents/crew.agent.md @@ -849,6 +849,32 @@ Before connecting to a GitHub repository, verify that the `gh` CLI is available | "work on issue #N" / "pick up #N" | Route issue to appropriate agent | | "work on all issues" / "start the backlog" | Route all open issues (batched) | +### Agent Comment Signing — MANDATORY + +Agents post through `gh`, authenticated as the operator, so **every agent comment is +authored by the human's GitHub account**. Author cannot distinguish agent output from a +genuine human reply, and without a distinction human replies on issues go undetected. + +**Append `` as the last line of every comment any agent posts +to an issue or a PR.** No exceptions — this applies to every member including Ralph, Rai, +Scribe and the Fact Checker, and to `gh issue comment`, `gh pr comment`, +`gh pr review --body`, and the equivalent GitHub MCP tools. It does not apply to commit +messages, issue bodies or PR bodies. + +Anything unmarked is a human comment **by definition**. One unsigned agent comment is a +false human alert; one unsigned agent comment in the other direction is a human reply that +never gets read. + +When the comment answers human input on that thread, extend the marker with the timestamp +of the newest human comment read — this is the high-water mark Ralph uses: + +``` + +``` + +Include this instruction in every spawn prompt for an agent that may comment on an issue +or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing". + --- ## Ralph — Work Monitor @@ -857,7 +883,15 @@ Ralph is the always-on work monitor. When active, Ralph runs a continuous scan Do not pause for permission between work items when Ralph is active. -**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, watch mode, state model, board format, and follow-up integration. +Ralph's scan also covers **unreviewed human comments** on open issues — comments with no +`` marker, newer than the issue's `seen=` high-water mark and newer than +the `commentWatch.since` adoption cutoff in `.crew/config.json`. These sort **first**: a +human reply can supersede a recommendation the crew is already acting on. Ralph surfaces +(issue, owner, gist) and routes to the issue's `crew:{member}`; it does not interpret. The +reviewer must state explicitly whether the reply supersedes an earlier recommendation, and +must close the loop with a `seen=` acknowledgement. + +**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, watch mode, state model, board format, and follow-up integration. ### Connecting to a Repo diff --git a/packages/crew-cli/templates/crew.agent.md.template b/packages/crew-cli/templates/crew.agent.md.template index 277195f79..5a19eb48f 100644 --- a/packages/crew-cli/templates/crew.agent.md.template +++ b/packages/crew-cli/templates/crew.agent.md.template @@ -849,6 +849,32 @@ Before connecting to a GitHub repository, verify that the `gh` CLI is available | "work on issue #N" / "pick up #N" | Route issue to appropriate agent | | "work on all issues" / "start the backlog" | Route all open issues (batched) | +### Agent Comment Signing — MANDATORY + +Agents post through `gh`, authenticated as the operator, so **every agent comment is +authored by the human's GitHub account**. Author cannot distinguish agent output from a +genuine human reply, and without a distinction human replies on issues go undetected. + +**Append `` as the last line of every comment any agent posts +to an issue or a PR.** No exceptions — this applies to every member including Ralph, Rai, +Scribe and the Fact Checker, and to `gh issue comment`, `gh pr comment`, +`gh pr review --body`, and the equivalent GitHub MCP tools. It does not apply to commit +messages, issue bodies or PR bodies. + +Anything unmarked is a human comment **by definition**. One unsigned agent comment is a +false human alert; one unsigned agent comment in the other direction is a human reply that +never gets read. + +When the comment answers human input on that thread, extend the marker with the timestamp +of the newest human comment read — this is the high-water mark Ralph uses: + +``` + +``` + +Include this instruction in every spawn prompt for an agent that may comment on an issue +or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing". + --- ## Ralph — Work Monitor @@ -857,7 +883,15 @@ Ralph is the always-on work monitor. When active, Ralph runs a continuous scan Do not pause for permission between work items when Ralph is active. -**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, watch mode, state model, board format, and follow-up integration. +Ralph's scan also covers **unreviewed human comments** on open issues — comments with no +`` marker, newer than the issue's `seen=` high-water mark and newer than +the `commentWatch.since` adoption cutoff in `.crew/config.json`. These sort **first**: a +human reply can supersede a recommendation the crew is already acting on. Ralph surfaces +(issue, owner, gist) and routes to the issue's `crew:{member}`; it does not interpret. The +reviewer must state explicitly whether the reply supersedes an earlier recommendation, and +must close the loop with a `seen=` acknowledgement. + +**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, watch mode, state model, board format, and follow-up integration. ### Connecting to a Repo diff --git a/packages/crew-cli/templates/issue-lifecycle.md b/packages/crew-cli/templates/issue-lifecycle.md index c5dacf291..4c2e16b40 100644 --- a/packages/crew-cli/templates/issue-lifecycle.md +++ b/packages/crew-cli/templates/issue-lifecycle.md @@ -320,6 +320,61 @@ When spawning an agent to work on an issue, include this context block: 4. Report PR URL to coordinator ``` +## Agent Comment Signing + +Crew agents post through `gh`, which authenticates as the operator. **Every agent +comment is therefore authored by the human's GitHub account.** Author is useless as a +discriminator, and style heuristics ("agents write long comments with `##` headers") +break the moment either party writes atypically. + +Sign agent comments instead. + +### The marker + +Append this as the **last line** of the body of every comment an agent posts to an +issue or a PR: + +``` + +``` + +- `{member}` is the crew member's roster name, lowercased (`link`, `sparks`, `ralph`). +- It renders as nothing on GitHub, survives comment edits, and is one `grep` away. +- **Anything without the marker is, by definition, a human comment.** That inverse is + the point: the marker is what makes human replies detectable at all. + +Applies to `gh issue comment`, `gh pr comment`, `gh pr review --body`, the equivalent +GitHub MCP tools, and any workflow that posts on an agent's behalf. It does **not** +apply to commit messages, PR bodies, or issue bodies — comments only. + +### Acknowledging a human reply — the `seen=` field + +When an agent comment is a **response to human input on that thread**, add a `seen=` +field carrying the ISO-8601 timestamp of the newest human comment the agent read: + +``` + +``` + +`seen=` is the per-issue high-water mark for human comments. Ralph treats every +unmarked comment newer than the highest `seen=` on that issue as unreviewed and keeps +surfacing it. A marked comment **without** `seen=` does not advance the mark — a +status update posted while a human question is still outstanding must not silence +that question. + +Acknowledge honestly. `seen=` asserts "an agent read this and responded to it". It is +not a dismissal button. + +### Anti-patterns + +- ❌ Omitting the marker "just this once" — one unmarked agent comment is a false human alert. +- ❌ Putting the marker anywhere but the last line. +- ❌ Setting `seen=` to the current time — the value is the timestamp of the comment + being acknowledged, not the time of acknowledgement. +- ❌ Setting `seen=` on a comment that did not actually address the human input. +- ❌ Back-filling markers onto historical comments — use an adoption cutoff instead + (see `ralph-reference.md` → "Adoption cutoff"). + ## Ralph's Role in Issue Lifecycle Ralph (the work monitor) continuously checks issue and PR state: @@ -402,6 +457,8 @@ All PRs reviewed → All PRs merged → Epic closed - ❌ Using `checkout -b` when parallel agents are active (causes working directory conflicts) - ❌ Manually transitioning issue states — let the platform and Crew automation handle it - ❌ Skipping the branch naming convention — breaks Ralph's tracking logic +- ❌ Posting an agent comment without the `` marker — it will + be misread as a human reply and re-routed for review ## Migration Notes diff --git a/packages/crew-cli/templates/ralph-reference.md b/packages/crew-cli/templates/ralph-reference.md index d1c3ce047..1dc83598d 100644 --- a/packages/crew-cli/templates/ralph-reference.md +++ b/packages/crew-cli/templates/ralph-reference.md @@ -44,6 +44,8 @@ gh pr list --state open --json number,title,author,labels,isDraft,reviewDecision # Draft PRs (agent work in progress) gh pr list --state open --draft --json number,title,author,labels,checks --limit 20 + +# Unreviewed human comments on open issues (see "Human Comment Detection" below) ``` **Step 2 — Categorize findings:** @@ -51,6 +53,7 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | Category | Signal | Action | |----------|--------|--------| | **Untriaged issues** | `crew` label, no `crew:{member}` label | Lead triages: reads issue, assigns `crew:{member}` label | +| **Human comments** | Comment on an open issue with no `` marker, newer than that issue's high-water mark | Report issue + gist, route to the issue's `crew:{member}` for review | | **Assigned but unstarted** | `crew:{member}` label, no assignee or no PR | Spawn the assigned agent to pick it up | | **Draft PRs** | PR in draft from crew member | Check if agent needs to continue; if stalled, nudge | | **Review feedback** | PR has `CHANGES_REQUESTED` review | Route feedback to PR author agent to address | @@ -59,7 +62,10 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | **No work found** | All clear | Report: "📋 Board is clear. Ralph is idling." Suggest `npx @blacklite/crew-cli watch` for persistent polling. | **Step 3 — Act on highest-priority item:** -- Process one category at a time, highest priority first (untriaged > assigned > CI failures > review feedback > approved PRs) +- Process one category at a time, highest priority first (human comments > untriaged > assigned > CI failures > review feedback > approved PRs) +- **Human comments sort first on purpose.** A human reply can supersede a recommendation the + crew is already acting on; surfacing it before spawning more work is what stops the crew + building on advice that has been overturned. - Spawn agents as needed, collect results - **⚡ CRITICAL: After results are collected, DO NOT stop. DO NOT wait for user input. IMMEDIATELY go back to Step 1 and scan again.** This is a loop — Ralph keeps cycling until the board is clear or the user says "idle". Each cycle is one "round". - If multiple items exist in the same category, process them in parallel (spawn multiple agents) @@ -77,6 +83,132 @@ After every 3-5 rounds, pause and report before continuing: **Do NOT ask for permission to continue.** Just report and keep going. The user must explicitly say "idle" or "stop" to break the loop. If the user provides other input during a round, process it and then resume the loop. +### Human Comment Detection + +A reply from the operator on an issue is a work signal, and it is the one signal Ralph +historically missed: the scan covers issues, PRs, labels and checks, but not comments. A +human answer could sit in a thread indefinitely while the crew kept acting on advice that +answer had already overturned. + +**Why author filtering does not work.** Crew agents post through `gh`, authenticated as the +operator, so agent comments and human comments carry the *same* `author.login`. Filtering on +author returns everything. Style heuristics are worse — they misfire the first time either +party writes atypically. + +Detection depends on the marker convention in `issue-lifecycle.md` → "Agent Comment Signing": +every agent comment ends with ``, so **an unmarked comment is a +human comment by definition**. Ralph does not need to recognise humans; it only needs to +recognise agents, which it can do exactly. + +#### High-water mark + +Two levels, deliberately: + +| Level | Lives in | Suppresses | +|-------|----------|------------| +| **Session** | Ralph's in-session state (`commentsReported`, a set of `{issue, createdAt}`) | Re-reporting the same comment on every round of the same session | +| **Durable** | The `seen=` field on agent comments **in the thread itself** | Re-reporting across sessions, machines and worktrees | + +The durable mark lives on GitHub, not on disk, and that is the design choice worth +defending. A file under `.crew/` would be a mutable, per-checkout, merge-conflicting +record of something GitHub already stores; agents run in worktrees, in CI and on more +than one machine, and a local file is wrong in all of those the moment two of them run. +Putting the mark in the thread means the thread is self-describing: anyone — Ralph, a +member, a human reading the issue — can tell what has been acknowledged from the issue +alone. + +Per-issue mark = the highest `seen=` value across that issue's marked comments. A marked +comment with no `seen=` does **not** advance it: an agent posting a status update while a +human question is outstanding must not silence the question. An unacknowledged human +comment therefore keeps being reported every session until someone actually answers it — +that nagging is the feature. + +#### Adoption cutoff + +Comments posted **before** the signing convention was adopted carry no marker, so a naive +first run classifies the entire back-catalogue of agent reports as new human comments. + +Set a floor once, at adoption, in `.crew/config.json`: + +```json +{ + "commentWatch": { + "repo": "{owner}/{repo}", + "since": "{ISO-8601 timestamp of adoption}" + } +} +``` + +Comments at or before `since` are never reported, marked or not. This is the only honest +option: the marker cannot retroactively classify comments that predate it, and back-filling +markers means editing historical comments in someone else's thread. Do a **one-time manual +sweep** of pre-cutoff threads when adopting, act on anything genuinely unanswered, and then +let the cutoff hold the line. + +`.crew/config.json` is operator config — `crew upgrade` reads it and never rewrites it — so +the cutoff survives upgrades. + +#### The scan + +One GraphQL call for the whole tracker, not one REST call per issue. The per-issue loop +(`gh issue list` → `gh issue view` per number) works but is O(open issues) requests per +cycle, and Ralph cycles continuously. + +```bash +CUTOFF=$(jq -r '.commentWatch.since // "1970-01-01T00:00:00Z"' .crew/config.json) +REPO=$(jq -r '.commentWatch.repo' .crew/config.json) + +gh api graphql -F owner="${REPO%%/*}" -F repo="${REPO##*/}" -F issues=50 -F comments=30 -f query=' + query($owner:String!, $repo:String!, $issues:Int!, $comments:Int!) { + repository(owner:$owner, name:$repo) { + issues(states:OPEN, first:$issues, orderBy:{field:UPDATED_AT, direction:DESC}) { + nodes { + number title + labels(first:20) { nodes { name } } + comments(last:$comments) { nodes { createdAt author { login } body } } + } + } + } + }' | jq -r --arg cutoff "$CUTOFF" ' + def marked: (.body // "") | test("`, which + advances the durable mark. + +Board line: + +``` +💬 Human comments: 2 unreviewed (#84 → link, #81 → sparks) +``` + ### Watch Mode (`crew watch`) Ralph's in-session loop processes work while it exists, then idles. For **persistent polling** between sessions or when you're away from the keyboard, use the `crew watch` CLI command: @@ -108,6 +240,12 @@ Ralph's state is session-scoped (not persisted to disk): - **Round count** — how many check cycles completed - **Scope** — what categories to monitor (default: all) - **Stats** — issues closed, PRs merged, items processed this session +- **Comments reported** — `{issue, createdAt}` pairs surfaced this session, so a comment is + reported once per session rather than once per round + +Session state deliberately does **not** hold the comment high-water mark. A session-scoped +mark re-reports every old comment at the start of every new session; the durable mark lives +in the thread as `seen=` (see [Human Comment Detection](#human-comment-detection)). ### Ralph on the Board @@ -117,6 +255,7 @@ When Ralph reports status, use this format: 🔄 Ralph — Work Monitor ━━━━━━━━━━━━━━━━━━━━━━ 📊 Board Status: + 💬 Human input: 1 unreviewed comment (#84 → link) 🔴 Untriaged: 2 issues need triage 🟡 In Progress: 3 issues assigned, 1 draft PR 🟢 Ready: 1 PR approved, awaiting merge diff --git a/packages/crew-sdk/templates/crew.agent.md.template b/packages/crew-sdk/templates/crew.agent.md.template index 277195f79..5a19eb48f 100644 --- a/packages/crew-sdk/templates/crew.agent.md.template +++ b/packages/crew-sdk/templates/crew.agent.md.template @@ -849,6 +849,32 @@ Before connecting to a GitHub repository, verify that the `gh` CLI is available | "work on issue #N" / "pick up #N" | Route issue to appropriate agent | | "work on all issues" / "start the backlog" | Route all open issues (batched) | +### Agent Comment Signing — MANDATORY + +Agents post through `gh`, authenticated as the operator, so **every agent comment is +authored by the human's GitHub account**. Author cannot distinguish agent output from a +genuine human reply, and without a distinction human replies on issues go undetected. + +**Append `` as the last line of every comment any agent posts +to an issue or a PR.** No exceptions — this applies to every member including Ralph, Rai, +Scribe and the Fact Checker, and to `gh issue comment`, `gh pr comment`, +`gh pr review --body`, and the equivalent GitHub MCP tools. It does not apply to commit +messages, issue bodies or PR bodies. + +Anything unmarked is a human comment **by definition**. One unsigned agent comment is a +false human alert; one unsigned agent comment in the other direction is a human reply that +never gets read. + +When the comment answers human input on that thread, extend the marker with the timestamp +of the newest human comment read — this is the high-water mark Ralph uses: + +``` + +``` + +Include this instruction in every spawn prompt for an agent that may comment on an issue +or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing". + --- ## Ralph — Work Monitor @@ -857,7 +883,15 @@ Ralph is the always-on work monitor. When active, Ralph runs a continuous scan Do not pause for permission between work items when Ralph is active. -**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, watch mode, state model, board format, and follow-up integration. +Ralph's scan also covers **unreviewed human comments** on open issues — comments with no +`` marker, newer than the issue's `seen=` high-water mark and newer than +the `commentWatch.since` adoption cutoff in `.crew/config.json`. These sort **first**: a +human reply can supersede a recommendation the crew is already acting on. Ralph surfaces +(issue, owner, gist) and routes to the issue's `crew:{member}`; it does not interpret. The +reviewer must state explicitly whether the reply supersedes an earlier recommendation, and +must close the loop with a `seen=` acknowledgement. + +**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, watch mode, state model, board format, and follow-up integration. ### Connecting to a Repo diff --git a/packages/crew-sdk/templates/issue-lifecycle.md b/packages/crew-sdk/templates/issue-lifecycle.md index c5dacf291..4c2e16b40 100644 --- a/packages/crew-sdk/templates/issue-lifecycle.md +++ b/packages/crew-sdk/templates/issue-lifecycle.md @@ -320,6 +320,61 @@ When spawning an agent to work on an issue, include this context block: 4. Report PR URL to coordinator ``` +## Agent Comment Signing + +Crew agents post through `gh`, which authenticates as the operator. **Every agent +comment is therefore authored by the human's GitHub account.** Author is useless as a +discriminator, and style heuristics ("agents write long comments with `##` headers") +break the moment either party writes atypically. + +Sign agent comments instead. + +### The marker + +Append this as the **last line** of the body of every comment an agent posts to an +issue or a PR: + +``` + +``` + +- `{member}` is the crew member's roster name, lowercased (`link`, `sparks`, `ralph`). +- It renders as nothing on GitHub, survives comment edits, and is one `grep` away. +- **Anything without the marker is, by definition, a human comment.** That inverse is + the point: the marker is what makes human replies detectable at all. + +Applies to `gh issue comment`, `gh pr comment`, `gh pr review --body`, the equivalent +GitHub MCP tools, and any workflow that posts on an agent's behalf. It does **not** +apply to commit messages, PR bodies, or issue bodies — comments only. + +### Acknowledging a human reply — the `seen=` field + +When an agent comment is a **response to human input on that thread**, add a `seen=` +field carrying the ISO-8601 timestamp of the newest human comment the agent read: + +``` + +``` + +`seen=` is the per-issue high-water mark for human comments. Ralph treats every +unmarked comment newer than the highest `seen=` on that issue as unreviewed and keeps +surfacing it. A marked comment **without** `seen=` does not advance the mark — a +status update posted while a human question is still outstanding must not silence +that question. + +Acknowledge honestly. `seen=` asserts "an agent read this and responded to it". It is +not a dismissal button. + +### Anti-patterns + +- ❌ Omitting the marker "just this once" — one unmarked agent comment is a false human alert. +- ❌ Putting the marker anywhere but the last line. +- ❌ Setting `seen=` to the current time — the value is the timestamp of the comment + being acknowledged, not the time of acknowledgement. +- ❌ Setting `seen=` on a comment that did not actually address the human input. +- ❌ Back-filling markers onto historical comments — use an adoption cutoff instead + (see `ralph-reference.md` → "Adoption cutoff"). + ## Ralph's Role in Issue Lifecycle Ralph (the work monitor) continuously checks issue and PR state: @@ -402,6 +457,8 @@ All PRs reviewed → All PRs merged → Epic closed - ❌ Using `checkout -b` when parallel agents are active (causes working directory conflicts) - ❌ Manually transitioning issue states — let the platform and Crew automation handle it - ❌ Skipping the branch naming convention — breaks Ralph's tracking logic +- ❌ Posting an agent comment without the `` marker — it will + be misread as a human reply and re-routed for review ## Migration Notes diff --git a/packages/crew-sdk/templates/ralph-reference.md b/packages/crew-sdk/templates/ralph-reference.md index d1c3ce047..1dc83598d 100644 --- a/packages/crew-sdk/templates/ralph-reference.md +++ b/packages/crew-sdk/templates/ralph-reference.md @@ -44,6 +44,8 @@ gh pr list --state open --json number,title,author,labels,isDraft,reviewDecision # Draft PRs (agent work in progress) gh pr list --state open --draft --json number,title,author,labels,checks --limit 20 + +# Unreviewed human comments on open issues (see "Human Comment Detection" below) ``` **Step 2 — Categorize findings:** @@ -51,6 +53,7 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | Category | Signal | Action | |----------|--------|--------| | **Untriaged issues** | `crew` label, no `crew:{member}` label | Lead triages: reads issue, assigns `crew:{member}` label | +| **Human comments** | Comment on an open issue with no `` marker, newer than that issue's high-water mark | Report issue + gist, route to the issue's `crew:{member}` for review | | **Assigned but unstarted** | `crew:{member}` label, no assignee or no PR | Spawn the assigned agent to pick it up | | **Draft PRs** | PR in draft from crew member | Check if agent needs to continue; if stalled, nudge | | **Review feedback** | PR has `CHANGES_REQUESTED` review | Route feedback to PR author agent to address | @@ -59,7 +62,10 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | **No work found** | All clear | Report: "📋 Board is clear. Ralph is idling." Suggest `npx @blacklite/crew-cli watch` for persistent polling. | **Step 3 — Act on highest-priority item:** -- Process one category at a time, highest priority first (untriaged > assigned > CI failures > review feedback > approved PRs) +- Process one category at a time, highest priority first (human comments > untriaged > assigned > CI failures > review feedback > approved PRs) +- **Human comments sort first on purpose.** A human reply can supersede a recommendation the + crew is already acting on; surfacing it before spawning more work is what stops the crew + building on advice that has been overturned. - Spawn agents as needed, collect results - **⚡ CRITICAL: After results are collected, DO NOT stop. DO NOT wait for user input. IMMEDIATELY go back to Step 1 and scan again.** This is a loop — Ralph keeps cycling until the board is clear or the user says "idle". Each cycle is one "round". - If multiple items exist in the same category, process them in parallel (spawn multiple agents) @@ -77,6 +83,132 @@ After every 3-5 rounds, pause and report before continuing: **Do NOT ask for permission to continue.** Just report and keep going. The user must explicitly say "idle" or "stop" to break the loop. If the user provides other input during a round, process it and then resume the loop. +### Human Comment Detection + +A reply from the operator on an issue is a work signal, and it is the one signal Ralph +historically missed: the scan covers issues, PRs, labels and checks, but not comments. A +human answer could sit in a thread indefinitely while the crew kept acting on advice that +answer had already overturned. + +**Why author filtering does not work.** Crew agents post through `gh`, authenticated as the +operator, so agent comments and human comments carry the *same* `author.login`. Filtering on +author returns everything. Style heuristics are worse — they misfire the first time either +party writes atypically. + +Detection depends on the marker convention in `issue-lifecycle.md` → "Agent Comment Signing": +every agent comment ends with ``, so **an unmarked comment is a +human comment by definition**. Ralph does not need to recognise humans; it only needs to +recognise agents, which it can do exactly. + +#### High-water mark + +Two levels, deliberately: + +| Level | Lives in | Suppresses | +|-------|----------|------------| +| **Session** | Ralph's in-session state (`commentsReported`, a set of `{issue, createdAt}`) | Re-reporting the same comment on every round of the same session | +| **Durable** | The `seen=` field on agent comments **in the thread itself** | Re-reporting across sessions, machines and worktrees | + +The durable mark lives on GitHub, not on disk, and that is the design choice worth +defending. A file under `.crew/` would be a mutable, per-checkout, merge-conflicting +record of something GitHub already stores; agents run in worktrees, in CI and on more +than one machine, and a local file is wrong in all of those the moment two of them run. +Putting the mark in the thread means the thread is self-describing: anyone — Ralph, a +member, a human reading the issue — can tell what has been acknowledged from the issue +alone. + +Per-issue mark = the highest `seen=` value across that issue's marked comments. A marked +comment with no `seen=` does **not** advance it: an agent posting a status update while a +human question is outstanding must not silence the question. An unacknowledged human +comment therefore keeps being reported every session until someone actually answers it — +that nagging is the feature. + +#### Adoption cutoff + +Comments posted **before** the signing convention was adopted carry no marker, so a naive +first run classifies the entire back-catalogue of agent reports as new human comments. + +Set a floor once, at adoption, in `.crew/config.json`: + +```json +{ + "commentWatch": { + "repo": "{owner}/{repo}", + "since": "{ISO-8601 timestamp of adoption}" + } +} +``` + +Comments at or before `since` are never reported, marked or not. This is the only honest +option: the marker cannot retroactively classify comments that predate it, and back-filling +markers means editing historical comments in someone else's thread. Do a **one-time manual +sweep** of pre-cutoff threads when adopting, act on anything genuinely unanswered, and then +let the cutoff hold the line. + +`.crew/config.json` is operator config — `crew upgrade` reads it and never rewrites it — so +the cutoff survives upgrades. + +#### The scan + +One GraphQL call for the whole tracker, not one REST call per issue. The per-issue loop +(`gh issue list` → `gh issue view` per number) works but is O(open issues) requests per +cycle, and Ralph cycles continuously. + +```bash +CUTOFF=$(jq -r '.commentWatch.since // "1970-01-01T00:00:00Z"' .crew/config.json) +REPO=$(jq -r '.commentWatch.repo' .crew/config.json) + +gh api graphql -F owner="${REPO%%/*}" -F repo="${REPO##*/}" -F issues=50 -F comments=30 -f query=' + query($owner:String!, $repo:String!, $issues:Int!, $comments:Int!) { + repository(owner:$owner, name:$repo) { + issues(states:OPEN, first:$issues, orderBy:{field:UPDATED_AT, direction:DESC}) { + nodes { + number title + labels(first:20) { nodes { name } } + comments(last:$comments) { nodes { createdAt author { login } body } } + } + } + } + }' | jq -r --arg cutoff "$CUTOFF" ' + def marked: (.body // "") | test("`, which + advances the durable mark. + +Board line: + +``` +💬 Human comments: 2 unreviewed (#84 → link, #81 → sparks) +``` + ### Watch Mode (`crew watch`) Ralph's in-session loop processes work while it exists, then idles. For **persistent polling** between sessions or when you're away from the keyboard, use the `crew watch` CLI command: @@ -108,6 +240,12 @@ Ralph's state is session-scoped (not persisted to disk): - **Round count** — how many check cycles completed - **Scope** — what categories to monitor (default: all) - **Stats** — issues closed, PRs merged, items processed this session +- **Comments reported** — `{issue, createdAt}` pairs surfaced this session, so a comment is + reported once per session rather than once per round + +Session state deliberately does **not** hold the comment high-water mark. A session-scoped +mark re-reports every old comment at the start of every new session; the durable mark lives +in the thread as `seen=` (see [Human Comment Detection](#human-comment-detection)). ### Ralph on the Board @@ -117,6 +255,7 @@ When Ralph reports status, use this format: 🔄 Ralph — Work Monitor ━━━━━━━━━━━━━━━━━━━━━━ 📊 Board Status: + 💬 Human input: 1 unreviewed comment (#84 → link) 🔴 Untriaged: 2 issues need triage 🟡 In Progress: 3 issues assigned, 1 draft PR 🟢 Ready: 1 PR approved, awaiting merge diff --git a/templates/crew.agent.md.template b/templates/crew.agent.md.template index 277195f79..5a19eb48f 100644 --- a/templates/crew.agent.md.template +++ b/templates/crew.agent.md.template @@ -849,6 +849,32 @@ Before connecting to a GitHub repository, verify that the `gh` CLI is available | "work on issue #N" / "pick up #N" | Route issue to appropriate agent | | "work on all issues" / "start the backlog" | Route all open issues (batched) | +### Agent Comment Signing — MANDATORY + +Agents post through `gh`, authenticated as the operator, so **every agent comment is +authored by the human's GitHub account**. Author cannot distinguish agent output from a +genuine human reply, and without a distinction human replies on issues go undetected. + +**Append `` as the last line of every comment any agent posts +to an issue or a PR.** No exceptions — this applies to every member including Ralph, Rai, +Scribe and the Fact Checker, and to `gh issue comment`, `gh pr comment`, +`gh pr review --body`, and the equivalent GitHub MCP tools. It does not apply to commit +messages, issue bodies or PR bodies. + +Anything unmarked is a human comment **by definition**. One unsigned agent comment is a +false human alert; one unsigned agent comment in the other direction is a human reply that +never gets read. + +When the comment answers human input on that thread, extend the marker with the timestamp +of the newest human comment read — this is the high-water mark Ralph uses: + +``` + +``` + +Include this instruction in every spawn prompt for an agent that may comment on an issue +or PR. Full spec: `.crew/templates/issue-lifecycle.md` → "Agent Comment Signing". + --- ## Ralph — Work Monitor @@ -857,7 +883,15 @@ Ralph is the always-on work monitor. When active, Ralph runs a continuous scan Do not pause for permission between work items when Ralph is active. -**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, watch mode, state model, board format, and follow-up integration. +Ralph's scan also covers **unreviewed human comments** on open issues — comments with no +`` marker, newer than the issue's `seen=` high-water mark and newer than +the `commentWatch.since` adoption cutoff in `.crew/config.json`. These sort **first**: a +human reply can supersede a recommendation the crew is already acting on. Ralph surfaces +(issue, owner, gist) and routes to the issue's `crew:{member}`; it does not interpret. The +reviewer must state explicitly whether the reply supersedes an earlier recommendation, and +must close the loop with a `seen=` acknowledgement. + +**On-demand reference:** Read `.crew/templates/ralph-reference.md` for the full work-check cycle, human comment detection, watch mode, state model, board format, and follow-up integration. ### Connecting to a Repo diff --git a/templates/issue-lifecycle.md b/templates/issue-lifecycle.md index c5dacf291..4c2e16b40 100644 --- a/templates/issue-lifecycle.md +++ b/templates/issue-lifecycle.md @@ -320,6 +320,61 @@ When spawning an agent to work on an issue, include this context block: 4. Report PR URL to coordinator ``` +## Agent Comment Signing + +Crew agents post through `gh`, which authenticates as the operator. **Every agent +comment is therefore authored by the human's GitHub account.** Author is useless as a +discriminator, and style heuristics ("agents write long comments with `##` headers") +break the moment either party writes atypically. + +Sign agent comments instead. + +### The marker + +Append this as the **last line** of the body of every comment an agent posts to an +issue or a PR: + +``` + +``` + +- `{member}` is the crew member's roster name, lowercased (`link`, `sparks`, `ralph`). +- It renders as nothing on GitHub, survives comment edits, and is one `grep` away. +- **Anything without the marker is, by definition, a human comment.** That inverse is + the point: the marker is what makes human replies detectable at all. + +Applies to `gh issue comment`, `gh pr comment`, `gh pr review --body`, the equivalent +GitHub MCP tools, and any workflow that posts on an agent's behalf. It does **not** +apply to commit messages, PR bodies, or issue bodies — comments only. + +### Acknowledging a human reply — the `seen=` field + +When an agent comment is a **response to human input on that thread**, add a `seen=` +field carrying the ISO-8601 timestamp of the newest human comment the agent read: + +``` + +``` + +`seen=` is the per-issue high-water mark for human comments. Ralph treats every +unmarked comment newer than the highest `seen=` on that issue as unreviewed and keeps +surfacing it. A marked comment **without** `seen=` does not advance the mark — a +status update posted while a human question is still outstanding must not silence +that question. + +Acknowledge honestly. `seen=` asserts "an agent read this and responded to it". It is +not a dismissal button. + +### Anti-patterns + +- ❌ Omitting the marker "just this once" — one unmarked agent comment is a false human alert. +- ❌ Putting the marker anywhere but the last line. +- ❌ Setting `seen=` to the current time — the value is the timestamp of the comment + being acknowledged, not the time of acknowledgement. +- ❌ Setting `seen=` on a comment that did not actually address the human input. +- ❌ Back-filling markers onto historical comments — use an adoption cutoff instead + (see `ralph-reference.md` → "Adoption cutoff"). + ## Ralph's Role in Issue Lifecycle Ralph (the work monitor) continuously checks issue and PR state: @@ -402,6 +457,8 @@ All PRs reviewed → All PRs merged → Epic closed - ❌ Using `checkout -b` when parallel agents are active (causes working directory conflicts) - ❌ Manually transitioning issue states — let the platform and Crew automation handle it - ❌ Skipping the branch naming convention — breaks Ralph's tracking logic +- ❌ Posting an agent comment without the `` marker — it will + be misread as a human reply and re-routed for review ## Migration Notes diff --git a/templates/ralph-reference.md b/templates/ralph-reference.md index d1c3ce047..1dc83598d 100644 --- a/templates/ralph-reference.md +++ b/templates/ralph-reference.md @@ -44,6 +44,8 @@ gh pr list --state open --json number,title,author,labels,isDraft,reviewDecision # Draft PRs (agent work in progress) gh pr list --state open --draft --json number,title,author,labels,checks --limit 20 + +# Unreviewed human comments on open issues (see "Human Comment Detection" below) ``` **Step 2 — Categorize findings:** @@ -51,6 +53,7 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | Category | Signal | Action | |----------|--------|--------| | **Untriaged issues** | `crew` label, no `crew:{member}` label | Lead triages: reads issue, assigns `crew:{member}` label | +| **Human comments** | Comment on an open issue with no `` marker, newer than that issue's high-water mark | Report issue + gist, route to the issue's `crew:{member}` for review | | **Assigned but unstarted** | `crew:{member}` label, no assignee or no PR | Spawn the assigned agent to pick it up | | **Draft PRs** | PR in draft from crew member | Check if agent needs to continue; if stalled, nudge | | **Review feedback** | PR has `CHANGES_REQUESTED` review | Route feedback to PR author agent to address | @@ -59,7 +62,10 @@ gh pr list --state open --draft --json number,title,author,labels,checks --limit | **No work found** | All clear | Report: "📋 Board is clear. Ralph is idling." Suggest `npx @blacklite/crew-cli watch` for persistent polling. | **Step 3 — Act on highest-priority item:** -- Process one category at a time, highest priority first (untriaged > assigned > CI failures > review feedback > approved PRs) +- Process one category at a time, highest priority first (human comments > untriaged > assigned > CI failures > review feedback > approved PRs) +- **Human comments sort first on purpose.** A human reply can supersede a recommendation the + crew is already acting on; surfacing it before spawning more work is what stops the crew + building on advice that has been overturned. - Spawn agents as needed, collect results - **⚡ CRITICAL: After results are collected, DO NOT stop. DO NOT wait for user input. IMMEDIATELY go back to Step 1 and scan again.** This is a loop — Ralph keeps cycling until the board is clear or the user says "idle". Each cycle is one "round". - If multiple items exist in the same category, process them in parallel (spawn multiple agents) @@ -77,6 +83,132 @@ After every 3-5 rounds, pause and report before continuing: **Do NOT ask for permission to continue.** Just report and keep going. The user must explicitly say "idle" or "stop" to break the loop. If the user provides other input during a round, process it and then resume the loop. +### Human Comment Detection + +A reply from the operator on an issue is a work signal, and it is the one signal Ralph +historically missed: the scan covers issues, PRs, labels and checks, but not comments. A +human answer could sit in a thread indefinitely while the crew kept acting on advice that +answer had already overturned. + +**Why author filtering does not work.** Crew agents post through `gh`, authenticated as the +operator, so agent comments and human comments carry the *same* `author.login`. Filtering on +author returns everything. Style heuristics are worse — they misfire the first time either +party writes atypically. + +Detection depends on the marker convention in `issue-lifecycle.md` → "Agent Comment Signing": +every agent comment ends with ``, so **an unmarked comment is a +human comment by definition**. Ralph does not need to recognise humans; it only needs to +recognise agents, which it can do exactly. + +#### High-water mark + +Two levels, deliberately: + +| Level | Lives in | Suppresses | +|-------|----------|------------| +| **Session** | Ralph's in-session state (`commentsReported`, a set of `{issue, createdAt}`) | Re-reporting the same comment on every round of the same session | +| **Durable** | The `seen=` field on agent comments **in the thread itself** | Re-reporting across sessions, machines and worktrees | + +The durable mark lives on GitHub, not on disk, and that is the design choice worth +defending. A file under `.crew/` would be a mutable, per-checkout, merge-conflicting +record of something GitHub already stores; agents run in worktrees, in CI and on more +than one machine, and a local file is wrong in all of those the moment two of them run. +Putting the mark in the thread means the thread is self-describing: anyone — Ralph, a +member, a human reading the issue — can tell what has been acknowledged from the issue +alone. + +Per-issue mark = the highest `seen=` value across that issue's marked comments. A marked +comment with no `seen=` does **not** advance it: an agent posting a status update while a +human question is outstanding must not silence the question. An unacknowledged human +comment therefore keeps being reported every session until someone actually answers it — +that nagging is the feature. + +#### Adoption cutoff + +Comments posted **before** the signing convention was adopted carry no marker, so a naive +first run classifies the entire back-catalogue of agent reports as new human comments. + +Set a floor once, at adoption, in `.crew/config.json`: + +```json +{ + "commentWatch": { + "repo": "{owner}/{repo}", + "since": "{ISO-8601 timestamp of adoption}" + } +} +``` + +Comments at or before `since` are never reported, marked or not. This is the only honest +option: the marker cannot retroactively classify comments that predate it, and back-filling +markers means editing historical comments in someone else's thread. Do a **one-time manual +sweep** of pre-cutoff threads when adopting, act on anything genuinely unanswered, and then +let the cutoff hold the line. + +`.crew/config.json` is operator config — `crew upgrade` reads it and never rewrites it — so +the cutoff survives upgrades. + +#### The scan + +One GraphQL call for the whole tracker, not one REST call per issue. The per-issue loop +(`gh issue list` → `gh issue view` per number) works but is O(open issues) requests per +cycle, and Ralph cycles continuously. + +```bash +CUTOFF=$(jq -r '.commentWatch.since // "1970-01-01T00:00:00Z"' .crew/config.json) +REPO=$(jq -r '.commentWatch.repo' .crew/config.json) + +gh api graphql -F owner="${REPO%%/*}" -F repo="${REPO##*/}" -F issues=50 -F comments=30 -f query=' + query($owner:String!, $repo:String!, $issues:Int!, $comments:Int!) { + repository(owner:$owner, name:$repo) { + issues(states:OPEN, first:$issues, orderBy:{field:UPDATED_AT, direction:DESC}) { + nodes { + number title + labels(first:20) { nodes { name } } + comments(last:$comments) { nodes { createdAt author { login } body } } + } + } + } + }' | jq -r --arg cutoff "$CUTOFF" ' + def marked: (.body // "") | test("`, which + advances the durable mark. + +Board line: + +``` +💬 Human comments: 2 unreviewed (#84 → link, #81 → sparks) +``` + ### Watch Mode (`crew watch`) Ralph's in-session loop processes work while it exists, then idles. For **persistent polling** between sessions or when you're away from the keyboard, use the `crew watch` CLI command: @@ -108,6 +240,12 @@ Ralph's state is session-scoped (not persisted to disk): - **Round count** — how many check cycles completed - **Scope** — what categories to monitor (default: all) - **Stats** — issues closed, PRs merged, items processed this session +- **Comments reported** — `{issue, createdAt}` pairs surfaced this session, so a comment is + reported once per session rather than once per round + +Session state deliberately does **not** hold the comment high-water mark. A session-scoped +mark re-reports every old comment at the start of every new session; the durable mark lives +in the thread as `seen=` (see [Human Comment Detection](#human-comment-detection)). ### Ralph on the Board @@ -117,6 +255,7 @@ When Ralph reports status, use this format: 🔄 Ralph — Work Monitor ━━━━━━━━━━━━━━━━━━━━━━ 📊 Board Status: + 💬 Human input: 1 unreviewed comment (#84 → link) 🔴 Untriaged: 2 issues need triage 🟡 In Progress: 3 issues assigned, 1 draft PR 🟢 Ready: 1 PR approved, awaiting merge