feat(templates): sign agent comments and detect human comments in Ralph's cycle - #3
Merged
Merged
Conversation
…ph's cycle
Crew agents post through `gh` with the operator's credentials, so agent and
human comments share an author. Ralph could not see a human reply on an issue —
the answer sat in the thread while the crew kept acting on superseded advice.
- Require `<!-- crew:agent={member} -->` on every agent-posted issue/PR comment,
making anything unmarked a human comment by definition.
- Add the `seen=` field as the durable per-issue high-water mark, stored in the
thread rather than on disk.
- Add Human Comment Detection to Ralph's work-check cycle: one GraphQL query for
the tracker, session + `seen=` two-level dedupe, and a `commentWatch.since`
adoption cutoff so pre-convention comments are not reported as human.
Refs david-driscoll/vault#105
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ✅ | Single commit | 1 commit — clean history |
| ✅ | Not in draft | Ready for review |
| ❌ | Branch up to date | dev is 1 commit(s) ahead — rebase recommended |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | Changeset file found |
| ✅ | Scope clean | No .crew/ or docs/proposals/ files |
| ✅ | No merge conflicts | Merge status unknown — GitHub is still computing |
| ✅ | Copilot threads resolved | No Copilot review threads |
| ❌ | CI passing | 1 check(s) failing: Policy Gates |
Files Changed (14 files, +975 −9)
| File | +/− |
|---|---|
.changeset/agent-comment-marker.md |
+12 −0 |
.crew-templates/crew.agent.md |
+35 −1 |
.crew-templates/issue-lifecycle.md |
+57 −0 |
.crew-templates/ralph-reference.md |
+140 −1 |
.github/agents/crew.agent.md |
+35 −1 |
packages/crew-cli/templates/crew.agent.md.template |
+35 −1 |
packages/crew-cli/templates/issue-lifecycle.md |
+57 −0 |
packages/crew-cli/templates/ralph-reference.md |
+140 −1 |
packages/crew-sdk/templates/crew.agent.md.template |
+35 −1 |
packages/crew-sdk/templates/issue-lifecycle.md |
+57 −0 |
packages/crew-sdk/templates/ralph-reference.md |
+140 −1 |
templates/crew.agent.md.template |
+35 −1 |
templates/issue-lifecycle.md |
+57 −0 |
templates/ralph-reference.md |
+140 −1 |
Total: +975 −9
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
🟠 Impact Analysis — PR #3Risk tier: 🟠 HIGH 📊 Summary
🎯 Risk Factors
📦 Modules Affectedci-workflows (1 file)
crew-cli (3 files)
crew-sdk (3 files)
root (4 files)
templates (3 files)
This report is generated automatically for every PR. See #733 for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Crew agents post through
gh, authenticated as the operator. Agent comments and human comments therefore carry the sameauthor.login. Filtering on author returns everything; style heuristics ("agents write long comments with##headers") misfire the first time either party writes atypically.The consequence is that a human reply on an issue is invisible to Ralph. Ralph's work-check cycle scans issues, PRs, labels, checks and alerts — not comments. A real instance in the estate tracker: six substantive answers were posted on one issue, two of which overturned recommendations the crew had already made, and they were only picked up because the operator asked for them explicitly. Left alone, the crew would have kept building on superseded advice.
Approach
Stop trying to recognise humans. Recognise agents — exactly — and take the complement.
1. Sign every agent comment (
crew.agent.md,issue-lifecycle.md)Last line of every comment an agent posts to an issue or PR. Invisible when rendered, survives edits, one
grepaway. Mandatory for every member and every posting path —gh issue comment,gh pr comment,gh pr review --body, GitHub MCP equivalents. Not applied to commit messages, issue bodies or PR bodies.Anything unmarked is a human comment by definition. That inverse is the whole mechanism.
2.
seen=as the durable high-water markAdded when the agent comment answers human input, carrying the timestamp of the newest human comment read. Two properties worth defending:
.crew/would be a mutable, per-checkout, merge-conflicting copy of something GitHub already stores — and agents run in worktrees, in CI and on more than one machine, where a local file is wrong in all of them. Putting the mark in the thread makes the thread self-describing.seen=does not advance it. A status update posted while a human question is outstanding must not silence that question. Unacknowledged human comments keep being reported until someone actually answers; the nagging is the feature.3. Human Comment Detection in Ralph's cycle (
ralph-reference.md)One GraphQL query for the whole tracker, replacing the O(open issues)
gh issue viewloop — Ralph cycles continuously, so per-issue REST calls scale badly. Two-level dedupe: a session-scoped set of reported{issue, createdAt}pairs stops re-reporting within a session;seen=stops it across sessions, machines and worktrees.Human comments sort first in the priority order. A human reply can supersede a recommendation the crew is already acting on, so surfacing it before spawning more work is what stops the crew building on advice that has been overturned. Ralph surfaces (issue, owner, gist) and routes to the issue's
crew:{member}; it does not interpret. The reviewer's brief requires an explicit supersession check, and closes the loop with aseen=acknowledgement.4. Adoption cutoff —
commentWatch.sincein.crew/config.jsonEvery comment posted before the convention exists is unmarked, so a naive first run classifies the entire back-catalogue of agent reports as new human comments. Verified on a live tracker: all 13 comments on open issues came back as "human". A
sincefloor is the only honest fix — the marker cannot retroactively classify comments that predate it, and back-filling means editing historical comments. Pair it with a one-time manual sweep at adoption..crew/config.jsonis operator config thatcrew upgradereads and never rewrites, so the cutoff survives upgrades.Verification
Tested against a real interleaved sample where agent and human comments alternate under the same author across four issues:
seen=acksScope
Docs/templates only — no SDK or CLI source changes. Edits are in
.crew-templates/(canonical) withscripts/sync-templates.mjsrun to propagate totemplates/,packages/crew-cli/templates/,packages/crew-sdk/templates/and.github/agents/. Changeset included (governed template path).Refs david-driscoll/vault#105