docs(e2e): add E4 agent-binding verification procedure for #1784 - #1791
Merged
Conversation
Adds .squad/e2e/E4-agent-binding-verification.md: a runnable, not-yet-executed procedure to verify #1784 (plan activate minting squad:lead/squad:devrel as owner labels instead of cast names) once Procedures' fix merges. Targets ~27 min instead of a full ~54 min long-path run by stopping at `plan implementation`, which is where the Agent-column leak originates (workflows/squad.md:913) rather than where it surfaces (:730). Key contents: - Three-way PASS/PARTIAL/FAIL criterion, stated before the run. squad:devrel is dispositive (DevRel appears nowhere in the fixture roster, verified), while squad:lead is ambiguous (Keaton's Role column reads "Lead / Architect"). A binary criterion would score a real partial fix as FAIL. - Fixture-refresh phase covering the traps that produce false results: two files named squad.md (56,525 vs 6,688 bytes), two independently stale surfaces (compiled lock + committed source), GH_AW_INFO_FRONTMATTER_SOURCE being provenance only, and PowerShell backtick/ConvertFrom-Json pitfalls. - Seed issue shaped for >=2 sibling epics per EECOM, with an explicit guard rail against contorting the fixture to force that shape. - #1779 / PR #1787 refill verification kept as a clearly separated follow-on with its own criterion, marked not-unattended-safe. Refs #1784. Deliberately does not close it: this procedure has not been executed and #1784 must not be marked verified until it runs green post-merge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ❌ | Single commit | 2 commits — consider squashing before review |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | No source files changed — changeset not required |
| ✅ | Scope clean | |
| ✅ | No merge conflicts | No merge conflicts |
| ❌ | Copilot threads resolved | 2 unresolved Copilot thread(s) — fix and resolve before merging |
| ✅ | CI passing | All checks passing |
Files Changed (1 file, +565 −0)
| File | +/− |
|---|---|
.squad/e2e/E4-agent-binding-verification.md |
+565 −0 |
Total: +565 −0
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Contributor
|
Contributor
🟢 Impact Analysis — PR #1791Risk tier: 🟢 LOW 📊 Summary
🎯 Risk Factors
📦 Modules Affectedsquad-state (1 file)
This report is generated automatically for every PR. See #733 for details. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new E2E verification procedure document for issue #1784 (agent/owner binding live leak), intended to be run post-merge of Procedures’ fix to provide a cheaper, targeted validation run.
Changes:
- Adds a runnable, step-by-step E4 procedure that stops at
plan implementationto catch the leak at theAgentcolumn source. - Defines a three-way verdict (PASS / PARTIAL / FAIL) to distinguish dispositive
devrelleakage from ambiguousleadleakage. - Documents fixture refresh traps, evidence capture expectations, and a clearly separated follow-on for #1779 / PR #1787.
Show a summary per file
| File | Description |
|---|---|
.squad/e2e/E4-agent-binding-verification.md |
New E4 runbook/procedure to verify #1784 via a shortened planning-path run and explicit verdict criteria. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Review effort level: Lite
Comment on lines
+314
to
+316
| gh issue view $ISSUE --repo $FIXTURE --json comments ` | ||
| --jq '.comments[-1].body' 2>$null | | ||
| Set-Content (Join-Path $EVIDENCE "plan-implementation-comment.md") |
Comment on lines
+318
to
+321
| # Pull every Agent cell out of the plan's task table | ||
| $plan = [IO.File]::ReadAllText((Join-Path $EVIDENCE "plan-implementation-comment.md")) | ||
| Write-Host "`n=== Agent column values ===" | ||
| [regex]::Matches($plan, '(?m)^\|.*$') | ForEach-Object { $_.Value } |
… label check Read run 32433493989 (plan implementation) and 32435055598 (plan activate) directly rather than relying on the relayed observation. Briefed counts confirmed exactly, plus four findings — one of which was a defect in this procedure. Confirmed: Agent column is `lead`, `lead`, `devrel` (lead x2, devrel x1); labels minted are squad:lead x3 + squad:devrel x1. New findings: - Squad's own "Validation Pre-check" printed "Agent assignments valid (cast Names) | OK (lead, lead, devrel)" — it passes on exactly the input it should reject. Added an explicit instruction not to trust it. - Zero legitimate cast-name labels have ever been minted (all five roster names ABSENT). The binding has never once worked. - squad:reviewer also leaked, in E1 (#9) — so all three tokens named in the two prohibitions have leaked, and nothing outside that set ever has. Added squad:reviewer to the criterion as a second dispositive token. - The plan program artifact contains NO owner assignment, so epic #17's squad:lead is minted at plan activate — downstream of where E4 stops. Documented as a scope limitation: a green E4 means the Agent column is clean, not that no label leaks. Procedure defect fixed: Phase 3b checked repo-global label existence, but labels persist once created and squad:lead/devrel/reviewer already exist from E1 and E3. That check returns true forever and would have failed a perfect run. Rescoped to issues created after an $E4_START cutoff, with the pre-existing contamination tabulated. Refs #1784. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 21, 2026
bradygaster
added a commit
that referenced
this pull request
Aug 21, 2026
…ch cross-checks (#1820) Follow-up to #1819 addressing all four review findings plus the Phase 0b fixture-freshness fail-open found during review. E4: Phase 0b gains a third UNREADABLE state so a double gh failure can no longer collapse into MATCH via \\ -eq \\; 0c post-condition now requires all four MATCH AND none UNREADABLE; 0d, 3c and 3e(B) guarded with distinct INCONCLUSIVE reasons; 3e(B) asserts both sets non-empty so SetEquals(empty, empty) cannot cancel two failures into a green; the 3e caveat now correctly documents fail-closed routing. E1: Rule D FAIL redefined as 'epic OPEN at window expiry, full stop', with attempt evidence demoted from precondition to recorded detail; late closure explicitly FAIL and non-retroactive; escape hatch gains three veracity cross-checks. Reviewed-by: Flight (Lead/Architect) - approved with follow-ups. All four #1819 findings verified fixed; Phase 0b guard ordering confirmed correct. Review overturns the Phase 3b clearance: 'gh' has no --arg flag, so the L625 labels query fails every run and vacuously passes. Pre-existing (PR #1791), untouched by this diff, tracked as P0 follow-up. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.squad/e2e/E4-agent-binding-verification.md— a runnable, not-yet-executed procedure to verify #1784 once Procedures' fix merges.Also folds E3's findings into the E2E runbook, which lives outside the repo (
Desktop\squad-e2e-runbook.md, 18,745 → 25,633 bytes) and is therefore not part of this diff.Why
E3 walked the full long planning path and surfaced #1784:
plan activatemintedsquad:lead×3 andsquad:devrel×1 as owner labels instead of cast names. Root cause is that the prompt's prohibitions spell out the forbidden tokens concretely (workflows/squad.md:730,:913) and the model copied them verbatim out of the parenthetical forbidding them.When the fix lands we need a cheap, targeted live verification — not another ~54-minute full path.
Two findings that shaped the doc
1. The procedure stops at
plan implementation, notplan activate. The leak originates in theAgentcolumn at:913and only surfaces as labels at:730. Catching it at the source cuts the run from ~54 min to ~27 min — a figure derived from E3's actual sub-window (00:18:10 → 00:44:54 = 26.7 min), not estimated.2. The verdict is three-way, not binary. The two leaked tokens are not equal evidence:
squad:devrelis dispositive. Verified against the fixture roster:DevRel→ False,devrel→ False. It appears nowhere in the fixture — not as a Name, not as a Role. Its only possible source is the prohibition text.squad:leadis ambiguous. Keaton's Role column literally reads "Lead / Architect", so it could be role-column derivation or prohibition-copying.⇒ A binary criterion would score "devrel cleared, lead persists" as FAIL and argue for reverting a fix that genuinely worked. PARTIAL exists to prevent that.
Also in the doc
squad.md(56,525 vs 6,688 bytes; I got four false negatives off the wrong one), two independently stale surfaces,GH_AW_INFO_FRONTMATTER_SOURCEbeing provenance only, and the PowerShell backtick /ConvertFrom-Jsonpitfalls. Byte checks are comparative, never hardcoded — the fix changes the sizes, so a hardcoded expectation would itself become a silent-success trap.Runbook changes (outside this PR)
app/github-actions), not repo settings — those are already permissive..git/info/exclude-excluded path that reported success. Three occurrences of one shape is a class, and naming the class is what stops the fourth.Verification
Nothing was fired — #1784 is OPEN and PR #1787 is OPEN/unmerged, both confirmed this session. All fixture reads were read-only
gh apicalls.Staging was checked per the data-loss postmortem:
.squad/is excluded at.git/info/exclude:9, so the file neededgit add -fand was confirmed tracked viagit ls-files --error-unmatch(exit 0). Deletion check clean. The three pre-existing churn files (docs/pagefind.yml, twosamples/**/*.sh) were left unstaged.Refs #1784. Deliberately does not close it.