docs(e2e): record E4 agent-binding verification result (PASS, n=1) - #1813
Merged
Conversation
Adds .squad/e2e/E4-agent-binding-verification.md: a runnable, not-yet-executed procedure to verify #1784 (plan activate minting squad:lead/squad:devrel as owner labels instead of cast names) once Procedures' fix merges. Targets ~27 min instead of a full ~54 min long-path run by stopping at `plan implementation`, which is where the Agent-column leak originates (workflows/squad.md:913) rather than where it surfaces (:730). Key contents: - Three-way PASS/PARTIAL/FAIL criterion, stated before the run. squad:devrel is dispositive (DevRel appears nowhere in the fixture roster, verified), while squad:lead is ambiguous (Keaton's Role column reads "Lead / Architect"). A binary criterion would score a real partial fix as FAIL. - Fixture-refresh phase covering the traps that produce false results: two files named squad.md (56,525 vs 6,688 bytes), two independently stale surfaces (compiled lock + committed source), GH_AW_INFO_FRONTMATTER_SOURCE being provenance only, and PowerShell backtick/ConvertFrom-Json pitfalls. - Seed issue shaped for >=2 sibling epics per EECOM, with an explicit guard rail against contorting the fixture to force that shape. - #1779 / PR #1787 refill verification kept as a clearly separated follow-on with its own criterion, marked not-unattended-safe. Refs #1784. Deliberately does not close it: this procedure has not been executed and #1784 must not be marked verified until it runs green post-merge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… label check Read run 32433493989 (plan implementation) and 32435055598 (plan activate) directly rather than relying on the relayed observation. Briefed counts confirmed exactly, plus four findings — one of which was a defect in this procedure. Confirmed: Agent column is `lead`, `lead`, `devrel` (lead x2, devrel x1); labels minted are squad:lead x3 + squad:devrel x1. New findings: - Squad's own "Validation Pre-check" printed "Agent assignments valid (cast Names) | OK (lead, lead, devrel)" — it passes on exactly the input it should reject. Added an explicit instruction not to trust it. - Zero legitimate cast-name labels have ever been minted (all five roster names ABSENT). The binding has never once worked. - squad:reviewer also leaked, in E1 (#9) — so all three tokens named in the two prohibitions have leaked, and nothing outside that set ever has. Added squad:reviewer to the criterion as a second dispositive token. - The plan program artifact contains NO owner assignment, so epic #17's squad:lead is minted at plan activate — downstream of where E4 stops. Documented as a scope limitation: a green E4 means the Agent column is clean, not that no label leaks. Procedure defect fixed: Phase 3b checked repo-global label existence, but labels persist once created and squad:lead/devrel/reviewer already exist from E1 and E3. That check returns true forever and would have failed a perfect run. Rescoped to issues created after an $E4_START cutoff, with the pre-existing contamination tabulated. Refs #1784. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… sync The fixture had drifted: workflows/squad.md was 4,665 bytes behind dev and squad-implement-worker.md 1,731 behind, so E4 would have executed the pre-#1789 prompt. Synced and recompiled (fixture commit 1b11c5f); all five prompt sources now byte-identical to dev by blob SHA. Phase 0d changes: - Assert BOTH directions. #1789 replaced the old prohibition rather than appending, so "never mint a role-derived" must be ABSENT, not just Check 10 present. An addition-only check passes a file carrying both rules at once, which is worse than either alone. - Replace the speculative DevRel/squad:lead probe with the now-known markers (Check 10 / sole source of truth / Non-roster), since #1789 has merged. - Warn explicitly against asserting on squad.lock.yml. The lock does not inline the prompt body; it emits {{#runtime-import .github/workflows/squad.md}} resolved at runtime, so the markers cannot appear there and checking it returns a guaranteed false FAIL. The lock's body_hash is what proves a recompile ingested the new source. Each artifact asserted for what it can actually represent. - Fix the gh api call: add ?ref=main and the -join/-replace needed before FromBase64String. Refs #1784, #1789. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
E4 executed 2026-08-21 against fixture bradygaster/aspiregregator-squad-e2e (seed #22). 8 gates, 8 green on first attempt, zero interventions, ~55 min. Verdict PASS on all four conditions: - Cond 0: 15 squad-authored issues created by activate (#23-#37) - Cond 1: Agent column 11/11 verbatim roster names, zero role-derived tokens - Cond 2: 15 timeline labeled events, all 'squad'; zero unlabeled - Cond 3: activation summary reports the missing-label prerequisite gap Same-fixture control: E3 produced 'lead, lead, devrel' on this exact column. Two qualifications recorded so the result is not overstated: - The activate-stage roster read still returns the generic uncast vocabulary (lead/reviewer/devrel/security/docs) while citing a file whose Name column is Keaton/McManus/Fenster/Hockney/Kint. Cond 2's green is structural (no create-label configured), not earned. Likely a separate defect from #1784. - The fixture cannot express a correct owner label at all; closing #1784's user-visible symptom needs an issues:write + create-label change. Also adds a known-gap note that the criterion detects wrong vocabulary, not wrong assignment, and records the first live #1787 observation: all 15 issues parent to root, epics have zero sub-issues, so refill draws from the root. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34df7b7a-c954-4109-99c7-7c20c72bf2a0
Contributor
🟢 Impact Analysis — PR #1813Risk tier: 🟢 LOW 📊 Summary
🎯 Risk Factors
📦 Modules Affectedsquad-state (1 file)
This report is generated automatically for every PR. See #733 for details. |
Contributor
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ❌ | Single commit | 6 commits — consider squashing before review |
| ✅ | Not in draft | Ready for review |
| ❌ | Branch up to date | dev is 1 commit(s) ahead — rebase recommended |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | No source files changed — changeset not required |
| ✅ | Scope clean | |
| ✅ | No merge conflicts | No merge conflicts |
| ❌ | Copilot threads resolved | 2 unresolved Copilot thread(s) — fix and resolve before merging |
| ✅ | CI passing | All checks passing |
Files Changed (1 file, +124 −14)
| File | +/− |
|---|---|
.squad/e2e/E4-agent-binding-verification.md |
+124 −14 |
Total: +124 −14
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Contributor
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an E2E documentation artifact under .squad/ that records the executed outcome of the E4 “Agent-Binding Verification” procedure (PASS, n=1) and preserves the full runnable procedure for future reruns.
Changes:
- Adds
.squad/e2e/E4-agent-binding-verification.mdwith an executed-results section (fixture, timestamps, condition table, qualifications). - Retains the complete pre-execution procedure (fixture refresh, short-path run, inspection steps, evidence checklist, and follow-on notes).
Show a summary per file
| File | Description |
|---|---|
.squad/e2e/E4-agent-binding-verification.md |
New E2E verification writeup: records E4’s executed result and includes the runnable procedure + evidence capture guidance |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 1/1 changed files
- Comments generated: 4
- Review effort level: Lite
Comment on lines
+11
to
+12
| Run against fixture `bradygaster/aspiregregator-squad-e2e`, seed issue **#22**, | ||
| `$E4_START` = `2026-08-21T09:18:32Z`. **8 gates, 8 green, zero interventions, ~55 min.** |
| # false negatives on prompt text (which is full of backticks). Use .Contains(). | ||
| $b = (gh api "repos/$FIXTURE/contents/.github/workflows/squad.md?ref=main" --jq '.content' 2>$null) -join '' -replace '\s','' | ||
| $txt = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($b)) | ||
| Write-Host "main squad.md length: $($txt.Length)" # expect ~60,589 chars, NOT ~6.6 KB |
Comment on lines
+490
to
+492
| gh issue view $ISSUE --repo $FIXTURE --json comments ` | ||
| --jq '.comments[-1].body' 2>$null | | ||
| Set-Content (Join-Path $EVIDENCE "plan-implementation-comment.md") |
Comment on lines
+494
to
+497
| # Pull every Agent cell out of the plan's task table | ||
| $plan = [IO.File]::ReadAllText((Join-Path $EVIDENCE "plan-implementation-comment.md")) | ||
| Write-Host "`n=== Agent column values ===" | ||
| [regex]::Matches($plan, '(?m)^\|.*$') | ForEach-Object { $_.Value } |
…is stage-local Two corrections to the E4 result, both measured: 1. REFUTED: the claim that with create-label enabled this run would have minted squad:lead again. activate applies what the plan hands it; E3's plan column was 'lead, lead, devrel' (exactly its minted set) while E4's contains only cast names, so there is no role token to apply. create-label is required for the correct behaviour, not what suppressed the defect. Condition 2's green is still structural, but the structure is the roster mismatch rather than the missing permission. 2. The activate roster read is stage-local and provably capable of being correct: plan validate in the same walk, 23 minutes earlier, read the same file/section/column correctly and verified all four Agent values verbatim via Check 10. Not a wrong path, not a hardcoded default, not a shared component - a working reference implementation exists one stage earlier. Also records that E3's validate enumerated no roster at all (false accept by omission, not by wrong roster), so #1789 added the roster echo and Check 10 at validate and left activate untouched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34df7b7a-c954-4109-99c7-7c20c72bf2a0
…fication-prep Resolve add/add conflict in .squad/e2e/E4-agent-binding-verification.md. Resolution: keep HEAD version (executed result + original procedure) which is a semantic superset of dev branch (procedure only). dev added E1 relay and windows-test-baseline files; both are included in the resolved tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the executed result of the E4 procedure added in #1791.
Working as Sims (E2E Test Engineer). Docs-only,
.squad/only.Result — 🟢 PASS on all four conditions
E4 ran 2026-08-21 against fixture
bradygaster/aspiregregator-squad-e2e, seed issue #22,$E4_START=2026-08-21T09:18:32Z(recorded before dispatch). 8 gates, 8 green on first attempt, zero human interventions, ~55 min.activate(excl. seed #22)Agentcolumn = verbatim roster namessquad:lead/devrel/reviewerin timelinelabeledeventssquad; zerounlabeledSame-fixture, same-day control: E3 produced
lead, lead, devrelon this exact column; E4 producedMcManus, Keaton, Fenster, Hockney…. One prompt change (#1789) between them.research → plan implementationran 26.3 min against 26.7 budgeted — the planning-path budget is now measured twice.🛑 Scope bound
Green means "the
Agentcolumn is clean on one artifact, n=1, salience path" — not "#1784 is retired."🔴 Finding A — the
activateroster read is wrong, and cites a file that disproves itThe activation summary claims it read
.squad/team.md## Members→Nameand gotlead, reviewer, devrel, security, docs. That file's Name column isKeaton, McManus, Fenster, Hockney, Kint, anddevrel/security/docsappear nowhere in it. The cited set matches no single file's Name column anywhere — it's the generic uncast Squad vocabulary. That makes it a hallucinated provenance claim, not a hardcoded default.Consequently
activatedeclared the correct values "non-roster" and advises the operator to "recast the team or update the implementation plan" — i.e. to break the one part of the pipeline that is working. That advice is the operationally dangerous part.The defect is stage-local to
activateplan validatein the same walk, 23 minutes earlier, read the same file/section/column correctly:So this is not a wrong path, not a hardcoded default, and not a shared roster-reading component. A working reference implementation exists one stage earlier; the fix is to make
activatedo whatvalidatealready does.validateAgent assignments valid (cast Names) ✅ (lead, lead, devrel)— no roster enumerated, no Check 10 ⇒ false accept by omissionactivatesquad:lead×3 +squad:devrel#1789 fixed the implementation-plan column and the
validategate — both confirmed — and left theactivateroster read untouched. Should be filed separately from #1784.An earlier revision asserted that with
create-labelenabled this run would have mintedsquad:leadagain. That is refuted.activateapplies what the plan hands it — E3's plan column waslead, lead, devrel, exactly its minted set, while E4's contains only cast names, so there is nothing to mint.create-labelis required for the correct behaviour, not what suppressed the defect. Condition 2's green is still structural, but the structure is the roster mismatch, not the missing permission.Separately, the fixture still cannot express a correct owner label at all: absence of
squad:keatonis expected and correct (the lock has zerocreate-labelreferences), and closing #1784's user-visible symptom needs anissues: write+create-labelchange.🟢 Finding B — first live #1787 observation, free
plan programproduced the 4-sibling-epic shape #1787 had never had a fixture for (E1 and E3 were both single-epic), and the seed wasn't bent to get it. Afteractivate: all 15 issues parent to root #22; every epic has 0 sub-issues. The hierarchy is flat, so refill necessarily draws from the root, not the parent epic — corroborated by the summary's own gap note thatparentis wired for epic→root only.Also in this change
Evidence for #1784, #1787, and #1812.