fix(workflow): harden safe-outputs reliability in Plan Activate - #1683
Closed
bradygaster wants to merge 2 commits into
Closed
fix(workflow): harden safe-outputs reliability in Plan Activate#1683bradygaster wants to merge 2 commits into
bradygaster wants to merge 2 commits into
Conversation
- Increase create-issue max from 50 to 75 (supports larger plans) - Add label pre-flight validation before issue creation - Add transient failure handling with single retry for 5xx errors - Add output budget awareness to prevent silent partial creation - Add graceful fallback for sub-issue API failures (404/422) - Recommend phase-by-phase activation for large plans (>15 tasks) Root cause: multiple failure modes compound during Plan Activate — token exhaustion, missing labels in new repos, sub-issue API failures, and no retry logic. These changes add defense-in-depth. Ref: #1678 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
🔴 Impact Analysis — PR #1683Risk tier: 🔴 CRITICAL 📊 Summary
🎯 Risk Factors
📦 Modules Affectedroot (84 files)
This report is generated automatically for every PR. See #733 for details. |
Contributor
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ❌ | Single commit | 2 commits — consider squashing before review |
| ✅ | Not in draft | Ready for review |
| ❌ | Branch up to date | dev is 1 commit(s) ahead — rebase recommended |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | Changeset file found |
| ✅ | Scope clean | No .squad/ or docs/proposals/ files |
| ✅ | No merge conflicts | No merge conflicts |
| ❌ | Copilot threads resolved | 5 unresolved Copilot thread(s) — fix and resolve before merging |
| ✅ | CI passing | All checks passing |
Files Changed (84 files, +149 −1422)
| File | +/− |
|---|---|
.changeset/1454-init-confirmation-roster.md |
+0 −6 |
.changeset/bump-otel-sdk-2x.md |
+0 −5 |
.changeset/bundle-missing-skills-1289-1264.md |
+0 −39 |
.changeset/canary-false-positives.md |
+0 −6 |
.changeset/casting-identity-not-state.md |
+0 −5 |
.changeset/cli-upgrade-fixes.md |
+0 −12 |
.changeset/conditional-state-gitignore.md |
+0 −10 |
.changeset/consolidate-watch-agent-spawn.md |
+0 −5 |
.changeset/consolidated-0.11.0.md |
+107 −0 |
.changeset/context-tier.md |
+0 −15 |
.changeset/docs-platform-path-fixes.md |
+0 −6 |
.changeset/feat-1170-update-check-json.md |
+0 −12 |
.changeset/feat-1314-ralph-instructions-template.md |
+0 −28 |
.changeset/feat-cost-policy-1080.md |
+0 −27 |
.changeset/feat-preset-install.md |
+0 −60 |
.changeset/fix-1014-parallel-spawn-warning.md |
+0 −6 |
.changeset/fix-1126-skills-to-github-skills.md |
+0 −68 |
.changeset/fix-1142-ralph-triage-ghe-api-host.md |
+0 −10 |
.changeset/fix-1190-esm-patch-doctor-commit-hooks.md |
+0 −5 |
.changeset/fix-1211-deletedir-external-stub.md |
+0 −6 |
.changeset/fix-1296-stop-mcp-home-pollution.md |
+0 −40 |
.changeset/fix-1299-fact-checker-full-plumbing.md |
+0 −77 |
.changeset/fix-1299-fact-checker-roster-instructions.md |
+0 −54 |
.changeset/fix-1305-state-backend-handshake.md |
+0 −67 |
.changeset/fix-1396-export-external-state.md |
+0 −5 |
.changeset/fix-1397-1398-external-state-reads.md |
+0 −5 |
.changeset/fix-1399-cast-external-state.md |
+0 −6 |
.changeset/fix-1441-pin-actions-sha.md |
+0 −6 |
.changeset/fix-1639-self-pull-stash-loss.md |
+0 −5 |
.changeset/fix-cli-sdk-version-range.md |
+0 −5 |
.changeset/fix-cli-sdk-workspace-pin.md |
+0 −5 |
.changeset/fix-coordinator-inline-dispatch-gate.md |
+0 −24 |
.changeset/fix-copilot-sdk-type-compat.md |
+0 −12 |
.changeset/fix-dep0190-shell-spawn.md |
+0 −11 |
.changeset/fix-dispatch-enforcement-policy-gate.md |
+0 −8 |
.changeset/fix-doctor-two-layer-decisions.md |
+0 −5 |
.changeset/fix-duplicate-cast-help.md |
+0 −5 |
.changeset/fix-externalize-binary-corruption.md |
+0 −5 |
.changeset/fix-fact-checker-auto-scaffold.md |
+0 −16 |
.changeset/fix-green-suite.md |
+0 −5 |
.changeset/fix-identity-write-author-validation.md |
+0 −8 |
.changeset/fix-permission-contract.md |
+0 −6 |
.changeset/fix-preset-routing-roundtrip.md |
+0 −5 |
.changeset/fix-release-pipeline-versions.md |
+0 −14 |
.changeset/fix-routing-strip-example-quotes.md |
+0 −9 |
.changeset/fix-sdk-dotted-tool-names.md |
+0 −15 |
.changeset/fix-sdk-export-gitignore-state.md |
+0 −11 |
.changeset/fix-skill-squad-rename-to-squad-help.md |
+0 −49 |
.changeset/fix-squad-home-env-bugs.md |
+0 −12 |
.changeset/fix-squad-slash-command.md |
+0 −76 |
| ... | +34 more files |
Total: +149 −1422
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the /squad plan activate (Plan Activate) workflow guidance to reduce safe-output reliability failures during large activations, addressing cases where issue creation stops early or fails due to repo/environment preconditions.
Changes:
- Raises the
safe-outputs.create-issue.maxlimit (50 → 75) to reduce silent caps on large activations. - Adds precondition guidance (label pre-flight), transient failure retry guidance, and output-budget-aware activation guidance.
- Documents graceful fallback behavior when GitHub sub-issue relationships cannot be created.
Show a summary per file
| File | Description |
|---|---|
| workflows/squad.md | Adjusts safe-output caps and strengthens Plan Activate operational guidance (labels, retries, budgeting, sub-issue fallback). |
Review details
Tip
Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 1/1 changed files
- Comments generated: 5
- Review effort level: Lite
Comment on lines
+2397
to
+2400
| 1. Check if the `squad` label exists using `gh label list --search squad`. | ||
| 2. If missing, create it: `gh label create squad --description "Squad-managed work item" --color 0075ca` | ||
| 3. For each agent label needed (`squad:{name}`), check and create if missing: | ||
| `gh label create "squad:{name}" --description "Assigned to {name}" --color e4e669` |
Comment on lines
+2469
to
+2472
| - **> 50 issues:** STOP and post a comment: | ||
| *"This plan requires {N} issues, which exceeds the safe single-run budget. | ||
| Use `/squad plan activate phase {N}` to activate one phase at a time."* | ||
| Then end the run. Do NOT attempt partial creation. |
Comment on lines
+2474
to
+2476
| > **Recommended:** For plans with more than 3 epics or 15 total tasks, | ||
| > prefer `/squad plan activate phase {N}` over full activation. Phase-by-phase | ||
| > is more reliable and stays within output budget limits. |
Comment on lines
+2454
to
+2459
| > If a `create-issue` or `add-comment` call returns a 5xx error or timeout: | ||
| > | ||
| > 1. **Retry once** — repeat the identical safe-output call. | ||
| > 2. **If retry fails** — STOP and report per Hallucination Guard rules above. | ||
| > 3. **Do NOT retry on 4xx errors** — these indicate a real problem (label | ||
| > missing, max limit reached, malformed body). Report immediately. |
Comment on lines
+2463
to
+2465
| Before beginning issue creation, count the total issues to create (epics + | ||
| tasks for the target scope): | ||
|
|
Consolidates accumulated changesets from the 0.11.0 development cycle into a single changelog entry. Unblocks CI changeset drift check (threshold: 25). No version bump or publish — just consolidation for CI health. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
🏗️ Architectural Review
Automated architectural review — informational only. |
bradygaster
added a commit
that referenced
this pull request
Aug 12, 2026
fix(workflow): forward-port safe-output reliability hardening from #1683
pull Bot
pushed a commit
to nagyist/squad
that referenced
this pull request
Aug 12, 2026
…adygaster#1683 - Increase create-issue max from 50 to 75 (supports larger plans) - Add Output Budget Awareness: phasing guidance when >50 issues, compact bodies when >30 - Add Label Pre-flight: ensure squad/squad:{agent} labels exist before first create-issue - Add Transient Failure Handling: single retry on 5xx, skip+report on second failure or 4xx - Add Sub-issue Fallback: degrade gracefully on 404/422, record parent as body reference - Extend gh-aw-quality tests: assert max=75, lock all four hardening behaviors, add headroom regression guard Architecture: changes are terse inline guidance within Plan Activate — no new phases, no speculative redesign, compressed prompt architecture preserved (33 KB, ~67 KB headroom). Label pre-flight uses safe-output permissions (issues:read is correct per gh-aw platform layer). Manual forward-port required: PR bradygaster#1683 branch contains pre-compression 112 KB squad.md — rebasing/merging would restore the uncompressed file and violate the 100 KB gh-aw ceiling. Refs bradygaster#1678 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d9bcc6-d667-485b-bccb-738bd9842102
bradygaster
added a commit
that referenced
this pull request
Sep 1, 2026
…cit (#1961) (#1964) * fix(gh-aw): prevent silent truncation of activation label operations Working as FIDO (Quality Owner). A 50-issue activation could lose `add_labels` operations while the run still reported success. Two causes: 1. `add-labels: max: 80` was below the worst case under the reading gh-aw's own injected prose invites. The compiler emits "Maximum {max} label(s) can be added", which reads as a budget of label NAMES; a full 50-issue activation applying `squad` + `squad:{agent}` needs 100. An agent taking that phrasing literally could conclude it had overrun and stop labeling early or batch issues together. 2. gh-aw v0.87.10 drops an over-limit item rather than failing. Its collector (`collect_ndjson_output.cjs`) rejects the item and `continue`s, pushing a string into `errors`; those are emitted with `core.warning`, never `core.setFailed`. The run finishes green with label operations missing and nothing announces it. Verified against the pinned runtime rather than inferred: `max` caps safe-output ITEMS (tool calls) per type, not label names inside a call. A two-label call costs one item. Capacity: largest supported activation is 50 issues (`enterprise` profile `max_issues: 50`, the highest documented profile limit, and the same threshold `squad-plan-activate` uses to force phased activation). Worst case at that size is 50 `create-issue` items, 50 `add_labels` calls, <=2 labels per call, 100 label names across the run. Changes: - `add-labels: max` 80 -> 110, sized to cover the worst case under BOTH readings (50 calls, 100 names) so no interpretation of the cap can justify dropping a label operation. `create-issue` stays at 75. - Record the derivation and the item-vs-label-name semantics in the activation skill, plus the fact that an over-limit item is dropped, not failed. - New Step 2e Label-Operation Reconciliation: track `activated` vs `labeled`, count a never-made/rejected/errored call as unlabeled, and on shortfall call `report_incomplete` naming the affected work items. gh-aw treats that as a failure signal even on successful exit, so a truncated activation can no longer be recorded as clean. Cap exhaustion becomes a nameable cause when observed; #1683's rule against guessing at caps is preserved and narrowed. Tests: `test/gh-aw-activation-capacity.test.ts` (27) covers the derived maximum, capacity under both readings, at-maximum and one-over boundaries, and the reconciliation contract. Compiled-artifact assertions prefer runtime evidence over prose: the declared cap reaching `GH_AW_SAFE_OUTPUTS_CONFIG`, the injected constraint carrying the same number, `report_incomplete` being present (and so callable), and agent-job permissions staying `issues: read`. Fails closed if `gh aw` is absent, per #1834. Mutation-checked: reverting the cap to 80 fails 4. No changes to temporary-ID linkage (#1962), fast-path parity (#1959), label result reporting (#1963), broad contract coverage (#1960), or E4 (#1958). Step 2e is deliberately neutral about how `add_labels` identifies its target so it does not conflict with #1962. Closes #1961 Parent #1957 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(gh-aw): correct runtime claims in activation capacity safeguards Independent review found three factual defects in the prose and tests shipped by the first commit. All three are corrected against the pinned gh-aw v0.87.10 sources rather than gh-aw's own injected descriptions. 1. `report_incomplete` does NOT make the run conclude non-successfully. `report_incomplete_handler.cjs` emits `core.warning` only, and `handle_agent_failure.cjs` contains zero `setFailed`/`process.exit` calls in 4453 lines. What it actually does is open or update a durable `[aw] {workflow} reported incomplete result` tracking issue. Step 2e now states that, says the run still reports success, and tells the agent never to rely on a red run to carry the signal. gh-aw's own tool description ("treated as a failure signal even when the agent exits successfully") is misleading; trusting it was exactly the injected-prose mistake #1961 exists to prevent. Limitation, stated honestly: no narrow mechanism in the pinned runtime makes an incomplete activation conclude red. Forcing one would require a custom safe-job, which is outside this issue's scope. The durable tracking record plus the "never report a clean activation you did not perform" rule are what carry the signal. 2. Cap enforcement is dual, not collection-only. Per Safe Outputs MCE4, `enforcePerTypeMax` in `safe_outputs_handlers.cjs` throws JSON-RPC `E002: {type} limit reached` at invocation time, which the agent DOES see; the collector then drops surplus items with a warning. Removed the claim that an over-limit item "never appears as an error to the agent". Reconciliation still keys on absence-of-success, not presence-of-error, because a call can also simply never be made. 3. The old `max: 80` did not cause runtime truncation — 50 calls never reached it. The real hazard is gh-aw's injected "Maximum 80 label(s) can be added" wording against a 100-label worst case, which invites agent self-truncation. 110 is justified by that hazard plus bounded margin, and the docblock now explicitly records what is NOT claimed. Also, per the PR #1965 review: Step 2e no longer demands a real issue number for items created during the run. Issue creation is deferred to the post-agent job, so it names the stable temporary ID plus title and intended labels, and quotes a real number only for a reused issue — avoiding reintroduction of #1962's invalid-number assumption. Tests: renamed the two overclaiming tests and gave each a negative guard that fails if the retracted wording returns; added coverage for the temporary-ID rule. Trimmed prose to stay under the existing 160 KB source-growth guard rather than raising another test's budget. Verified: 221/221 across the three gh-aw suites; mutation-checked (cap -> 80 fails 5 tests incl. the compiled-runtime one, reintroducing either retracted claim fails its guard); strict compile clean on all four workflows; npm run build passes. Closes #1961 Parent #1957 Working as FIDO (Quality Owner) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2471face-3b27-4419-bd48-0becd2642d7f * test(gh-aw): use explicit .js extension on helper import Addresses Copilot review on PR #1964. The reviewer is correct on both counts. `tsc --module nodenext --moduleResolution nodenext` reports exactly one error on this file, and TypeScript names the fix itself: test/gh-aw-activation-capacity.test.ts(94,46): error TS2835: Relative import paths need explicit file extensions in ECMAScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean './helpers/gh-aw-lock.js'? The repo is `"type": "module"` with `module`/`moduleResolution: NodeNext` in the root tsconfig, and the four other suites importing this helper already use the `.js` specifier. This file was the lone outlier; it now matches. Verified: 293/293 across all eight suites that import the helper, all four workflows compile with `--strict` (pre-existing warnings only), `npm run build` passes, and the NodeNext type-check on this file is now clean. Refs #1961. Parent #1957. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2471face-3b27-4419-bd48-0becd2642d7f * fix(workflows): state Step 2e counts as accepted operations, not applied labels Copilot review flagged that Step 2e's reconciliation claimed issues "received their labels". #1963 (now merged to dev) settled the vocabulary: an accepted safe output is queued this turn and applied by the post-agent job, so the agent has evidence only that a call was accepted for a specific target -- never that a label reached GitHub. Step 2e was the remaining place still asserting application, which conflicts with the accepted-vs-applied rule the same skill now states. - `labeled` is defined as issues whose add_labels call was accepted. - The report_incomplete reason reads "had a label operation accepted". - "proof that every label landed" -> "every label operation was accepted". - Added an explicit statement that the counts track label operations, not labels present on GitHub. Tests: new assertion guards the positive and both retired over-claim phrasings; verified by reverting the prose (1 failed, 28 passed) rather than assuming. Tightened the sibling assertion to match the exact count definitions -- the previous `.*`-joined form matched a distant "was accepted" elsewhere in the flattened skill and so failed to detect a reverted definition. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0515ad5d-4c5b-48bb-92a6-4e1296594cab * test(gh-aw): anchor the false-success assertion, which was vacuous Copilot review caught that the regex ended in an `|activated.*artifact` alternative. Alternation binds loosest, so the whole pattern collapsed to that branch and matched any mention of the artifact -- the test would have passed with the `labeled < activated` condition deleted from the prose, which is the entire invariant it names. Anchored to the full sentence and verified by mutation: removing the condition from Step 2e now fails (1 failed / 28 passed) where it previously passed. Second instance of this defect class in this suite; the sibling count assertions were tightened in 4ec89dd for the same reason. Audited the remaining alternations here -- `/finish green|still succeeds/` is legitimate (two accepted phrasings, both specific). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0515ad5d-4c5b-48bb-92a6-4e1296594cab --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2471face-3b27-4419-bd48-0becd2642d7f Copilot-Session: 0515ad5d-4c5b-48bb-92a6-4e1296594cab
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the
safe-outputssystem reliability in Plan Activate mode to address Brady's report of issues not being outputted from Actions.Root Cause Analysis
Multiple failure modes compound during Plan Activate:
max: 50limit — large plans could hit the cap silentlysquadlabel, potentially blockingcreate-issueChanges
maxfrom 50 → 75How safe-outputs permissions work
issues: readis correct. gh-aw safe-outputs bypass normal GITHUB_TOKEN permissions — writes are handled by the platform layer. Thecreate-issuesafe-output does NOT needissues: write.Full analysis
See
.squad/decisions/inbox/copilot-safe-outputs-reliability.md(gitignored, available locally).Ref: #1678