Harden CI-fixer discovery and safe-output transport

[PureWeen]

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description of Change

Hardens both scheduled CI-fixer workflows after two production failures following #36775:

• main run 30269817669 lost all 50 broad-search issue bodies to the integrity filter, so the agent had no usable fresh-candidate evidence.
• net11 run 30269934433 prepared a valid one-file update for PR [ci-fix-net11] Needs review: Fix CA1416 platform-compatibility build errors in Essentials (refs #36451) #36619, but capture-time validation compared its branch with main. The resulting 3,377-file stale-base divergence produced an oversized allowed-files request, killed the Safe Outputs backend, and still left a green run with empty output.

This change:

1. Builds a deterministic pre-agent snapshot containing only open issues with the caller's exact label (ci-scan or ci-scan-net11), optionally scoped to a dispatch issue number. Issue counts, titles, and bodies are bounded and sanitized; title/body content is explicitly untrusted inert data.
2. Removes broad live issue discovery from the agent tool surface and makes the bounded snapshot authoritative.
3. Pins the capture-time MCP gateway base before startup with the supported pre-agent-steps hook, while independently pinning apply-time handlers. The net11 workflow now validates against net11.0, not repository-default main.
4. Gates code transport on append-only ancestry, merge-free history, allowed paths, at most 3 commits, 20 files, and 256 KiB. Existing PR advances use the saved PR head so only the intended delta is transported.
5. Registers required safe outputs before emission and reconciles them with authoritative agent_output.json after the agent exits. Backend errors, malformed expectations, or missing required captures fail the agent job; a legitimate no-op has no mutation expectation and remains green.
6. Adds hermetic PowerShell and Vally coverage for both incidents while preserving the separate ownership gate.

Follow-up hardening (commit 1564b2b8b2)

A first adversarial review round found and fixed six defects in the work above:

1. Issue stranding. The candidate query fetched a single newest-first page bounded by MaxIssues (20). With 51 open ci-scan and 58 open ci-scan-net11 issues, the oldest issues could never enter any run's window. The query is now oldest-first (sort=created, direction=asc) and fully paginated, so bounding happens after the complete eligible set is known. Only the bounded batch is emitted, so the snapshot payload size is unchanged.
2. Silent truncation. The snapshot now reports totalMatched and truncated alongside count, so a bounded batch can never be read as "no other candidates exist".
3. Unenforceable twin ownership. The ownership rule was prose-only, but the agent can no longer read raw labels once the issues toolset is removed, so it was undecidable at runtime. Ownership is now decided deterministically in the prefetch and reported as excludedDualLabelled. (The first attempt applied exclusion symmetrically and introduced a regression — corrected in 5bb908cade below.)
4. Scoped-read hard failure. A 404 on a dispatch-scoped issue read aborted the entire prefetch. The scoped read now tolerates failure and skips, instead of taking down discovery for every other candidate.
5. dry_run false red. Preview runs prepare mutation expectations but intentionally emit nothing, so reconciliation failed a run that behaved correctly. Reconciliation is now skipped on dry_run=true via a host-evaluated Actions if: expression rather than a script switch, so the guard cannot be disabled by the agent.
6. Case-insensitive path matching. The transport allowlist used -match, so src/core/... passed a gate written for src/Core/.... It now uses -cmatch, matching Git's case-sensitive path semantics.

Follow-up regression fixes (commits 5bb908cade, 80fe851229)

A second adversarial review round audited the commit above and found two further defects — one introduced by the round-1 fix, one latent in the original work.

5bb908cade — dual-labelled issues were stranded by both twins.

Ownership is asymmetric: both prompts agree the net11 twin owns an issue carrying both labels (ci-status-fix.md skips them because "the net11.0 workflow owns those"; ci-status-fix-net11.md skips only issues carrying ci-scan but NOT ci-scan-net11). But fix #3 wired the exclusion symmetrically. Because net11's exact-label filter already drops ci-scan-only issues, the sole net effect of its exclusion was to drop the dual-labelled issues it owns — so such an issue was processed by neither twin and stranded permanently, reproducing the very bug class fix #1 closed. No dual-labelled issue exists today, so this was latent rather than live.

The exclusion is removed from the net11 twin only; main's is correct and necessary. The asymmetry is documented at both call sites. The original test only exercised the main configuration — and a unit test cannot observe a wiring mistake in the workflow source regardless — so both layers were added: a unit test asserting the net11 configuration retains dual-labelled issues, and a source-level parity guard asserting main excludes ci-scan-net11 while net11 passes no -ExcludeIssueLabel at all.

80fe851229 — the transport gate rejected every FRESH create-PR.

Test-CiFixTransport.ps1 always bound -PullRequestNumber when calling Register-CiFixSafeOutputExpectation.ps1, including on the create_pull_request path where no PR exists yet and the parameter sits at its unset default of 0. The registrar declares [ValidateRange(1, [int]::MaxValue)] on that parameter, so binding 0 fails during parameter binding — before the registrar's own "non-PR types don't need a number" branch (which lists create_pull_request as exactly such a type) can run.

Every FRESH transport therefore died with Cannot validate argument on parameter 'PullRequestNumber', even for a valid one-file, one-commit, append-only, in-allowlist diff. Both prompts invoke exactly that form, so the fixer could not open a new PR at all. Advancing an existing PR was unaffected, which is why the staged proofs — a no-op and an existing-PR advance — never surfaced it. -PullRequestNumber is now bound only when actually set.

This shipped undetected because no test ever exercised a successful create_pull_request; the only such test asserts a rejection, and it passed for an unrelated reason. A FRESH-path test now asserts the transport succeeds, reports a null pullRequestNumber, and registers a matching expectation.

Both regression fixes are mutation-verified: reverting either makes its new test fail.

Follow-up CI wiring (commit 9520f4407e)

A third review round found that none of the Pester coverage above was gated by CI: nothing under .github/workflows or eng/pipelines referenced Invoke-Pester or these test files, so a regression in the transport/expectation logic could only surface during a live scheduled Actions run — exactly how 80fe851229 shipped.

The earlier deferral reason no longer held. It claimed a blanket gate over .github/scripts/** would immediately fail on pre-existing failures in Fix-MilestoneDrift.Tests.ps1; re-measured at this head the full suite is 631/631 green, and stays green with gh stubbed to fail and both GH_TOKEN and GITHUB_TOKEN cleared. The suite is hermetic, so it is safe to gate.

.github/workflows/powershell-script-tests.yml runs the whole .github/scripts suite on any PR touching those paths:

• pull_request, not pull_request_target. The job executes PowerShell authored by the PR, so it must run with no base-repo secrets and a read-only token. permissions: contents: read, and checkout uses persist-credentials: false.
• Pester pinned to 5.9.0, so an upstream release cannot silently change discovery or assertion behavior.
• The run step deliberately does not Set-StrictMode. StrictMode set in the host session leaks into every test body Pester dot-sources and turns 16 otherwise-passing tests red for reasons unrelated to the code under test. This was caught while validating the gate.
• Fails on zero discovered tests, so the gate cannot pass vacuously if the path or filter ever breaks.

Mutation-verified: re-introducing the unconditional -PullRequestNumber binding fails accepts a FRESH create_pull_request transport that has no PR number yet, so the gate is load-bearing.

Follow-up security-review fixes (commit cb31b580af)

A 4-model adversarial security review (#36842 review by @kubaflo) ran against head 9520f4407e and produced four findings. Each was independently re-verified against that head before acting.

1. update_pull_request was the one mutating handler not scoped to this workflow's own PRs. Every other mutating handler is locked to required-title-prefix + required-labels; update_pull_request alone shipped target: "*" with allow_body: true and neither constraint, so a prompt-injected agent could replace the body of any PR in the repo, up to max: 3 per run. The tightened patch caps do not apply — this is not a patch operation.

The source carried a NOTE asserting that gh-aw v0.82.14 "silently drops required-*" for this output. That claim is wrong. Recompiling with the repository-pinned compiler emits both keys into GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG, and gh-aw additionally generates the enforcement text Only PRs with labels [agentic-workflows] can be updated. Only PRs with title prefix "[ci-fix] " ... into the agent constraint string. --strict rejects unknown safe-output fields (verified by compiling a deliberately bogus key), so these are schema-supported rather than ignored. Both constraints are now set on both twins, and the stale NOTE plus the matching Hard Rule 6 prompt text are corrected.

2. Reconciliation was one-directional and skippable. It only detected registered-but-not-captured, so extra captured items were ignored; worse, it returned exit 0 before checking anything when no expectation existed — precisely the run shape an out-of-band emitter produces. Reconciliation now also checks the reverse direction: for every mutating output type, captured must not exceed registered, and that check runs even when the expectation set is empty. Diagnostic types (missing_tool / missing_data / noop / report_incomplete) are excluded because they are emitted outside Hard Rule 11 and must not redden a legitimate run.

The expectation set is now resolved once with a plain find. The previous find ... -print -quit | grep -q . probe can surface a SIGPIPE (141) under pipefail, which would read as "no expectations" and skip reconciliation entirely — an exposure this change would otherwise have multiplied.

3. -AllowFailure made a transient outage look like "no ci-fix work". The scoped-dispatch and priority-watch issue reads collapsed every non-transient failure to an empty result, so a 401/403/429/5xx/network failure was indistinguishable from "this issue is not in scope". A new -AllowNotFound suppresses only a confirmed HTTP 404 and propagates everything else; both discovery reads now use it. Broad -AllowFailure is retained only where the caller tracks its own "known" flag and fails closed from it (Get-HeadCheckState, Get-PullRequestBody).

4. Mutable action reference. powershell-script-tests.yml used actions/upload-artifact@v4 while SHA-pinning everything else; pinned to 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1, the SHA already used ~15× in this repo and recorded in the gh-aw manifest.

Not actioned (with reasons). The reviewer's suggestion to gate the safe_outputs job on needs.agent.result == 'success' targets a gh-aw-generated job condition (if: (!cancelled()) && needs.agent.result != 'skipped' && ...) that exists only in the compiled lock; changing it would require hand-editing generated output, which this repo forbids. The reviewer's own analysis agrees the real boundary is the config-locked handler plus the read-only agent job, both unweakened here. The unconfirmed update_branch: false → item-level-true override remains unverified upstream behavior; the config sets false.

Locks were regenerated with the repository-pinned gh-aw v0.82.14; no .lock.yml was hand-edited, and the lock diff is confined to the metadata hashes, the three safe-output config copies, the generated constraint string, and the reconciliation step.

Security Model

• Issue titles and bodies are never executed or interpolated into commands.
• Evidence is capped at 20 issues, 256 title characters, and 12,000 body characters. Pagination widens only the in-memory eligibility scan, not the emitted payload.
• No direct or post-agent GitHub write path was added; all writes still use supported gh-aw Safe Outputs.
• Oversized, unrelated, stale-base, merge, or non-append-only transports fail before a mutation output is emitted.
• Scope of the 20-file gate. gh-aw v0.82.14 rejects max-patch-files on push-to-pull-request-branch and does not propagate it from top-level safe-outputs, so the file-count bound is enforced only by Test-CiFixTransport.ps1 before emission. Treat it as defense-in-depth, not as a privileged push-handler boundary. The handler's own max-patch-size cap still applies and measures the increment against origin/<branch>.
• Scope of the expectation/reconciliation mechanism. The agent can write to /tmp/gh-aw/, so expectation files are cooperative-agent diagnostics that catch backend failures and dropped captures — not a boundary against a hostile agent. The privileged boundary remains gh-aw's own safe-output handlers.
• Scope of dry_run. dry_run=true is an in-prompt preview instruction, not gh-aw staged: mode, so it is not a framework-level write barrier. For a preview that blocks GitHub API calls at the framework level, use gh aw trial, as the workflow notes direct.

Validation

• PowerShell/Pester: 657/657 passed for the full .github/scripts suite now gated in CI (also green with gh stubbed to fail and GH_TOKEN/GITHUB_TOKEN cleared, confirming hermeticity). Coverage includes the exact 3,377-file stale-base fixture and regression tests for pagination/ordering, truncation totals, both directions of twin-label ownership, twin wiring parity, scoped-404 skip, case-sensitive path rejection, and the FRESH create-PR transport. The cb31b580af round adds coverage that asserts the compiled lock scopes every mutating handler to this workflow's own PRs, and that executes the compiled post-steps: reconciliation shell against fixtures — closing the previously-listed gap that this bash had no unit coverage.
• gh-aw v0.82.14 strict compile/validation: 2 workflows, 0 errors, 0 warnings; both .lock.yml files regenerate byte-identically from source (re-verified after cb31b580af).
• git diff --check: clean.
• Strict Vally lint: capability and ownership specs valid.
• Vally capability eval: 98.3% across 30 trials; all four incident-focused scenarios passed 12/12 trials.
• Vally ownership gate: 100% across 12 trials.
• ⚠️ The Vally figures above were measured on the first two commits and have not been re-run against the four follow-up commits. Treat them as evidence for the original change only. Review also found the four new stimuli are satisfiable by a content-free response, because each stimulus scores a weighted mean across graders and the two mechanical graders can outvote a failing rubric grader — see "Known gaps" below.
• Independent review: three multi-reviewer adversarial rounds. Round 1 produced the six findings fixed in 1564b2b8b2; round 2 audited that commit and produced the two regressions fixed in 5bb908cade and 80fe851229; round 3 produced the missing CI gate wired in 9520f4407e; a fourth, security-focused 4-model round produced the four findings addressed in cb31b580af. This supersedes the earlier "no defects found" claim, which reflected a narrower review pass.
• Pipeline security grep checks: no changed-file violations.
• Poutine: no findings in the changed workflows; reported only unrelated repository baseline findings.
• Zizmor: no warning/error-severity findings; 8 low-confidence informational findings in v0.82.14-generated MCP heredocs.
• Actionlint: only the four known v0.82.14 generated-expression schema mismatches (secret_verification_result and github.aw.import-inputs.random_seed, once per workflow).

Known gaps (not addressed here)

• Vally grader strictness. Raising per-stimulus thresholds (or making the rubric grader independently gating) requires re-running the live evals to confirm legitimate responses still pass; since skill-validation.yml gates PRs on these evals, that is deliberately left to a follow-up rather than tuned blind.

Staged Fork Proof

The fork-only workflow is guarded to PureWeen/maui, requires dry_run=true, uses global safe-outputs.staged: true, and performs no real writes.

Scenario	Run	Result
Main exact-label evidence survives 50/50 live-search filtering and emits a legitimate no-op	30295671565	Green
Net11 saved-head one-file delta captures and previews both PR push and body update	30294895100	Green
Exact wrong-base fixture exposes 3,378 changed files and deliberately ends non-green	30295671617	Expected failure

The successful net11 proof shows DEFAULT_BRANCH=net11.0 in the capture-time MCP gateway and apply-time handler, previews only src/Essentials/test/UnitTests/ForkValidationTransport.txt, and leaves fork PR #169's head, body, labels, draft state, and updated timestamp unchanged.

These staged runs predate the four follow-up commits and were not re-run against them. Note that none of them exercised a FRESH create-PR, which is why the 80fe851229 defect survived them; that path is now covered by the Pester suite.

gh-aw v0.83.1 / #36772

#36772 is a broader fleet upgrade and is intentionally not bundled here. Both v0.82.14 and v0.83.1 schemas reject push-to-pull-request-branch.base-branch, even though runtime code recognizes that field. This PR instead uses the supported pre-agent environment hook plus existing Safe Outputs configuration, so the focused production fix does not depend on that upgrade.

What NOT to Do

• Do not restore broad live issue search; the bounded exact-label snapshot is the authoritative candidate source.
• Do not compare a net11 PR branch with repository-default main.
• Do not trim an oversized/unrelated diff to make it pass the transport gate.
• Do not convert a missing required safe output or backend error into noop or a direct write.
• Do not read a bounded snapshot as an exhaustive one; check truncated/totalMatched.
• Do not make twin label exclusion symmetric. Main excludes ci-scan-net11; net11 excludes nothing. Symmetry strands every dual-labelled issue.
• Do not bind -PullRequestNumber on the create_pull_request path; there is no PR yet and the registrar validates the range.
• Do not hand-edit either .lock.yml; regenerate both from source with the pinned gh-aw v0.82.14.
• Do not add Set-StrictMode to the Pester gate's run step; it leaks into dot-sourced test bodies and fails 16 unrelated tests.
• Do not switch the Pester gate to pull_request_target; it executes PR-authored PowerShell and must stay unprivileged.

Issues Fixed

Follow-up to #36775.

Follow-up: dry-run seam (commit 7bec5da438)

A fourth adversarial review round questioned whether the reconciliation added in item 5 above is fail-closed at the mutation boundary. Split in two, because only one half is author-controllable.

The dry_run skip was a real seam — and item 5's fix is superseded.

Item 5 resolved the false red by skipping reconciliation entirely on dry_run=true (if: always() && github.event.inputs.dry_run != 'true'). That went too far: the privileged safe_outputs job has no dry-run predicate of its own, so a mutating output emitted against the dry-run contract was both applied and completely unreported. It is the one run shape where the check went from detective to absent.

The step now runs always() and inverts under dry_run rather than disappearing. A dry run promises to emit nothing, so its mutating allowance is zero no matter how many expectations were registered while validating the candidate diff; only the forward registered-but-not-captured direction — which a correct dry run legitimately trips, and which was the original reason for the skip — stays suspended. The dry-run signal is read from a step env: populated by Actions (CI_FIX_DRY_RUN: ${{ github.event.inputs.dry_run }}), so item 5's actual guarantee (host-evaluated, not agent-disableable) is preserved.

Gating safe_outputs on needs.agent.result == 'success' is not expressible from source.

That job condition is compiler-generated by gh-aw:

# ci-status-fix.lock.yml
if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success'

Pinned gh-aw v0.82.14 exposes no frontmatter knob over it, and the only alternative is hand-editing the lock — which this repo forbids and which would silently vanish on the next gh aw compile. The reconciliation step is therefore deliberately detective with respect to agent-job failure, and now says so inline, alongside the real preventive boundary: the SHA-pinned, config-locked handler that scopes every mutating output to this workflow's own [ci-fix]-prefixed, agentic-workflows-labelled PRs within per-run caps.

Validation. Full .github/scripts suite 665/665 (657 + 8 new). The new tests execute the compiled lock script under dry_run for both twins — asserting the .md would not prove the generated lock equivalent — reusing the existing Invoke-Reconcile harness with a new -DryRun switch. Mutation-verified in both directions: restoring the registration-count allowance under dry_run fails exactly the 2 emit-under-dry-run tests; removing the forward-check suspension fails exactly the 2 correct-dry-run tests. Locks regenerated with pinned gh-aw v0.82.14 and byte-stable on recompile; gh aw validate 0 errors; git diff --check clean.

Review follow-up (commit 0ed9cd964f)

A later round noted that report_incomplete is excluded from the reconciliation, so a dry_run could still create that one diagnostic issue. Accurate — and broader than raised, since noop is configured report-as-issue: true and files an issue too. It stays that way deliberately, and the code previously justified the carve-out only for legitimate runs while saying nothing about dry_run.

Suppressing them would be a regression, not a fix. This step is detective, so listing those types could not prevent the write — only redden the run after the issue was already filed. And report_incomplete is emitted from the snapshot guard, which runs before any Step 0 dry-run gate: it is how a preview reports that it could not proceed. A write-free canary that cannot report its own blocker is strictly worse than one that files a diagnostic, and a dry run is precisely when a broken snapshot most needs to be heard. The contract enforced here is "emit no MUTATION", not "emit no telemetry".

That reasoning now lives in the reconciliation script itself in both twins, and the existing dry-run diagnostics case was extended to cover the exact type raised (create_report_incomplete_issue, alongside noop). Mutation-verified — adding those types to the zero-allowance loop fails the case on both twins, so a future "fix" for this note trips a test that explains why it is wrong. No behaviour change; suite still 665/665, gh aw compile --strict 0 errors and idempotent.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 36842

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 36842"

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Hardens the CI-fixer (main + net11.0) gh-aw workflows by making issue discovery deterministic/bounded, pinning the correct base branch for safe-output validation, and enforcing fail-closed safe-output transport + capture verification.

Changes:

• Replace live/broad issue discovery with a bounded pre-agent snapshot of exact-label open issues plus open CI-fix PR watch state.
• Pin DEFAULT_BRANCH for capture-time and apply-time safe-output handlers (including correct net11.0 base), and tighten safe-output transport limits.
• Add PowerShell/Pester + Vally coverage for the two reported production incidents (integrity filtering + stale-base oversized transport).

Show a summary per file

File	Description
.github/workflows/ci-status-fix.md	Main workflow: bounded authoritative prefetch, remove issues/search toolsets, pin capture/apply base, add post-step to fail on missing safe-output captures, tighten patch size/files.
.github/workflows/ci-status-fix.lock.yml	Regenerated lock reflecting the new toolsets, DEFAULT_BRANCH pinning, patch limits, and post-step validation.
.github/workflows/ci-status-fix-net11.md	Net11 workflow: same hardening as main, but pins capture/apply base to net11.0 to avoid main-vs-net11 divergence.
.github/workflows/ci-status-fix-net11.lock.yml	Regenerated lock reflecting net11 base pinning, toolset changes, patch limits, and post-step validation.
.github/skills/ci-fix/tests/eval.vally.yaml	Adds incident-focused evaluation scenarios ensuring authoritative prefetch usage and fail-closed emission behavior.
.github/skills/ci-fix/SKILL.md	Updates CI-fix skill rules to fail closed at the write boundary and handle attempt caps explicitly.
.github/scripts/Test-CiFixTransport.ps1	New transport gate script enforcing append-only ancestry, bounded commits/files/bytes, allowed paths, and expectation registration.
.github/scripts/Test-CiFixTransport.Tests.ps1	Pester coverage for transport gating, including the stale-base 3,377-file divergence fixture and base pin checks.
.github/scripts/Register-CiFixSafeOutputExpectation.ps1	New helper to register expected safe outputs for post-step reconciliation.
.github/scripts/Register-CiFixSafeOutputExpectation.Tests.ps1	Pester coverage for expectation registration rules (PR-targeted vs non-targeted outputs).
.github/scripts/Query-CiFixPRs.ps1	Extends pre-agent prefetch to include bounded exact-label issue evidence (sanitized/untrusted) alongside open PR watch state.
.github/scripts/Query-CiFixPRs.Tests.ps1	Pester coverage for bounded issue evidence selection, scoping, sanitization, and ordering rules.

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 1

[PureWeen]

Superseded — this was already addressed in 1564b2b8b2, before this thread's commit was superseded, and the rationale is now inline at Test-CiFixTransport.ps1:51-53.

The divergence is deliberate and fails closed, not open: .+/ is stricter than the handler's **/PublicAPI.Unshipped.txt, so the gate can only reject something the handler would have accepted — it can never admit something the handler would reject. A stricter local gate produces a clear, local transport rejection instead of a confusing "registered an expectation but captured 0 outputs" failure after the privileged handler declines it.

Verified at head 9520f4407e:

$ ls PublicAPI.Unshipped.txt
ls: PublicAPI.Unshipped.txt: No such file or directory
$ git ls-files '*PublicAPI.Unshipped.txt' | wc -l
      71
$ git ls-files '*PublicAPI.Unshipped.txt' | grep -v '^src/'
(no output)

All 71 live under src/**, and no root-level file exists, so there is no reachable case where this gate fails closed on an otherwise-allowed CI-fix change. Leaving the stricter form as-is.

[github-actions]

Skill Validation Results

@PureWeen — new skill validation results are available based on this last commit: 0ed9cd9.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

✅ Skill Validation Results — 0ed9cd9 · Harden CI-fixer discovery and safe-output transport · 2026-07-28T17:45:26Z

✅ Static Checks Passed

Skills: 24 | Eval specs linted: 14

Full lint output

── .github/skills/agentic-labeler/tests/eval.vally.yaml
npm warn deprecated prebuild-install@7.1.3: No longer maintained. Please contact the author of the relevant native addon; alternatives are available.
✔ .github/skills/agentic-labeler/tests/eval.vally.yaml is valid
── .github/skills/analyze-sessions/tests/eval.vally.yaml
✔ .github/skills/analyze-sessions/tests/eval.vally.yaml is valid
── .github/skills/ci-fix/tests/eval.ownership.vally.yaml
✔ .github/skills/ci-fix/tests/eval.ownership.vally.yaml is valid
── .github/skills/ci-fix/tests/eval.vally.yaml
✔ .github/skills/ci-fix/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/eval.capability.vally.yaml
✔ .github/skills/code-review/tests/eval.capability.vally.yaml is valid
── .github/skills/code-review/tests/eval.inline-findings.vally.yaml
✔ .github/skills/code-review/tests/eval.inline-findings.vally.yaml is valid
── .github/skills/code-review/tests/eval.producer-trace.vally.yaml
✔ .github/skills/code-review/tests/eval.producer-trace.vally.yaml is valid
── .github/skills/code-review/tests/eval.trim-aot.vally.yaml
✔ .github/skills/code-review/tests/eval.trim-aot.vally.yaml is valid
── .github/skills/code-review/tests/eval.vally.yaml
✔ .github/skills/code-review/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/hermeticity.vally.yaml
✔ .github/skills/code-review/tests/hermeticity.vally.yaml is valid
── .github/skills/evaluate-pr-tests/tests/eval.vally.yaml
✔ .github/skills/evaluate-pr-tests/tests/eval.vally.yaml is valid
── .github/skills/pr-review/tests/eval.gh-auth.vally.yaml
✔ .github/skills/pr-review/tests/eval.gh-auth.vally.yaml is valid
── .github/skills/try-fix/tests/eval.vally.yaml
✔ .github/skills/try-fix/tests/eval.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml is valid

✅ LLM Evaluation Passed

2/2 eval suite(s) met threshold

Suite	Before	After	Threshold	Verdict
ci-fix-ownership-capabilities	1.00	1.00	1.00	✅
ci-fix-capabilities	0.80	0.84 📈	0.60	✅

Before = these specs run against the skill on the PR base (the pre-change reviewer); After = with this PR. A rise (📈) means the change made the reviewer catch a regression it previously missed. The Before run is informational and never gates.

Harness hermeticity (negative control)

✅ Hermetic — the negative-control stimulus correctly came back unauthenticated (anonymous core rate limit; no GitHub token leaked into the agent env).

📊 ci-fix — eval report

Eval Results

Timestamp: 2026-07-28T17:45:02.742Z

ci-fix-ownership-capabilities [claude-opus-4.6] (/home/runner/work/maui/maui/.github/skills/ci-fix/tests/eval.ownership.vally.yaml)

Must-pass ownership scenarios for the CI-fix triage skill. They distinguish explicit human ownership declarations from incidental issue references.

Stimulus	Skills	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
comment-only-reference-does-not-stop-triage	ci-fix (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3	3/3	100.0%	100.0%	13.4s	43,811	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
explicit-human-reference-stops-competing-fix	ci-fix (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3	3/3	100.0%	100.0%	10.9s	43,588	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
explicit-human-url-reference-stops-competing-fix	ci-fix (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3	3/3	100.0%	100.0%	9.3s	43,606	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
incidental-body-reference-does-not-stop-triage	ci-fix (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3	3/3	100.0%	100.0%	12.1s	43,861	2	1 calls (median) total across 3 trials: skill: 3</details>	✅

Model: claude-opus-4.6 | Judge: claude-opus-4.6 | Executor: copilot-sdk

---

ci-fix-capabilities [claude-opus-4.6] (/home/runner/work/maui/maui/.github/skills/ci-fix/tests/eval.vally.yaml)

General capability suite for the CI-fix triage skill. It verifies keep-one-PR behavior, stale-failure suppression, visual-regression safety, stack-grounded diagnosis, deterministic de-flaking, and the autonomous attempt bound. Ownership decisions are gated separately by eval.ownership.vally.yaml.

Stimulus	Skills	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
current-stack-evidence-prevents-stale-adjacent-fix	ci-fix (3×)	✅ output-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	25.8s	43,981	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
deterministic-deflake-never-mutes-or-retries	ci-fix (3×)	✅ output-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	26.0s	43,919	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
effective-attempt-cap-defers-instead-of-replacing-pr	ci-fix (2×)	❌ output-contains 2/3 ✅ prompt 3/3	2/3	100.0%	29.6%	27.3s	44,147	2	1 calls (median) total across 3 trials: skill: 2</details>	🟡 1
existing-ci-fix-pr-enters-watch-mode	—	❌ output-contains 0/3 ✅ prompt 3/3	0/3	0.0%	0.0%	21.3s	21,428	1	0	❌
existing-watch-pr-advances-only-itself	ci-fix (3×)	✅ output-contains 3/3 ✅ output-matches 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	23.6s	43,902	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
merged-fix-suppresses-stale-reopen	ci-fix (3×)	✅ output-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	19.8s	43,698	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
oversized-unrelated-diff-is-never-transported	ci-fix (3×)	❌ output-contains 1/3 ❌ output-matches 1/3 ✅ prompt 3/3	1/3	100.0%	3.7%	27.4s	43,954	2	1 calls (median) total across 3 trials: skill: 3</details>	🟡 2
prefetched-evidence-survives-live-integrity-filtering	ci-fix (3×)	✅ output-not-matches 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	27.0s	43,910	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
safe-output-backend-failure-is-incomplete	ci-fix (3×)	❌ output-contains 1/3 ✅ output-matches 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	22.3s	43,857	2	1 calls (median) total across 3 trials: skill: 3</details>	✅ 3
visual-regression-never-modifies-baseline	ci-fix	❌ output-contains 2/3 ✅ prompt 3/3	2/3	100.0%	29.6%	22.4s	21,746	1	0	🟡 4

Model: claude-opus-4.6 | Judge: claude-opus-4.6 | Executor: copilot-sdk

🔍 Full results and investigation steps

Footnotes

1. ⚠️ Flaky (33% minority outcome). Grader breakdown: output-contains passed 2/3 trials ↩

2. ⚠️ Flaky (33% minority outcome). Grader breakdown: output-contains passed 1/3 trials, output-matches passed 1/3 trials ↩

3. Grader breakdown: output-contains passed 1/3 trials ↩

4. ⚠️ Flaky (33% minority outcome). Grader breakdown: output-contains passed 2/3 trials ↩

[Copilot]

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 0 new

[PureWeen]

@kubaflo — thanks, this was a high-quality review. I re-verified all four findings against 9520f4407e before acting; three are accepted and fixed in cb31b580af, one is not actionable from source. Details:

1. update_pull_request not [ci-fix]-scoped — accepted and fixed

You were right, and the in-repo NOTE claiming gh-aw v0.82.14 "silently drops required-*" for this output was wrong. I tested it rather than trusting the comment: adding required-title-prefix + required-labels and recompiling with the pinned v0.82.14 emits both into the handler config:

"update_pull_request":{"allow_body":true,"allow_title":false,"max":3,
  "required_labels":["agentic-workflows"],"required_title_prefix":"[ci-fix] ",
  "target":"*","update_branch":false}

gh-aw also generates the enforcement text into the agent constraint string: Only PRs with labels [agentic-workflows] can be updated. Only PRs with title prefix "[ci-fix] " .... And --strict does reject unknown safe-output keys (verified by compiling a deliberate bogus-unknown-field), so these are schema-supported, not silently ignored. Applied to both twins ([ci-fix] / [ci-fix-net11] ); the stale NOTE and the matching Hard Rule 6 prompt text are corrected.

One thing your review surfaced that I'd have missed: push_to_pull_request_branch compiles required-title-prefix to title_prefix, not required_title_prefix. The new test accepts either compiled spelling so it can't be fooled by that naming difference.

2. Reconciliation is best-effort — partially accepted, core gap fixed

The safe_outputs if: you pointed at is gh-aw-generated (lock line ~1885), so gating it on needs.agent.result == 'success' would mean hand-editing generated output, which this repo forbids. I agree with your down-rating: the config-locked handler plus the read-only agent job is the real boundary and neither is weakened here.

But your "should reconcile in both directions" point was a genuine hole, and worse than the write-up implies — the check returned exit 0 before evaluating anything when no expectation existed, which is exactly the run shape an out-of-band emitter produces. Demonstrated against the pre-fix shell:

Scenario	pre-fix	now
update_pull_request captured, zero expectations registered	rc=0 (silently green)	rc=1
2 add_comment captured, 1 registered	rc=0 (extra ignored)	rc=1
registered but not captured	rc=1	rc=1 (unchanged)
diagnostics-only (missing_tool/missing_data/noop)	rc=0	rc=0 (no false red)

Reverse direction now applies to the six mutating types and runs even with an empty expectation set. Diagnostics are excluded because they're emitted outside Hard Rule 11.

While wiring this I also removed the find ... -print -quit | grep -q . probe: under pipefail it can surface SIGPIPE (141), which reads as "no expectations" and skips reconciliation. It's now a single plain find with no pipeline.

3. -AllowFailure treats transient failures as authoritative-empty — accepted and fixed

Added -AllowNotFound, which suppresses only a confirmed HTTP 404 and propagates auth/rate-limit/5xx/network. Both discovery reads (scoped-dispatch and priority-watch) use it. Broad -AllowFailure is retained only where the caller keeps its own "known" flag and fails closed from it (Get-HeadCheckState, Get-PullRequestBody). Matching on the HTTP code, not gh's prose, so a body containing "Not Found" can't fake it.

4. Mutable actions/upload-artifact@v4 — accepted and fixed

Pinned to 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1, matching the ~15 other uses in this repo and the gh-aw manifest.

Unverified item

I couldn't confirm the update_branch: false → item-level-true override either; the config sets false and the merge logic isn't in the compiled lock. Leaving it as an open upstream question rather than speculating.

Validation

• Full .github/scripts Pester suite: 657/657 (was 631; +26).
• New coverage asserts the compiled lock scopes every mutating handler, and executes the compiled post-steps: reconciliation shell against fixtures — which also closes the "reconciliation bash has no unit coverage" item previously listed under Known gaps.
• Mutation-verified, reverting each fix fails only its own tests: 3 fail for the -AllowNotFound call sites, 8 for the 404 classification, 7 for the lock scoping + reconciliation.
• Locks regenerated with the pinned gh-aw v0.82.14 (--strict, 0 errors / 0 warnings), idempotent on re-run; no .lock.yml hand edits. Lock diff is confined to metadata hashes, the three safe-output config copies, the generated constraint string, and the reconciliation step.
• git diff --check: clean.

Your framing correction on the two models that rated the reconciliation fail-open as HIGH privilege-escalation was right, and it's why this landed as a correctness/observability fix rather than a security rewrite.

[Copilot]

Copilot's findings

• Files reviewed: 13/13 changed files
• Comments generated: 0 new

[kubaflo]

AI-assisted adversarial security review — round 2, 4-model ensemble (Claude Opus 4.8 / GPT-5.5 / Gemini 3.1 Pro / GPT-5.6 Sol), independently cross-verified. Head cb31b580afd (was 9520f4407ec at round 1). I verified the fixes down into the code and the compiled handler.

Excellent round-1 response — every actionable item is fixed. Verified:

• ✅ update_pull_request now [ci-fix]-scoped (my priority item): the config gained required_labels:["agentic-workflows"] + required_title_prefix:"[ci-fix] " (ci-status-fix.lock.yml:1990, md :296-319), and Opus/Sol confirmed the SHA-pinned gh-aw handler enforces it at runtime — so the "rewrite any PR body" exposure is closed, and the update_branch item-override is now [ci-fix]-bounded too.
• ✅ -AllowFailure → -AllowNotFound: Test-IsGhNotFoundFailure matches the structured \bHTTP 404\b token (so a body containing "Not Found" can't spoof it) and the transient set @(429,500,502,503,504) + auth/network now propagate; only a confirmed 404 degrades to null. (Query-CiFixPRs.ps1:116-124, 9/9 targeted tests.)
• ✅ upload-artifact SHA-pinned (powershell-script-tests.yml:89 → @043fb46d…# v7.0.1).
• ✅ Reconciliation hardened: you added the reverse-direction check I'd hoped for — "captured > registered is always a defect and must fail closed" — so an unregistered mutating output now fails the reconciliation step.

gh aw validate --strict passes, lock↔md in sync, 631+/… Pester green, no new permission/trigger/egress, no dedup/pagination regression. Nicely done.

⚠️ MEDIUM (defense-in-depth, for your call) — the reconciliation and dry-run don't fail-closed at the mutation boundary

Three of the four models flagged this and I confirmed it: the reconciliation now fails the agent job on an anomaly, but the privileged safe_outputs job still runs, because its gate is if: … needs.agent.result != 'skipped' (ci-status-fix.lock.yml:1885; net11 :1895) — i.e. it runs even when the agent job failed — and the agent artifact (if: always(), includes agent_output.json, :1222/:1235) is still downloaded and processed (:1980). Sol confirmed this on a real run (agent=failure, detection=success, safe_outputs=success). Likewise dry_run:true skips reconciliation (:1211-1213) but safe_outputs has no dry-run predicate, so a mutating output emitted against the dry-run contract would still apply.

Important calibration: the blast radius is bounded — the agent job is read-only (contents/issues/pull-requests: read), and the real preventive boundary is the SHA-pinned, config-locked handler, which this PR tightened and which now [ci-fix]-scopes every mutating handler. So an out-of-band output can only touch this workflow's own [ci-fix]-labelled PRs within the caps — this is not a privilege escalation, which is why I've kept it MEDIUM and one model's HIGH framing overstates it. But since this PR's stated purpose is safe-output transport integrity, whether the reconciliation should be preventive (fail-closed at the write boundary) rather than detective (flag-and-continue) is a real design call, and dry-run arguably shouldn't be able to mutate at all. Two concrete options: gate safe_outputs on needs.agent.result == 'success' (and add inputs.dry_run != 'true'), or move reconciliation into a required job the privileged job depends on. If you intend it as detective-only (the config-locked handler being the boundary), a one-line comment saying so would settle it.

CI

license/cla pass; maui-pr skipping (workflow/scripts only).

Verdict: NEEDS_DISCUSSION (not blocking)

Confidence: medium-high. From my analysis this is LGTM-modulo-one-design-call: all four round-1 items are genuinely fixed, the change net-improves the trust boundary, and the residual reconciliation/dry-run non-gating is bounded to [ci-fix] PRs by the (now-tightened) config-locked handler. I'm flagging it NEEDS_DISCUSSION rather than approving only because three models consistently want that control to fail-closed and there's a real dry-run seam — both your call to fix here or fast-follow. No changes are required to land this safely.

[PureWeen]

Thanks — worked the MEDIUM. Splitting it in two, because only one half is author-controllable.

Half 1 — the dry_run seam: accepted and fixed in 7bec5da.

You were right that this was a real gap, and it's sharper than the write-up suggests. The post-step carried if: always() && github.event.inputs.dry_run != 'true', so on a dry run the reconciliation didn't merely weaken — it never ran at all. Meanwhile safe_outputs has no dry-run predicate of its own, so a mutating output emitted against the dry-run contract was both applied and completely unreported. That's the one run shape where "detective" degraded to "absent".

The step now runs always() and inverts under dry_run instead of disappearing. A dry run promises to emit nothing, so its mutating allowance is zero no matter how many expectations were registered while validating the candidate diff:

# A dry_run promises to emit nothing, so its allowance is zero no matter how
# many expectations were registered while validating the candidate diff.
if [ -n "${dry_run}" ]; then
  registered=0
else
  registered="$(count_registered "${mutating_type}")"
fi

Only the forward registered-but-not-captured direction stays suspended, since a correct dry run legitimately trips it (that's why the step was skipped in the first place). The dry-run signal is read from a step env: populated by Actions (CI_FIX_DRY_RUN: ${{ github.event.inputs.dry_run }}), not from anything the agent can reach.

Half 2 — gating safe_outputs on needs.agent.result == 'success': not expressible, and documented as a deliberate design call.

Your option (a) is the right instinct but it isn't reachable from the source. That gate is compiler-generated by gh-aw:

# ci-status-fix.lock.yml:1885
if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success'

I checked v0.82.14 (the pinned compiler) for a frontmatter knob over the safe_outputs job condition and there isn't one. The only way to change it is to hand-edit the lock, which this repo forbids — the lock is regenerated on every compile and the edit would silently vanish on the next gh aw compile.

So I took the alternative you explicitly offered, and the step now carries the rationale inline rather than leaving a future reader to rediscover it:

DETECTIVE, NOT PREVENTIVE — and deliberately so. This step runs inside the agent job, whose failure does NOT skip the privileged safe_outputs job: gh-aw generates that gate as needs.agent.result != 'skipped' and exposes no source-level knob for it, and the lock is compiler-generated and must never be hand-edited. The actual preventive boundary is the SHA-pinned, config-locked gh-aw handler, which this workflow constrains so that EVERY mutating output is bounded to this workflow's own [ci-fix]-prefixed, agentic-workflows-labelled PRs within per-run caps.

That matches your own blast-radius calibration, and it's the boundary this PR tightened.

Validation

• 665/665 Pester across .github/scripts (657 before, +8 new).
• The 8 new tests execute the compiled lock script under dry_run — not the .md — for both ci-status-fix and ci-status-fix-net11, reusing the existing Invoke-Reconcile harness with a new -DryRun switch. Asserting the source wouldn't prove the generated lock is equivalent.
• Mutation-verified, both directions:
  • Restoring the registration-count allowance under dry_run → fails exactly the 2 dry_run emits a registered <Type> tests.
  • Removing the forward-check suspension → fails exactly the 2 dry_run ... emits nothing tests.
• Pre-fix behaviour reproduced directly: with the old script body, a dry run that registered a push_to_pull_request_branch and then emitted one exited 0. It now exits 1 — and in reality the old step didn't even execute.
• Locks regenerated with the pinned gh-aw v0.82.14 and byte-stable on recompile; gh aw validate → 0 errors; git diff --check clean.

[Copilot]

Copilot's findings

• Files reviewed: 13/13 changed files
• Comments generated: 0 new

[kubaflo]

AI-assisted adversarial security review — round 3, 4-model ensemble (Claude Opus 4.8 / GPT-5.5 / Gemini 3.1 Pro / GPT-5.6 Sol), independently cross-verified. Head 7bec5da4386.

This is essentially LGTM from my analysis — every actionable round-1/round-2 item is fixed, and I found no exploitable privilege boundary. Verified across all four models:

• ✅ update_pull_request remains [ci-fix]/agentic-workflows-scoped (config + handler-enforced)
• ✅ -AllowNotFound suppresses only a confirmed HTTP 404; 401/403/429/5xx/network propagate
• ✅ upload-artifact SHA-pinned (043fb46d… = v7.0.1)
• ✅ reverse reconciliation intact (captured > registered fails closed)
• ✅ dry-run detection added (CI_FIX_DRY_RUN from github.event.inputs.dry_run, agent cannot forge; forces the allowance to zero so any captured mutating output fails reconciliation)
• ✅ no lock/source drift; permissions/triggers/egress unchanged; 665 script tests pass; gh aw compile --strict clean

🟡 One design call for you (documented, bounded — not blocking)

The reconciliation and the new dry-run guard remain detective, not preventive: safe_outputs is gated if: … needs.agent.result != 'skipped', so it still runs (and applies captured outputs) even when the agent job fails reconciliation — the run reddens alongside/after the write rather than blocking it. Same for dry-run: a mutating output emitted against the emit-nothing contract is detected (run goes red) but still applied.

I want to be precise about severity, because the models split (Opus & Sol: LOW/READY; GPT: MEDIUM): this is not a privilege escalation. The agent job is read-only, and the real preventive boundary is the SHA-pinned, config-locked safe_outputs handler — every mutating output is capped and scoped to this workflow's own [ci-fix] PRs regardless of reconciliation. So the blast radius of a "detected-but-applied" output is bounded to your own ci-fix PRs within the caps. You've also now documented this as a deliberate detective design (md DETECTIVE, NOT PREVENTIVE), which was exactly my round-2 ask.

So the only open question is a genuine design judgment: is detective reconciliation sufficient, or do you want it preventive? If preventive: gate safe_outputs on needs.agent.result == 'success' (+ an inputs.dry_run != 'true' predicate), or move reconciliation into a required job the privileged job needs. If detective-by-design is intended (which your comments now state), it's shippable as-is.

(Minor, non-blocking: report_incomplete — a diagnostic tracking-issue output — is deliberately excluded from the reconciliation, so a dry-run could still create that one diagnostic issue. Low impact; it's the ci-fix system's own reporting, not an agent code/PR mutation.)

CI

license/cla pass; maui-pr skipping (workflow/scripts only).

Verdict: NEEDS_DISCUSSION — but read as LGTM-modulo-one-design-call

Confidence: medium-high. All round-1/2 findings are genuinely fixed and there's no exploitable boundary — from my end this is good to merge. I'm marking it NEEDS_DISCUSSION only because the detective-vs-preventive reconciliation is a real (now-documented, bounded) design choice that's yours to sign off on, not mine to unilaterally approve. If you're comfortable with detective-by-design, ship it.

[PureWeen]

Thanks — and thanks for being explicit about the split between the models rather than flattening it. I re-verified both items against 7bec5da4386 before touching anything. The design call I'm keeping as-is (detective); the minor note is real, is intended, and is now pinned by a test. Head is 0ed9cd964f.

🟡 The design call: detective, and staying that way

Accepted as an accurate description, declined as a change — and I want to be clear that the second half is not a preference, it's a constraint.

Your suggested preventive fix is to gate safe_outputs on needs.agent.result == 'success' plus an inputs.dry_run != 'true' predicate. I can't express either from source. That gate is compiler-generated:

# ci-status-fix.lock.yml:1887
if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success'

gh-aw exposes no frontmatter knob for that condition, and the lock is generated — hand-editing it is prohibited by ci-copilot-pipeline-security.instructions.md rule 8 and would be silently reverted by the next gh aw compile. So "make it preventive" is an upstream gh-aw change, not a change available to this PR.

That's exactly why the rationale is written into the .md as DETECTIVE, NOT PREVENTIVE rather than left as an unexplained gap, and your framing of the real boundary matches mine: the SHA-pinned, config-locked handler is what actually bounds this, and every mutating output is capped and scoped to this workflow's own [ci-fix]/agentic-workflows PRs regardless of what reconciliation concludes. Signing off on detective-by-design.

🟢 The minor note: report_incomplete under dry-run — real, and correct

This one I did chase, because your read of the mechanism is right and it's actually broader than you wrote. It isn't only report_incomplete:

"noop":{"max":1,"report-as-issue":"true"}

So noop files an issue too. Neither is in the zero-allowance loop, so under dry_run both can create a GitHub issue undetected.

But suppressing them would be a bug, not a fix, and that's why I didn't take the obvious path:

1. This step is detective. Adding those types to the zero-allowance can't prevent the issue — it only reddens the run after it was filed. Zero prevention, non-zero false alarms.
2. report_incomplete is emitted from the snapshot guard, which runs before any Step 0 dry-run gate: "If the snapshot is missing, malformed, not authoritative, or names any label other than exact ci-scan, register and emit report_incomplete once and stop the whole run." That is how a preview reports it couldn't proceed. A write-free canary that can't report its own blocker is strictly worse than one that files a diagnostic — and a dry-run canary is precisely when you want to hear that the snapshot is broken.

So the contract enforced here is "emit no MUTATION", not "emit no telemetry". The run's own reporting about itself isn't an agent code/PR mutation — which is the same conclusion you reached on impact, arrived at from the emit path.

What was genuinely missing is that the code only justified the carve-out for legitimate runs and said nothing about why it survives dry_run. That silence is why this got raised. Fixed in 0ed9cd964f — the rationale is now in the reconciliation script itself (both twins), and the existing dry-run diagnostics case now covers the exact type you named:

-AgentOutput '{"items":[{"type":"missing_tool"},{"type":"missing_data"},{"type":"noop"},{"type":"create_report_incomplete_issue"}]}'

Mutation-verified, since a carve-out test that passes vacuously is worthless: adding noop and create_report_incomplete_issue to the zero-allowance loop in both locks fails it on both twins — Expected 0, but got 1. ×2. So a future "fix" for this exact review note now trips a test that explains why it's wrong.

Validation

• gh aw compile --strict with the pinned v0.82.14 (matches compiler_version in both locks): 0 errors. Source + lock committed together; only the two affected locks changed (frontmatter hash + the post-step script), and the unrelated actions-lock.json cache entry the compiler added was reverted to keep the diff scoped.
• Recompile is idempotent — a second --strict run produced no further drift.
• 665/665 in .github/scripts (same count you measured), git diff --check clean.
• No behaviour change: no permissions, triggers, egress, caps, or allow-lists touched.

[Copilot]

Copilot's findings

• Files reviewed: 13/13 changed files
• Comments generated: 0 new