Skip to content

Harden CI-fixer discovery and safe-output transport - #36840

Closed
PureWeen wants to merge 2 commits into
dotnet:mainfrom
PureWeen:pureween-fix-ci-fixer-runtime
Closed

Harden CI-fixer discovery and safe-output transport#36840
PureWeen wants to merge 2 commits into
dotnet:mainfrom
PureWeen:pureween-fix-ci-fixer-runtime

Conversation

@PureWeen

Copy link
Copy Markdown
Member

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description of Change

Hardens both scheduled CI-fixer workflows after two production failures following #36775:

This change:

  1. Builds a deterministic pre-agent snapshot containing only open issues with the caller's exact label (ci-scan or ci-scan-net11), optionally scoped to a dispatch issue number. Issue counts, titles, and bodies are bounded and sanitized; title/body content is explicitly untrusted inert data.
  2. Removes broad live issue discovery from the agent tool surface and makes the bounded snapshot authoritative.
  3. Pins the capture-time MCP gateway base before startup with the supported pre-agent-steps hook, while independently pinning apply-time handlers. The net11 workflow now validates against net11.0, not repository-default main.
  4. Gates code transport on append-only ancestry, merge-free history, allowed paths, at most 3 commits, 20 files, and 256 KiB. Existing PR advances use the saved PR head so only the intended delta is transported.
  5. Registers required safe outputs before emission and reconciles them with authoritative agent_output.json after the agent exits. Backend errors, malformed expectations, or missing required captures fail the agent job; a legitimate no-op has no mutation expectation and remains green.
  6. Adds hermetic PowerShell and Vally coverage for both incidents while preserving the separate ownership gate.

Security Model

  • Issue titles and bodies are never executed or interpolated into commands.
  • Evidence is capped at 20 issues, 256 title characters, and 12,000 body characters.
  • No direct or post-agent GitHub write path was added; all writes still use supported gh-aw Safe Outputs.
  • Oversized, unrelated, stale-base, merge, or non-append-only transports fail before a mutation output is emitted.

Validation

  • PowerShell/Pester: 20/20 passed, including the exact 3,377-file stale-base fixture.
  • gh-aw v0.82.14 strict compile/validation: 2 workflows, 0 errors, 0 warnings.
  • Strict Vally lint: capability and ownership specs valid.
  • Vally capability eval: 98.3% across 30 trials; all four incident-focused scenarios passed 12/12 trials.
  • Vally ownership gate: 100% across 12 trials.
  • Focused independent review: no high-confidence correctness or security defects.
  • Pipeline security grep checks: no changed-file violations.
  • Poutine: no findings in the changed workflows; reported only unrelated repository baseline findings.
  • Zizmor: no warning/error-severity findings; 8 low-confidence informational findings in v0.82.14-generated MCP heredocs.
  • Actionlint: only the four known v0.82.14 generated-expression schema mismatches (secret_verification_result and github.aw.import-inputs.random_seed, once per workflow).

Staged Fork Proof

The fork-only workflow is guarded to PureWeen/maui, requires dry_run=true, uses global safe-outputs.staged: true, and performs no real writes.

Scenario Run Result
Main exact-label evidence survives 50/50 live-search filtering and emits a legitimate no-op 30295671565 Green
Net11 saved-head one-file delta captures and previews both PR push and body update 30294895100 Green
Exact wrong-base fixture exposes 3,378 changed files and deliberately ends non-green 30295671617 Expected failure

The successful net11 proof shows DEFAULT_BRANCH=net11.0 in the capture-time MCP gateway and apply-time handler, previews only src/Essentials/test/UnitTests/ForkValidationTransport.txt, and leaves fork PR #169's head, body, labels, draft state, and updated timestamp unchanged.

gh-aw v0.83.1 / #36772

#36772 is a broader fleet upgrade and is intentionally not bundled here. Both v0.82.14 and v0.83.1 schemas reject push-to-pull-request-branch.base-branch, even though runtime code recognizes that field. This PR instead uses the supported pre-agent environment hook plus existing Safe Outputs configuration, so the focused production fix does not depend on that upgrade.

What NOT to Do

  • Do not restore broad live issue search; the bounded exact-label snapshot is the authoritative candidate source.
  • Do not compare a net11 PR branch with repository-default main.
  • Do not trim an oversized/unrelated diff to make it pass the transport gate.
  • Do not convert a missing required safe output or backend error into noop or a direct write.

Issues Fixed

Follow-up to #36775.

Copilot AI added 2 commits July 27, 2026 12:12
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bfd33e26-0ff8-45d4-9ef3-72a4ea1f93cf
Pin the capture-time MCP gateway base before startup, keep apply-time handlers aligned, and make attempt-cap decisions deterministic.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: bfd33e26-0ff8-45d4-9ef3-72a4ea1f93cf
Copilot AI review requested due to automatic review settings July 27, 2026 19:17
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:17 — with GitHub Actions Inactive
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 36840

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 36840"

@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:17 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

@PureWeen — new skill validation results are available based on this last commit: 5937117.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Failed Static Failed LLM Needs attention Skills 0 Agents 0

Skill Validation Results5937117 · Harden CI-fixer discovery and safe-output transport · 2026-07-27T19:17:43Z

❌ Static Checks Failed

Skills: 0 | Eval specs linted:

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Hardens the CI-fixer (main + net11) agentic workflows by making issue discovery deterministic and bounded, pinning the safe-output validation base branch correctly, and failing closed when safe outputs or transport constraints aren’t met.

Changes:

  • Replace broad live issue discovery with a bounded, exact-label pre-agent snapshot (plus watch PR state) and remove issues/search from the agent tool surface.
  • Pin capture/apply validation base (DEFAULT_BRANCH) via pre-agent-steps and safe-outputs env; tighten transport limits (commits/files/bytes) and add expectation-based safe-output reconciliation.
  • Add Pester + Vally coverage for the incidents, plus new transport/expectation helper scripts.
Show a summary per file
File Description
.github/workflows/ci-status-fix.md Main CI-fixer: authoritative prefetch, toolset tightening, base pinning, transport bounds, and post-step safe-output reconciliation.
.github/workflows/ci-status-fix.lock.yml Regenerated lock reflecting the workflow hardening (toolsets, pins, bounds, post-step).
.github/workflows/ci-status-fix-net11.md net11 CI-fixer: same hardening, pinned to net11.0 base for capture/apply validation.
.github/workflows/ci-status-fix-net11.lock.yml Regenerated lock reflecting the net11 workflow hardening.
.github/skills/ci-fix/tests/eval.vally.yaml Adds incident-focused eval scenarios (integrity filtering, stale-base divergence, backend failure).
.github/skills/ci-fix/SKILL.md Updates skill guidance to fail-closed at write boundary and enforce transport gating.
.github/scripts/Test-CiFixTransport.Tests.ps1 New Pester tests covering transport validation scenarios (incl. stale-base fixture).
.github/scripts/Test-CiFixTransport.ps1 New transport gate enforcing append-only ancestry, allowed paths, bounded commits/files/patch bytes, and expectation registration.
.github/scripts/Register-CiFixSafeOutputExpectation.Tests.ps1 New Pester tests for expectation registration.
.github/scripts/Register-CiFixSafeOutputExpectation.ps1 New expectation registration helper used to reconcile intended vs captured safe outputs.
.github/scripts/Query-CiFixPRs.Tests.ps1 New Pester coverage for issue evidence bounding/sanitization and dispatch scoping behavior.
.github/scripts/Query-CiFixPRs.ps1 Extends pre-agent prefetch to include bounded exact-label issue evidence and schemaVersion=2 output.

Copilot's findings

  • Files reviewed: 12/12 changed files
  • Comments generated: 2

Comment on lines +100 to +104
$registerScript = Join-Path $PSScriptRoot 'Register-CiFixSafeOutputExpectation.ps1'
& $registerScript `
-Type $ExpectedOutputType `
-PullRequestNumber $PullRequestNumber `
-OutputDirectory $ExpectationDirectory | Out-Null
Comment on lines +25 to +41
It 'accepts a small append-only allowed diff and registers the expected output' {
'fix' | Set-Content -LiteralPath 'src/Essentials/Test.cs'
git add .
git commit --quiet -m fix

$result = & $script:scriptPath `
-BaseRef $script:base `
-ExpectedOutputType push_to_pull_request_branch `
-PullRequestNumber 36619 `
-ExpectationDirectory $script:expectations | ConvertFrom-Json

$result.commitCount | Should -Be 1
$result.changedFiles | Should -Be @('src/Essentials/Test.cs')
$result.patchBytes | Should -BeGreaterThan 0
@(Get-ChildItem -LiteralPath $script:expectations -Filter '*.json').Count | Should -Be 1
}

@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:21 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:22 — with GitHub Actions Inactive
@github-actions github-actions Bot added the area-ai-agents Copilot CLI agents, agent skills, AI-assisted development label Jul 27, 2026
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:22 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:22 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:22 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:22 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:26 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:26 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:26 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:34 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:37 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:37 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool July 27, 2026 19:38 — with GitHub Actions Inactive
@PureWeen PureWeen closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants