Harden CI scanner issue payload normalization - #37125
Merged
Merged
Conversation
Canonicalize typographic title dashes and repair evidence-backed match-pattern handoff at the trusted publication boundary. Align both scanner prompts with full evidence-line and order-independent cap semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 41235a9a-79b4-433f-9e0d-7278c916a7c9
PureWeen
temporarily deployed
to
copilot-pat-pool
August 5, 2026 20:32 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 37125Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 37125" |
PureWeen
temporarily deployed
to
copilot-pat-pool
August 5, 2026 20:32 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
PureWeen
temporarily deployed
to
copilot-pat-pool
August 5, 2026 20:33 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
There was a problem hiding this comment.
Pull request overview
Hardens the CI scanner’s trusted publication boundary and its gh-aw workflow prompts to reduce agent-manifest mismatches (typographic dashes in titles, missing duplicated match_pattern in bodies) and to make the 5-issue cap semantics order-independent, while preserving the existing “full frozen evidence line” identity requirement.
Changes:
- Canonicalize U+2013/U+2014 to ASCII
-in issue titles at the trusted boundary before the printable-ASCII gate. - Evidence-bind
match_patternand repair bodies that omitted it by appending a bounded “Trusted Match Pattern” excerpt (without weakening full-evidence-line verification). - Define
cap-reachedsemantics across the whole manifest (order-independent), update both workflow prompts accordingly, and regenerate both gh-aw lock files; extend Pester/mutation coverage.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/ci-status-net11.md | Updates agent/threat-detection prompt text for evidence-bound match-pattern repair and order-independent cap semantics. |
| .github/workflows/ci-status-net11.lock.yml | Regenerated compiled lock with updated CUSTOM_PROMPT content. |
| .github/workflows/ci-status-main.md | Same prompt-contract updates as net11 twin. |
| .github/workflows/ci-status-main.lock.yml | Regenerated compiled lock with updated CUSTOM_PROMPT content. |
| .github/scripts/Validate-CiScanManifest.Tests.ps1 | Updates/extends Pester coverage for title dash canonicalization, match_pattern repair injection, and order-independent cap behavior. |
| .github/scripts/Validate-CiScanManifest.ps1 | Implements title canonicalization, match_pattern hidden/control rejection, trusted match_pattern excerpt injection, and order-independent cap validation. |
| .github/scripts/CiScanMutation.Tests.ps1 | Extends mutation coverage to ensure regressions re-break as expected (dash canonicalization and match_pattern repair). |
Copilot's findings
- Files reviewed: 7/7 changed files
- Comments generated: 0
PureWeen
temporarily deployed
to
copilot-pat-pool
August 5, 2026 20:37 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
PureWeen
temporarily deployed
to
copilot-pat-pool
August 5, 2026 20:38 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
github-actions
Bot
added
the
area-infrastructure
PureWeen
temporarily deployed
to
copilot-pat-pool
August 5, 2026 20:38 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
github-actions
Bot
added
the
s/agent-review-in-progress
MauiBot
added
s/agent-review-incomplete
MauiBot
suggested changes
Aug 5, 2026
MauiBot
left a comment
MauiBot
left a comment
Collaborator
There was a problem hiding this comment.
AI Review Summary
@PureWeen — new AI review results are available based on this last commit:
ac84032.
🗂️ Review Sessions — click to expand
🚦 Gate — Test Before & After Fix
Gate Result: ⚠️ INCONCLUSIVE
Platform: ANDROID
⚠️ verify-tests-fail.ps1exited before writing a verification report. Diagnostics below.
Exit code: 3
Likely cause:
- Test detection failed — no runnable tests were found in the PR diff.
- No fix files detected in the diff (PR may be test-only — should now run in failure-only mode).
Gate output log (last 60 lines)
📁 Output directory: CustomAgentLogsTmp/PRState/37125/PRAgent/gate/verify-tests-fail
🔍 Detecting base branch and merge point...
No PR detected, scanning remote branches for closest base...
✅ Base branch: main (via closest-merge-base)
✅ Merge base commit: 8266dc21
(1 commits ahead of main)
╔═══════════════════════════════════════════════════════════╗
║ VERIFY FAILURE ONLY MODE ║
╠═══════════════════════════════════════════════════════════╣
║ No fix files detected - will only verify: ║
║ 1. Tests FAIL (proving they catch the bug) ║
║ ║
║ Use this mode when creating tests before writing a fix. ║
╚═══════════════════════════════════════════════════════════╝
🔍 Auto-detecting test filter from changed test files...
⚠️ No tests detected in this PR.
Searched for: UI tests, unit tests, XAML tests, device tests
Consider adding tests via write-tests-agent.
📋 Pre-Flight — Context & Validation
Issue: N/A - production CI scanner dry-run payload failures
PR: #37125 - Harden CI scanner issue payload normalization
Platforms Affected: CI scanner workflow/publisher infrastructure; test platform requested: android, but changed code is not Android runtime code
Files Changed: 7 implementation/workflow/test assets, 0 app/runtime tests
Key Findings
- PR #37125 changes the trusted CI scanner publication boundary in
.github/scripts/Validate-CiScanManifest.ps1, scanner Pester coverage, and bothci-statusgh-aw twins/locks. - The PR addresses three production dry-run issues: U+2013/U+2014 dash titles rejected by ASCII validation,
match_patternselected from evidence but missing from body, and order-dependentcap-reachedclassification. ghmetadata/check commands are unauthenticated in this environment; public API/local git supplied PR context. Gate was already reported inconclusive by the caller and was not rerun.
Code Review Summary
Verdict: NEEDS_DISCUSSION
Confidence: low
Errors: 0 | Warnings: 0 | Suggestions: 0
Key code review findings:
- ℹ No concrete code findings; review risk is CI/status uncertainty, not a located implementation defect.
- ℹ External-output contract remains the critical surface: scanner manifest fields are untrusted until proven against frozen evidence and canonical publisher markers.
Fix Candidates
| # | Source | Approach | Test Result | Files Changed | Notes |
|---|---|---|---|---|---|
| PR | PR #37125 | Trusted-boundary en/em dash title canonicalization, evidence-bound match_pattern validation and trusted excerpt repair, full evidence-line identity check, order-independent cap semantics |
.github/scripts/Validate-CiScanManifest.ps1, scanner Pester tests, gh-aw twins/locks |
Original PR; code review found no concrete issues. |
🔬 Code Review — Deep Analysis
Code Review — PR #37125
Independent Assessment
What this changes: Hardens CI scanner manifest validation: title dash canonicalization, evidence-bound match_pattern repair, hidden/control checks for patterns, full evidence-line requirements, and order-independent cap semantics.
Inferred motivation: Make scanner publication robust against model output variance while preserving trusted evidence guarantees.
External Output Contract
| Consumer token/pattern | Producer location | Producer emission condition | Consumer assumption | Ordinary negative case | Downstream effect |
|---|---|---|---|---|---|
U+2013/U+2014 in title |
agent manifest | Model emits typographic dash in issue title | Dash is non-evidence punctuation and can be normalized | Curly quote / emoji / NBSP | Still rejected by ASCII gate |
match_pattern |
agent manifest + frozen CI logs | Agent selects one-line substring from evidence | Pattern is trusted only after frozen evidence proof | Pattern absent from claimed log or hidden/control content | Manifest rejected |
missing body match_pattern |
agent issue body | Agent omits duplicated substring | Trusted publisher may append bounded excerpt after evidence proof | Short pattern but no full evidence line | Full-line hash check still rejects |
cap-reached |
agent disposition | Actionable signature omitted solely due global 5-filed cap | Valid if exactly five filed across complete manifest | Fewer than five filed | Manifest rejected |
Reconciliation with PR Narrative
Author claims: PR fixes dash-title rejection, missing match_pattern body duplication, full evidence-line prompting, and order-independent cap handling.
Agreement/disagreement: Matches the committed code and tests.
Prior Review Reconciliation
No prior ❌ Error findings found. Queried top-level reviews, inline comments, and issue comments; only a bot summary with no findings was present.
Blast Radius Assessment
- Runs for all instances: Yes — all CI scanner manifest publication.
- Startup impact: No app startup impact; CI infrastructure path only.
- Static/shared state: Script constants only; no persistent mutable shared state.
CI Status
- Required-check result: undetermined/pending.
gh pr checks --requiredfailed due missing auth; public check-runs showBuild Analysisin progress and commit status pending. - Classification: undetermined.
- Action taken: invoked
azdo-build-investigator;ci-analysiswas unavailable. Confidence capped low; no LGTM.
Findings
No code findings. maui-expert-reviewer also returned no inline findings for the committed PR diff.
Failure-Mode Probing
- Model emits em/en dash title: normalized before ASCII gate; other Unicode remains rejected.
- Agent omits
match_patternfrom body: appended only after evidence proof; length rechecked. - Agent supplies paraphrase instead of full evidence line: post-publish evidence-line hash check still fails.
cap-reachedbefore fifth filed entry: accepted only if final manifest has exactly five filed entries.- Substantive skip after cap: remains allowed and evidence-backed.
Verdict: NEEDS_DISCUSSION
Confidence: low
Summary: Code review found no concrete correctness issues, but CI is pending/undetermined and required checks could not be verified with unauthenticated gh. Per the skill rules, this cannot be LGTM until CI status is resolved.
🛠️ Fix — Analysis & Comparison
Fix Candidates
| # | Source | Approach | Test Result | Files Changed | Notes |
|---|---|---|---|---|---|
| 1 | try-fix | Inline title normalization plus prompt-side ASCII dash enforcement | ✅ PASS | 5 files | Valid alternative, but not demonstrably better than PR named helper; same trusted-boundary behavior with less reusable structure. |
| 2 | try-fix | Remove match_pattern body-substring repair and rely only on full evidence-line hash identity |
✅ PASS | 5 files | Simpler/stricter, but rejects the production omission case instead of repairing it; not better for the PR goal. |
| 3 | try-fix | Canonical payload model with trusted frozen-evidence excerpt repair | ✅ PASS | 7 files | Repairs missing body evidence from frozen trusted evidence, not agent text; stronger than PR's match-pattern-only excerpt, but more invasive. |
| PR | PR #37125 | Trusted-boundary dash canonicalization helper, evidence-bound match_pattern repair, full-line hash assertion, complete-manifest cap semantics | 7 files | Original PR; code review found no concrete issues. |
Cross-Pollination
| Model | Round | New Ideas? | Details |
|---|---|---|---|
| claude-opus-4.6 | 1 | Yes | Candidate 1 tried prompt/inline title normalization. |
| maui-expert-reviewer + claude-opus-4.7 | 2 | Yes | Candidate 2 tried treating match_pattern as metadata and removed body repair. |
| maui-expert-reviewer + gpt-5.3-codex | 3 | Yes | Candidate 3 repaired from a bounded full trusted evidence excerpt. |
Exhausted: No — stopped because Candidate #3 passed the focused scanner regression suite and is materially stronger than the PR for the body-repair failure mode.
Selected Fix: Candidate #3 — it fixes the production omission using frozen trusted evidence, preserves the full evidence-line identity invariant, and avoids publishing an agent-selected match_pattern as a standalone synthetic excerpt. Tradeoff: it is larger/more invasive than the PR.
📝 Recommended PR Title & Description
Assessment: ✏️ Recommend updating — the current description accurately describes the submitted PR, but the winning candidate changes the repair from a Trusted Match Pattern substring excerpt to a trusted full evidence excerpt.
Recommended title
CI scanner: Harden issue payload normalization
Recommended description
### Root Cause
The first post-merge dry runs after #36958 exposed two independent agent-manifest composition failures at `16671cadb63739c47da264953ee3e1267806cbb8`:
- [Net11 run 31038805792](https://github.com/dotnet/maui/actions/runs/31038805792) produced five otherwise valid issue titles with U+2014 EM DASH even though the prompt required printable ASCII. The trusted validator correctly rejected the first title.
- [Main run 31038805939](https://github.com/dotnet/maui/actions/runs/31038805939) selected the evidence-backed pattern `XHarness exit code: 1 (TESTS_FAILED)` for the CarouselView leak signature but omitted that duplicated value from its issue body. The trusted validator correctly rejected the mismatch.
Replaying the net11 artifact also exposed two bodies that paraphrased rather than copied an entire trusted evidence line, plus an order-dependent `cap-reached` classification. Prompt compliance alone is not a reliable publication boundary for duplicated or normalized payload fields.
### Description of Change
- Canonicalize only U+2013 EN DASH and U+2014 EM DASH to ASCII `-` in issue titles before the existing printable-ASCII gate. Curly quotes, non-breaking spaces, emoji, controls, and all other non-ASCII remain rejected.
- Validate every agent-selected `match_pattern` against frozen same-run evidence before use.
- If an otherwise valid body omitted either the exact `match_pattern` or a complete trusted evidence-line identity, append a bounded, safely rendered `Trusted Evidence Excerpt` copied from frozen trusted evidence. Hidden/control/marker content still fails closed.
- Preserve the independent full-evidence-line hash check over the exact published body. The trusted excerpt repairs from a full evidence line and does not weaken evidence identity to a substring check.
- Define the five-issue cap across the complete manifest rather than traversal order. `cap-reached` may appear before or after the fifth filed entry, while substantive skip reasons remain valid regardless of position.
- Align both gh-aw twins and their threat-detection prompts with trusted canonicalization, evidence-bound augmentation, timestamp ownership, and complete evidence-line requirements.
- Regenerate both lock files with strict gh-aw compilation.
### What NOT to Do
- Do not broadly normalize Unicode titles; only the two unambiguous typographic dash separators are accepted.
- Do not inject an agent-selected pattern before frozen evidence proves it in every claimed source log.
- Do not weaken complete evidence-line identity to a substring check.
- Do not make cap exhaustion depend on manifest traversal order.
### Validation
- Focused scanner Pester suite: `Invoke-Pester -Path .github/scripts/Validate-CiScanManifest.Tests.ps1,.github/scripts/CiScanMutation.Tests.ps1 -Output Detailed` passed for the winning candidate.
- Candidate review found no unresolved code findings.
### Safety and Rollout
The changed code is CI scanner workflow/publisher infrastructure, not Android runtime code. The publisher remains the trusted boundary: it owns canonical markers, recounts matches from frozen evidence, rejects unsafe hidden/control/marker content, and fails closed when the published body cannot be bound to trusted evidence.
🏁 Report — Final Recommendation
Comparative Candidate Report — PR #37125
Candidate Ranking
| Rank | Candidate | Regression result | Assessment |
|---|---|---|---|
| 1 | try-fix-3 |
✅ PASS | Best candidate. It preserves the PR's title dash canonicalization, evidence-bound match_pattern proof, full-line hash invariant, and order-independent cap semantics, while strengthening body repair to append a bounded trusted full evidence excerpt from frozen evidence when either the exact pattern or a full trusted evidence-line identity is missing. |
| 2 | pr |
Correct and review-clean, but repairs only by appending the selected match_pattern. This fixes the production omission when the selected pattern is itself a full trusted line, but does not repair the adjacent case where the body contains the pattern without a complete trusted evidence line. |
|
| 3 | pr-plus-reviewer |
Same as pr. The expert review produced no actionable inline findings to apply before timing out, so this candidate has no material delta from the submitted PR fix. |
|
| 4 | try-fix-1 |
✅ PASS | Functionally similar to the PR for trusted-boundary behavior, with inline title normalization and prompt-side ASCII dash reinforcement. It is valid but not materially stronger than the PR's named helper and does not improve the body-repair model. |
| 5 | try-fix-2 |
✅ PASS | Simpler and stricter by removing body repair and relying only on full evidence-line identity. It is secure, but it intentionally rejects the production omission case instead of repairing it, so it is less aligned with the PR goal. |
Winning Candidate
Winner: try-fix-3
try-fix-3 is the strongest fix because it keeps the trusted publisher, not the model, responsible for repairing duplicated evidence payload fields, and the repair payload is a full frozen evidence line rather than an agent-selected substring. It still fails closed for unsafe excerpt content, excessive length, marker-like content, missing evidence proof, and manifests that do not satisfy the full evidence-line hash invariant after injection.
Why the PR does not win
The raw PR fix is acceptable on its merits and has no concrete code-review finding. However, its Trusted Match Pattern repair is narrower than the failure class uncovered during replay: it can repair an omitted pattern, but only satisfies the full-line identity invariant when the selected pattern is also a complete trusted evidence line. try-fix-3 repairs from the canonical trusted evidence model and therefore covers both omitted-pattern and missing-full-line cases with a safer evidence-owned excerpt.
pr-plus-reviewer
No actionable reviewer feedback was available to apply. The required inline findings artifact was written as an empty JSON array, so pr-plus-reviewer is equivalent to pr for this comparison.
Final Recommendation
Use try-fix-3 as the candidate to advance. Its diff is larger than the submitted PR's focused repair, but the extra complexity is targeted: one shared published-evidence hash helper, representative-line capture during trusted evidence proof, and tests that prove repair from frozen full evidence instead of agent prose.
🧭 Next Steps — alternative fix proposed (try-fix-3)
Automated review — alternative fix proposed
The expert-reviewer evaluation compared the PR fix against automatically generated candidates and selected try-fix-3 as the strongest fix.
Why: try-fix-3 won because it preserves the PR hardening while repairing from a bounded full trusted evidence line copied from frozen evidence, not merely the agent-selected match_pattern substring. All try-fix candidates passed their focused regression tests; try-fix-3 provides the strongest trusted-boundary behavior while the PR/pr-plus-reviewer gate remained inconclusive.
Please consider applying the candidate diff below (or use it as guidance). Once you push an update, this workflow will re-trigger and re-evaluate.
Candidate diff (try-fix-3)
diff --git a/.github/scripts/CiScanMutation.Tests.ps1 b/.github/scripts/CiScanMutation.Tests.ps1
index d0817201a6..64514fcd93 100644
--- a/.github/scripts/CiScanMutation.Tests.ps1
+++ b/.github/scripts/CiScanMutation.Tests.ps1
@@ -21,387 +21,468 @@ BeforeAll {
$script:ValidatorPath = Join-Path $PSScriptRoot 'Validate-CiScanManifest.ps1'
$script:ValidatorSource = Get-Content -LiteralPath $script:ValidatorPath -Raw
$script:Mutations = @{
# The publisher stops adding the canonical marker block.
'no-injection' = @{
Find = '$publishedBody = (New-CanonicalMarkerBlock `
-Fingerprint $Fingerprint `
-MatchCount $trustedEvidenceProof.MatchCount `
-EvidenceKey $trustedEvidenceProof.EvidenceKey) + "`n`n" + $body'
Replace = '$publishedBody = $body'
}
# The fingerprint marker is sourced from agent-controlled body text
# instead of the validated manifest structure.
'fingerprint-from-body' = @{
Find = '$publishedBody = (New-CanonicalMarkerBlock `
-Fingerprint $Fingerprint `
-MatchCount $trustedEvidenceProof.MatchCount `
-EvidenceKey $trustedEvidenceProof.EvidenceKey) + "`n`n" + $body'
Replace = '$publishedBody = (New-CanonicalMarkerBlock `
-Fingerprint ([regex]::Match($rawBody, ''(?m)^claimed-fingerprint: (.+)$'').Groups[1].Value) `
-MatchCount $trustedEvidenceProof.MatchCount `
-EvidenceKey $trustedEvidenceProof.EvidenceKey) + "`n`n" + $body'
}
# The count marker is no longer the frozen-evidence recount.
'untrusted-count' = @{
Find = '-MatchCount $trustedEvidenceProof.MatchCount `
-EvidenceKey $trustedEvidenceProof.EvidenceKey) + "`n`n" + $body'
Replace = '-MatchCount ($trustedEvidenceProof.MatchCount + 7) `
-EvidenceKey $trustedEvidenceProof.EvidenceKey) + "`n`n" + $body'
}
# Validation happens only before injection: the post-injection assertion
# over the exact published payload is removed.
'no-post-injection-check' = @{
Find = ' Assert-CanonicalPublishedBody `
-Body $publishedBody `'
Replace = ' Assert-NoOpPublishedBody `
-Body $publishedBody `'
}
# Pre-existing / duplicate / evasive marker content is accepted from the agent.
'no-duplicate-rejection' = @{
Find = ' if (Test-MarkerLikeContent -Value $rawBody) {'
Replace = ' if ($false) {'
}
# The hidden/control-content rejection is a distinct trusted-boundary
# layer from the marker check. Disabling it lets a body carrying an HTML
# comment (which the canonical marker also is) or invisible content flow
# to the deeper post-injection backstop instead of stopping at the edge.
'no-hidden-content-rejection' = @{
Find = ' if ($hiddenReason) {'
Replace = ' if ($false) {'
}
# Marker-like match patterns can replay trusted publisher state.
'no-marker-pattern-rejection' = @{
Find = ' if (Test-MarkerLikeContent -Value $matchPattern) {'
Replace = ' if ($false) {'
}
# Synthetic framing is put back into the countable raw segment set.
'synthetic-framing-counted' = @{
Find = ' -TrustedEvidencePath $TrustedEvidencePath)
foreach ($segment in $segments) {'
Replace = ' -TrustedEvidencePath $TrustedEvidencePath)
$segments += [pscustomobject]@{ content = "===== AzDO log $BuildId/$sourceLogId =====" }
foreach ($segment in $segments) {'
}
# Run-specific AzDO transport timestamps remain part of evidence identity.
'timestamp-sensitive-identity' = @{
Find = ' if ($StripAzdoTransportTimestamp) {
# Azure DevOps prepends a run-specific UTC timestamp to every stored log
# line. Segment provenance decides whether it is transport framing; the
# same timestamp in Helix or other evidence remains part of the message.
$normalized = [regex]::Replace(
$normalized,
''^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,7})?Z[ \t]+'',
''''
)
}'
Replace = ''
}
+ # The trusted boundary once again relies on the model to avoid typographic
+ # dashes despite an explicit ASCII prompt.
+ 'title-dash-not-canonicalized' = @{
+ Find = ' return $Value.
+ Replace([char]0x2013, [char]0x002D).
+ Replace([char]0x2014, [char]0x002D).
+ Trim()'
+ Replace = ' return $Value.Trim()'
+ }
+ # The publisher once again requires the agent to duplicate match_pattern in
+ # its issue body instead of repairing from frozen full evidence.
+ 'no-match-pattern-injection' = @{
+ Find = ' $body = Add-TrustedEvidenceExcerpt `
+ -Body $body `
+ -MatchPattern $matchPattern `
+ -EvidenceLineHashes $trustedEvidenceProof.EvidenceLineHashes `
+ -RepresentativeEvidenceLine $trustedEvidenceProof.RepresentativeEvidenceLine'
+ Replace = ' $body = $body'
+ }
}
function New-MutatedValidator {
param(
[Parameter(Mandatory = $true)][string[]]$Mutation,
[Parameter(Mandatory = $true)][string]$Path
)
$source = $script:ValidatorSource
foreach ($name in $Mutation) {
$definition = $script:Mutations[$name]
if (-not $definition) {
throw "Unknown mutation '$name'."
}
if (-not $source.Contains($definition.Find)) {
throw "Mutation '$name' no longer matches the validator source; update the mutation."
}
$source = $source.Replace($definition.Find, $definition.Replace)
}
# Stand-in for the removed post-injection assertion, so the mutant runs
# instead of dying on a missing command.
$source = $source.Replace(
'function Assert-CanonicalPublishedBody {',
"function Assert-NoOpPublishedBody { param(`$Body, `$Fingerprint, `$MatchCount, `$EvidenceKey, `$EvidenceLineHashes, `$MatchPattern, `$PipelineName, `$BuildId) }`n`nfunction Assert-CanonicalPublishedBody {")
Set-Content -LiteralPath $Path -Value $source -Encoding utf8
return $Path
}
function Test-FixedManifestHandoff {
param([Parameter(Mandatory = $true)][string]$Source)
return $Source -match 'CI_SCAN_MANIFEST_PATH: \$\{\{ runner\.temp \}\}/gh-aw/safe-jobs/agent/manifest_final\.json' -and
$Source -match 'argument-free `submit_ci_scan`' -and
$Source -notmatch '(?ms)^\s{6}inputs:\s*\r?\n\s{8}(?:manifest|manifest_path):' -and
$Source -notmatch 'one `manifest` argument'
}
function Test-BoundedThreatDetectionStaging {
param([Parameter(Mandatory = $true)][string]$Source)
return $Source -match '\[ -L "\$manifest" \] \|\| \[ ! -f "\$manifest" \]' -and
$Source -match '\[ "\$manifest_size" -eq 0 \] \|\| \[ "\$manifest_size" -gt 500000 \]' -and
$Source -match 'cp --no-dereference -- "\$manifest" "\$staged"' -and
$Source -match '\[ -L "\$staged" \] \|\| \[ ! -f "\$staged" \]' -and
$Source -match '\[ "\$staged_size" -ne "\$manifest_size" \]'
}
function Test-TrustedEmojiSelectorPrompt {
param(
[Parameter(Mandatory = $true)][string]$Source,
[Parameter(Mandatory = $true)][string]$ValidatorSource
)
$validatorMatch = [regex]::Match(
$ValidatorSource,
'(?s)\$isEmojiVariationBase = \$previousCode -in @\((?<bases>.*?)\r?\n\s+\)')
$promptMatch = [regex]::Match(
$Source,
'Approved VS15/VS16 bases \(exactly\): (?<bases>U\+[0-9A-F]+(?:, U\+[0-9A-F]+)*)\.')
if (-not $validatorMatch.Success -or -not $promptMatch.Success) {
return $false
}
$validatorBases = @(
[regex]::Matches($validatorMatch.Groups['bases'].Value, '0x(?<code>[0-9A-F]+)') |
ForEach-Object { "U+$($_.Groups['code'].Value)" }
)
$promptBases = @($promptMatch.Groups['bases'].Value -split ', ')
return @(
Compare-Object `
-ReferenceObject $validatorBases `
-DifferenceObject $promptBases `
-SyncWindow 0
).Count -eq 0 -and
$Source -match 'Do not flag VS15 \(U\+FE0E\) or\s+VS16 \(U\+FE0F\) solely when it immediately follows one of the approved bases' -and
$Source -match 'Flag an isolated VS15/VS16 or a selector following any other base\.'
}
+ function Test-FullEvidenceLinePrompt {
+ param([Parameter(Mandatory = $true)][string]$Source)
+
+ return $Source -match 'Copy at least one \*\*entire matching line\*\* from a frozen evidence file' -and
+ $Source -match 'do not summarize it or replace\s+volatile fields with placeholders such as `<id>`' -and
+ $Source -match 'shorter `match_pattern` substring is not sufficient for trusted evidence\s+identity' -and
+ $Source -match 'Do not attempt to classify or remove timestamps yourself; copy them\s+verbatim\.' -and
+ $Source -match 'trusted validator alone normalizes a recognized leading AzDO\s+transport timestamp'
+ }
+
+ function Test-MatchPatternRepairPrompt {
+ param([Parameter(Mandatory = $true)][string]$Source)
+
+ return $Source -match 'trusted\s+publisher verifies it against frozen evidence and appends a\s+canonical trusted evidence excerpt if the safe-rendered body omitted either\s+the exact `match_pattern` or a full trusted evidence-line identity' -and
+ $Source -match 'does not replace\s+the\s+full-evidence-line requirement'
+ }
+
+ function Test-OrderIndependentCapPrompt {
+ param([Parameter(Mandatory = $true)][string]$Source)
+
+ return $Source -match 'exactly five\s+entries are actually marked `filed` across the complete manifest' -and
+ $Source -match 'may appear before or after the fifth filed entry in fixed traversal order' -and
+ $Source -match 'Use a substantive\s+skip reason whenever it applies, even after the cap is reached' -and
+ $Source -match 'do not replace\s+it with `cap-reached` merely because of its position'
+ }
+
function Get-CompiledThreatDetectionPrompt {
param([Parameter(Mandatory = $true)][string]$LockPath)
$lockSource = Get-Content -LiteralPath $LockPath -Raw
$promptMatch = [regex]::Match(
$lockSource,
'(?m)^\s+CUSTOM_PROMPT: (?<json>".*")$')
if (-not $promptMatch.Success) {
throw "The compiled lock '$LockPath' no longer contains the threat-detection CUSTOM_PROMPT."
}
return [System.Text.Json.JsonSerializer]::Deserialize[string](
$promptMatch.Groups['json'].Value)
}
function New-ProbeManifest {
param(
[string]$Path,
[string]$Body,
- [string]$MatchPattern = 'Assertion failed'
+ [string]$MatchPattern = 'Assertion failed',
+ [string]$Title = 'Sample test fails on Windows'
)
$fingerprint = 'ci-scan-net11|net11.0|maui-pr|sample test|assertion failed|windows'
$manifest = [pscustomobject]@{
pipelines = @(
[pscustomobject]@{
name = 'maui-pr'
definition_id = 302
status = 'scanned'
build_id = 123456
signatures = @(
[pscustomobject]@{
fingerprint = $fingerprint
disposition = 'filed'
source_log_ids = @(1001)
- title = 'Sample test fails on Windows'
+ title = $Title
match_pattern = $MatchPattern
body = $Body
}
)
}
[pscustomobject]@{ name = 'maui-pr-devicetests'; definition_id = 314; status = 'scanned'; build_id = 123457; signatures = @() }
[pscustomobject]@{ name = 'maui-pr-uitests'; definition_id = 313; status = 'scanned'; build_id = 123458; signatures = @() }
)
}
Set-Content -LiteralPath $Path -Value ($manifest | ConvertTo-Json -Depth 12)
return $Path
}
function New-ProbeEvidence {
param(
[string]$Root,
[string[]]$Lines = @('Assertion failed', 'Assertion failed')
)
$directory = Join-Path $Root 'maui-pr'
New-Item -ItemType Directory -Path $directory -Force | Out-Null
Set-Content `
-LiteralPath (Join-Path $directory '123456-1001.log') `
-Value $Lines
[pscustomobject]@{
schema_version = 1
pipeline = 'maui-pr'
build_id = 123456
log_id = 1001
segments = @(
[pscustomobject]@{
kind = 'azdo-log'
source = '123456/1001'
content = $Lines -join "`n"
}
)
} | ConvertTo-Json -Depth 6 | Set-Content `
-LiteralPath (Join-Path $directory '123456-1001.evidence.json')
return $Root
}
function Invoke-ValidatorProbe {
param(
[string[]]$Mutation = @(),
[string]$Body = "## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed",
[string]$MatchPattern = 'Assertion failed',
- [string[]]$EvidenceLines = @('Assertion failed', 'Assertion failed')
+ [string[]]$EvidenceLines = @('Assertion failed', 'Assertion failed'),
+ [string]$Title = 'Sample test fails on Windows'
)
$work = Join-Path $TestDrive ('mutation-' + [guid]::NewGuid().ToString('n'))
New-Item -ItemType Directory -Path $work -Force | Out-Null
$validator = if ($Mutation.Count -eq 0) {
$copy = Join-Path $work 'Validate-CiScanManifest.ps1'
Set-Content -LiteralPath $copy -Value $script:ValidatorSource -Encoding utf8
$copy
} else {
New-MutatedValidator -Mutation $Mutation -Path (Join-Path $work 'Validate-CiScanManifest.ps1')
}
$manifestPath = New-ProbeManifest `
-Path (Join-Path $work 'manifest.json') `
-Body $Body `
- -MatchPattern $MatchPattern
+ -MatchPattern $MatchPattern `
+ -Title $Title
$evidencePath = New-ProbeEvidence `
-Root (Join-Path $work 'evidence') `
-Lines $EvidenceLines
$probePath = Join-Path $work 'probe.ps1'
Set-Content -LiteralPath $probePath -Value @'
param([string]$ValidatorPath, [string]$ManifestPath, [string]$EvidencePath)
$ErrorActionPreference = 'Stop'
. $ValidatorPath
try {
$manifest = Get-Content -Raw -LiteralPath $ManifestPath | ConvertFrom-Json
$plan = Test-CiScanManifest -Manifest $manifest -TrustedEvidencePath $EvidencePath
$body = if (@($plan.issues).Count -gt 0) { $plan.issues[0].Body } else { '' }
- Write-Output ('RESULT ' + (ConvertTo-Json -Compress -InputObject @{ ok = $true; body = $body }))
+ $title = if (@($plan.issues).Count -gt 0) { $plan.issues[0].Title } else { '' }
+ Write-Output ('RESULT ' + (ConvertTo-Json -Compress -InputObject @{ ok = $true; body = $body; title = $title }))
} catch {
Write-Output ('RESULT ' + (ConvertTo-Json -Compress -InputObject @{ ok = $false; error = "$($_.Exception.Message)" }))
}
'@
# A child process keeps each mutant's function definitions out of the
# test session, so one mutation cannot leak into the next assertion.
$output = & pwsh -NoProfile -File $probePath $validator $manifestPath $evidencePath 2>&1
$line = @($output | Where-Object { "$_" -like 'RESULT *' }) | Select-Object -Last 1
if (-not $line) {
throw "validator probe produced no result: $output"
}
return ("$line".Substring(7) | ConvertFrom-Json)
}
$script:CanonicalMarker = '<!-- ci-scan-fingerprint: ci-scan-net11|net11.0|maui-pr|sample test|assertion failed|windows -->'
}
Describe 'CI scanner marker mutation coverage' {
It 'baseline: the real validator injects exactly one canonical marker block' {
$result = Invoke-ValidatorProbe
$result.ok | Should -BeTrue
([regex]::Matches($result.body, '<!-- ci-scan-fingerprint:')).Count | Should -Be 1
([regex]::Matches($result.body, '<!-- ci-scan-match-count:')).Count | Should -Be 1
([regex]::Matches($result.body, '<!-- ci-scan-evidence-key:')).Count | Should -Be 1
$result.body.StartsWith($script:CanonicalMarker) | Should -BeTrue
$result.body | Should -Match '(?m)^<!-- ci-scan-match-count: 2 hits in failure\.log -->$'
}
+ It 'mutation "title-dash-not-canonicalized": the production title is rejected again' {
+ $title = "Recurring Android device test failure $([char]0x2014) StatusBarThemeAppliesWhenHandlerConnects fails"
+ $baseline = Invoke-ValidatorProbe -Title $title
+ $mutated = Invoke-ValidatorProbe `
+ -Mutation @('title-dash-not-canonicalized') `
+ -Title $title
+
+ $baseline.ok | Should -BeTrue
+ $baseline.title | Should -BeExactly '[ci-scan-net11] Recurring Android device test failure - StatusBarThemeAppliesWhenHandlerConnects fails'
+ $mutated.ok | Should -BeFalse
+ $mutated.error | Should -Match 'must contain printable single-line ASCII only'
+ }
+
+ It 'mutation "no-match-pattern-injection": the main production mismatch is rejected again' {
+ $pattern = 'XHarness exit code: 1 (TESTS_FAILED)'
+ $body = "## Summary`nRecurring CarouselView leak.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nReference to Microsoft.Maui.Controls.CarouselView is still alive"
+ $baseline = Invoke-ValidatorProbe `
+ -Body $body `
+ -MatchPattern $pattern `
+ -EvidenceLines @($pattern)
+ $mutated = Invoke-ValidatorProbe `
+ -Mutation @('no-match-pattern-injection') `
+ -Body $body `
+ -MatchPattern $pattern `
+ -EvidenceLines @($pattern)
+
+ $baseline.ok | Should -BeTrue
+ $baseline.body | Should -Match '(?ms)## Trusted Evidence Excerpt\r?\n\r?\n XHarness exit code: 1 \(TESTS_FAILED\)$'
+ $mutated.ok | Should -BeFalse
+ $mutated.error | Should -Match 'must contain match_pattern exactly'
+ }
+
It 'mutation "no-injection": removing injection cannot produce a marked issue' {
$result = Invoke-ValidatorProbe -Mutation @('no-injection')
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*does not begin with the canonical marker block*'
}
It 'mutation "no-injection + no-post-injection-check": reproduces the unmarked-issue incident' {
# This is the production failure mode, reconstructed: with both the
# injection and the post-injection assertion gone, a perfectly valid-looking
# run publishes an issue with neither marker and reports success.
$result = Invoke-ValidatorProbe -Mutation @('no-injection', 'no-post-injection-check')
$result.ok | Should -BeTrue
$result.body | Should -Not -Match '<!-- ci-scan-fingerprint:'
$result.body | Should -Not -Match '<!-- ci-scan-match-count:'
}
It 'mutation "fingerprint-from-body": untrusted fingerprint sourcing is rejected' {
$body = "## Summary`nRecurring sample failure.`nclaimed-fingerprint: ci-scan-net11|net11.0|maui-pr|attacker chosen|attacker error|linux`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed"
$result = Invoke-ValidatorProbe -Mutation @('fingerprint-from-body') -Body $body
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*canonical marker block*'
}
It 'mutation "fingerprint-from-body + no-post-injection-check": body content would own the marker' {
# Shows the previous assertion is not vacuous: without the post-injection
# check, the attacker-chosen fingerprint really does reach the marker.
$body = "## Summary`nRecurring sample failure.`nclaimed-fingerprint: ci-scan-net11|net11.0|maui-pr|attacker chosen|attacker error|linux`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed"
$result = Invoke-ValidatorProbe -Mutation @('fingerprint-from-body', 'no-post-injection-check') -Body $body
$result.ok | Should -BeTrue
$result.body | Should -Match 'attacker chosen'
$result.body.StartsWith($script:CanonicalMarker) | Should -BeFalse
}
It 'mutation "untrusted-count": a count that is not the evidence recount is rejected' {
$result = Invoke-ValidatorProbe -Mutation @('untrusted-count')
$result.ok | Should -BeFalse
# Post-injection validation rebuilds the expected block from the trusted
# count, so an inflated count fails the exact-payload comparison.
$result.error | Should -BeLike '*does not begin with the canonical marker block*'
}
It 'mutation "untrusted-count + no-post-injection-check": wrong count would be published' {
$result = Invoke-ValidatorProbe -Mutation @('untrusted-count', 'no-post-injection-check')
$result.ok | Should -BeTrue
$result.body | Should -Match '(?m)^<!-- ci-scan-match-count: 9 hits in failure\.log -->$'
}
It 'mutation "no-duplicate-rejection": a pre-marked body is rejected downstream' {
$body = "$script:CanonicalMarker`n## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed"
# Both edge-layer rejections (marker-like content and hidden/HTML-comment
# content) are disabled so this proves the *post-injection* backstop is
# independently load-bearing against duplicate markers.
$result = Invoke-ValidatorProbe -Mutation @('no-duplicate-rejection', 'no-hidden-content-rejection') -Body $body
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*exactly one canonical fingerprint marker*'
}
It 'mutation "no-duplicate-rejection + no-post-injection-check": duplicate markers would ship' {
$body = "$script:CanonicalMarker`n## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed"
$result = Invoke-ValidatorProbe -Mutation @('no-duplicate-rejection', 'no-hidden-content-rejection', 'no-post-injection-check') -Body $body
$result.ok | Should -BeTrue
([regex]::Matches($result.body, '<!-- ci-scan-fingerprint:')).Count | Should -Be 2
}
It 'baseline: the real validator rejects a body carrying hidden control content' {
$body = "## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed$([char]0x1B)[31m"
$result = Invoke-ValidatorProbe -Body $body
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*must not contain*C0 control character*'
}
@@ -410,285 +491,347 @@ Describe 'CI scanner marker mutation coverage' {
# hidden/control-content layer stands between it and publication.
$body = "## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed`n<!-- reviewer will not see this -->"
$result = Invoke-ValidatorProbe -Mutation @('no-hidden-content-rejection') -Body $body
# Prove the layer is load-bearing by asserting the concrete bypass: with the
# guard disabled the body is fully published with the hidden comment intact,
# not merely that some other error message differs.
$result.ok | Should -BeTrue
$result.error | Should -Not -BeLike '*HTML comment sequence*'
$result.body | Should -BeLike '*<!-- reviewer will not see this -->*'
}
It 'baseline: the real validator rejects that same pre-marked body outright' {
$body = "$script:CanonicalMarker`n## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`nAssertion failed"
$result = Invoke-ValidatorProbe -Body $body
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*must not contain scanner marker content*'
}
It 'mutation "synthetic-framing-counted": a header-only pattern becomes false evidence' {
$header = '===== AzDO log 123456/1001 ====='
$result = Invoke-ValidatorProbe `
-Mutation @('synthetic-framing-counted') `
-MatchPattern '===== AzDO log' `
-EvidenceLines @('Different raw failure') `
-Body "## Summary`nHeader replay.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`n$header"
$result.ok | Should -BeTrue
}
It 'baseline: header-only evidence fails closed when synthetic framing is not countable' {
$header = '===== AzDO log 123456/1001 ====='
$result = Invoke-ValidatorProbe `
-MatchPattern '===== AzDO log' `
-EvidenceLines @('Different raw failure') `
-Body "## Summary`nHeader replay.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`n$header"
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*must occur in trusted source log 1001*'
}
It 'mutation "no-marker-pattern-rejection": marker state can become evidence when both marker gates are removed' {
$pattern = 'ci-scan-fingerprint'
$body = "## Summary`nMarker replay.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`n$pattern"
$result = Invoke-ValidatorProbe `
-Mutation @('no-marker-pattern-rejection', 'no-duplicate-rejection') `
-MatchPattern $pattern `
-EvidenceLines @($pattern) `
-Body $body
$result.ok | Should -BeTrue
}
It 'baseline: marker-like match patterns are rejected before evidence can replay them' {
$pattern = 'ci-scan-fingerprint'
$result = Invoke-ValidatorProbe `
-MatchPattern $pattern `
-EvidenceLines @($pattern) `
-Body "## Summary`nMarker replay.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`n$pattern"
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*scanner marker content*'
}
It 'baseline: AzDO transport timestamps do not prevent trusted body binding' {
$trusted = '2026-07-20T18:34:13.9100750Z ##[error]Path does not exist: artifacts/bin'
$body = "## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`n##[error]Path does not exist: artifacts/bin"
$result = Invoke-ValidatorProbe `
-MatchPattern 'Path does not exist' `
-EvidenceLines @($trusted) `
-Body $body
$result.ok | Should -BeTrue
}
It 'mutation "timestamp-sensitive-identity": realistic cross-build body binding fails' {
$trusted = '2026-07-20T18:34:13.9100750Z ##[error]Path does not exist: artifacts/bin'
$body = "## Summary`nRecurring sample failure.`n`n## Build Information`n- **Pipeline**: maui-pr`n- **Build ID**: 123456`n`n## Error Message`n##[error]Path does not exist: artifacts/bin"
$result = Invoke-ValidatorProbe `
- -Mutation @('timestamp-sensitive-identity') `
+ -Mutation @('timestamp-sensitive-identity', 'no-match-pattern-injection') `
-MatchPattern 'Path does not exist' `
-EvidenceLines @($trusted) `
-Body $body
$result.ok | Should -BeFalse
$result.error | Should -BeLike '*must contain a full trusted evidence line*'
}
}
Describe 'CI scanner twin discovery mutation coverage' {
It 'baseline: discovery finds both compiled twins' {
@(Get-CiScanTwin).Count | Should -Be 2
}
Describe 'CI scanner fixed manifest handoff mutation coverage' {
BeforeAll {
$script:WorkflowSources = @(
Get-Content -LiteralPath (Join-Path $PSScriptRoot '../workflows/ci-status-main.md') -Raw
Get-Content -LiteralPath (Join-Path $PSScriptRoot '../workflows/ci-status-net11.md') -Raw
)
$script:CompiledThreatPrompts = @(
Get-CompiledThreatDetectionPrompt `
-LockPath (Join-Path $PSScriptRoot '../workflows/ci-status-main.lock.yml')
Get-CompiledThreatDetectionPrompt `
-LockPath (Join-Path $PSScriptRoot '../workflows/ci-status-net11.lock.yml')
)
$script:SafeJobStepsNeedle = " steps:`n - name: Require successful agent submission gate"
}
It 'baseline: both twins use the fixed argument-free artifact handoff' {
@($script:WorkflowSources | Where-Object { Test-FixedManifestHandoff -Source $_ }).Count |
Should -Be 2
}
It 'baseline: both twins bound regular-file threat-detection staging' {
@($script:WorkflowSources | Where-Object { Test-BoundedThreatDetectionStaging -Source $_ }).Count |
Should -Be 2
}
It 'baseline: both twins mirror the trusted emoji-selector rule in threat detection' {
@(
$script:WorkflowSources |
Where-Object {
Test-TrustedEmojiSelectorPrompt `
-Source $_ `
-ValidatorSource $script:ValidatorSource
}
).Count | Should -Be 2
}
It 'baseline: both compiled twins execute the trusted emoji-selector rule' {
@(
$script:CompiledThreatPrompts |
Where-Object {
Test-TrustedEmojiSelectorPrompt `
-Source $_ `
-ValidatorSource $script:ValidatorSource
}
).Count | Should -Be 2
}
+ It 'baseline: both twins require a full frozen evidence line in filed bodies' {
+ @($script:WorkflowSources | Where-Object { Test-FullEvidenceLinePrompt -Source $_ }).Count |
+ Should -Be 2
+ }
+
+ It 'baseline: both twins describe trusted match-pattern repair' {
+ @($script:WorkflowSources | Where-Object { Test-MatchPatternRepairPrompt -Source $_ }).Count |
+ Should -Be 2
+ }
+
+ It 'baseline: both twins describe cap exhaustion independent of traversal order' {
+ @($script:WorkflowSources | Where-Object { Test-OrderIndependentCapPrompt -Source $_ }).Count |
+ Should -Be 2
+ }
+
It 'mutation "nested-string-transport": a manifest tool input fails the handoff invariant' {
foreach ($source in $script:WorkflowSources) {
$source.Contains($script:SafeJobStepsNeedle) | Should -BeTrue
$mutated = $source.Replace(
$script:SafeJobStepsNeedle,
" inputs:`n manifest:`n required: true`n type: string`n$($script:SafeJobStepsNeedle)")
$mutated | Should -Not -BeExactly $source
(Test-FixedManifestHandoff -Source $mutated) | Should -BeFalse
}
}
It 'mutation "agent-selected-path": a manifest_path tool input fails the handoff invariant' {
foreach ($source in $script:WorkflowSources) {
$source.Contains($script:SafeJobStepsNeedle) | Should -BeTrue
$mutated = $source.Replace(
$script:SafeJobStepsNeedle,
" inputs:`n manifest_path:`n required: true`n type: string`n$($script:SafeJobStepsNeedle)")
$mutated | Should -Not -BeExactly $source
(Test-FixedManifestHandoff -Source $mutated) | Should -BeFalse
}
}
It 'mutation "symlink-staging": removing the source symlink guard fails the staging invariant' {
foreach ($source in $script:WorkflowSources) {
$mutated = $source.Replace(
'if [ -L "$manifest" ] || [ ! -f "$manifest" ]; then',
'if [ ! -f "$manifest" ]; then')
$mutated | Should -Not -BeExactly $source
(Test-BoundedThreatDetectionStaging -Source $mutated) | Should -BeFalse
}
}
It 'mutation "unbounded-staging": removing the byte cap fails the staging invariant' {
foreach ($source in $script:WorkflowSources) {
$mutated = $source.Replace(
'if [ "$manifest_size" -eq 0 ] || [ "$manifest_size" -gt 500000 ]; then',
'if [ "$manifest_size" -eq 0 ]; then')
$mutated | Should -Not -BeExactly $source
(Test-BoundedThreatDetectionStaging -Source $mutated) | Should -BeFalse
}
}
It 'mutation "selector-carveout-removed": restoring generic selector rejection fails the prompt invariant' {
foreach ($source in $script:WorkflowSources) {
$mutated = [regex]::Replace(
$source,
'(?ms)\n Apply this exact rule to variation selectors\..*?Flag an isolated VS15/VS16 or a selector following any other base\.\r?\n',
"`n Flag variation selectors as hidden or invisible content.`n")
$mutated | Should -Not -BeExactly $source
(Test-TrustedEmojiSelectorPrompt `
-Source $mutated `
-ValidatorSource $script:ValidatorSource) |
Should -BeFalse
}
}
It 'mutation "selector-carveout-widened": adding an untrusted base fails the prompt invariant' {
foreach ($source in $script:WorkflowSources) {
$mutated = $source.Replace(
'U+2764, U+1F6E0.',
'U+2764, U+1F600, U+1F6E0.')
$mutated | Should -Not -BeExactly $source
(Test-TrustedEmojiSelectorPrompt `
-Source $mutated `
-ValidatorSource $script:ValidatorSource) |
Should -BeFalse
}
}
It 'mutation "selector-negative-rule-removed": dropping disallowed-base rejection fails the prompt invariant' {
foreach ($source in $script:WorkflowSources) {
$mutated = $source.Replace(
' Flag an isolated VS15/VS16 or a selector following any other base.',
'')
$mutated | Should -Not -BeExactly $source
(Test-TrustedEmojiSelectorPrompt `
-Source $mutated `
-ValidatorSource $script:ValidatorSource) |
Should -BeFalse
}
}
It 'mutation "stale-compiled-selector-rule": an omitted approved base fails the compiled prompt invariant' {
foreach ($prompt in $script:CompiledThreatPrompts) {
$mutated = $prompt.Replace(
', U+1F6E0.',
'.')
$mutated | Should -Not -BeExactly $prompt
(Test-TrustedEmojiSelectorPrompt `
-Source $mutated `
-ValidatorSource $script:ValidatorSource) |
Should -BeFalse
}
}
+
+ It 'mutation "matching-substring-only": removing the full-line requirement fails the prompt invariant' {
+ foreach ($source in $script:WorkflowSources) {
+ $mutated = [regex]::Replace(
+ $source,
+ '(?ms)\n3\. Copy at least one \*\*entire matching line\*\*.*?identity\.\r?\n',
+ "`n")
+
+ $mutated | Should -Not -BeExactly $source
+ (Test-FullEvidenceLinePrompt -Source $mutated) | Should -BeFalse
+ }
+ }
+
+ It 'mutation "sequential-cap-contract": restoring traversal-order cap semantics fails the prompt invariant' {
+ foreach ($source in $script:WorkflowSources) {
+ $mutated = $source.Replace(
+ 'so it may appear before or after the fifth filed entry in fixed traversal order.',
+ 'so it must appear only after the fifth filed entry in fixed traversal order.')
+
+ $mutated | Should -Not -BeExactly $source
+ (Test-OrderIndependentCapPrompt -Source $mutated) | Should -BeFalse
+ }
+ }
+
+ It 'mutation "agent-normalizes-timestamps": removing trusted timestamp ownership fails the prompt invariant' {
+ foreach ($source in $script:WorkflowSources) {
+ $mutated = [regex]::Replace(
+ $source,
+ '(?ms) Do not attempt to classify or remove timestamps yourself; copy them\r?\n verbatim\. The trusted validator alone normalizes a recognized leading AzDO\r?\n transport timestamp when computing evidence identity\.',
+ '')
+
+ $mutated | Should -Not -BeExactly $source
+ (Test-FullEvidenceLinePrompt -Source $mutated) | Should -BeFalse
+ }
+ }
+
+ It 'mutation "agent-only-pattern-handoff": removing trusted repair fails the prompt invariant' {
+ foreach ($source in $script:WorkflowSources) {
+ $mutated = [regex]::Replace(
+ $source,
+ '(?ms)\n\s*The trusted publisher verifies it against frozen evidence and appends a.*?full-evidence-line requirement below\.',
+ '')
+
+ $mutated | Should -Not -BeExactly $source
+ (Test-MatchPatternRepairPrompt -Source $mutated) | Should -BeFalse
+ }
+ }
}
It 'mutation "one-twin-omitted": discovery reports a single twin' {
# Proves the anti-vacuity assertion in Validate-CiScanPublisher.Tests.ps1
# is load-bearing: dropping a twin changes what discovery returns, so the
# "exactly two" assertion fails rather than silently testing one scanner.
$root = Join-Path $TestDrive 'one-twin'
New-Item -ItemType Directory -Path $root -Force | Out-Null
Copy-Item `
-LiteralPath (Join-Path $PSScriptRoot '../workflows/ci-status-net11.lock.yml') `
-Destination $root
$twins = @(Get-CiScanTwin -WorkflowRoot $root)
$twins.Count | Should -Be 1
{ $twins.Count | Should -Be 2 } | Should -Throw
}
It 'mutation "discovery-empty": an empty discovery fails the anti-vacuity gate' {
$root = Join-Path $TestDrive 'no-twins'
New-Item -ItemType Directory -Path $root -Force | Out-Null
$twins = @(Get-CiScanTwin -WorkflowRoot $root)
$twins.Count | Should -Be 0
{ $twins.Count | Should -Be 2 } | Should -Throw
}
It 'mutation "publisher-step-renamed": extraction fails instead of testing nothing' {
$root = Join-Path $TestDrive 'renamed-step'
New-Item -ItemType Directory -Path $root -Force | Out-Null
$lockPath = Join-Path $root 'ci-status-net11.lock.yml'
(Get-Content -LiteralPath (Join-Path $PSScriptRoot '../workflows/ci-status-net11.lock.yml') -Raw).Replace(
'Preflight references and publish validated issues',
'Publish issues') | Set-Content -LiteralPath $lockPath
@(Get-CiScanTwin -WorkflowRoot $root).Count | Should -Be 0
{ Get-CiScanPublisherScript -LockPath $lockPath } |
Should -Throw '*no longer contains the publisher step*'
}
}
diff --git a/.github/scripts/Validate-CiScanManifest.Tests.ps1 b/.github/scripts/Validate-CiScanManifest.Tests.ps1
index 6ecc9af921..40af33148b 100644
--- a/.github/scripts/Validate-CiScanManifest.Tests.ps1
+++ b/.github/scripts/Validate-CiScanManifest.Tests.ps1
@@ -314,330 +314,400 @@ Describe 'CI scanner pipeline coverage gate' {
New-TestEvidence -Root $evidenceRoot -LogId 1002
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -SourceLogIds @(1001))
)
{ Test-CiScanManifest `
-Manifest $manifest `
-ExpectedBuilds (New-ExpectedBuilds -MainFailedRecordCount 2 -MainRequiredLogIds @(1001, 1002)) `
-TrustedEvidencePath $evidenceRoot } |
Should -Throw '*missing terminal coverage for trusted log IDs: 1002*'
}
It 'accepts one deduplicated signature that covers multiple trusted logs' {
$evidenceRoot = Join-Path $TestDrive 'dedup-coverage-evidence'
New-TestEvidence -Root $evidenceRoot -LogId 1001
New-TestEvidence -Root $evidenceRoot -LogId 1002
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -SourceLogIds @(1001, 1002) -Body (New-TestBody -MatchCount 4))
)
$plan = Test-CiScanManifest `
-Manifest $manifest `
-ExpectedBuilds (New-ExpectedBuilds -MainFailedRecordCount 2 -MainRequiredLogIds @(1001, 1002)) `
-TrustedEvidencePath $evidenceRoot
$plan.pipelines[0].signatures[0].source_log_ids | Should -Be @(1001, 1002)
}
It 'accepts actual cap exhaustion with explicit remaining-pipeline skips' {
$signatures = for ($i = 1; $i -le 5; $i++) {
$fingerprint = "ci-scan-net11|net11.0|maui-pr|sample test $i|assertion failed|windows"
New-TestSignature -Fingerprint $fingerprint -Body (
New-TestBody -Fingerprint $fingerprint
)
}
$manifest = [pscustomobject]@{
pipelines = @(
(New-TestPipeline -Name 'maui-pr' -DefinitionId 302 -Signatures $signatures)
(New-TestPipeline -Name 'maui-pr-devicetests' -DefinitionId 314 -BuildId 123457 -Signatures @(
(New-TestSignature `
-Pipeline 'maui-pr-devicetests' `
-Fingerprint 'ci-scan-net11|net11.0|maui-pr-devicetests|device sample|assertion failed|android' `
-Disposition 'skipped' `
-SkipReason 'cap-reached')
))
(New-TestPipeline -Name 'maui-pr-uitests' -DefinitionId 313 -BuildId 123458)
)
}
$plan = Test-CiScanManifest `
-Manifest $manifest `
-TrustedEvidencePath (New-DefaultEvidenceRoot)
$plan.filed_count | Should -Be 5
$plan.has_cap_skip | Should -BeTrue
}
It 'rejects cap skips when fewer than five issues are filed' {
$manifest = [pscustomobject]@{
pipelines = @(
(New-TestPipeline -Name 'maui-pr' -DefinitionId 302 -Signatures @(
(New-TestSignature)
))
(New-TestPipeline -Name 'maui-pr-devicetests' -DefinitionId 314 -BuildId 123457 -Signatures @(
(New-TestSignature `
-Pipeline 'maui-pr-devicetests' `
-Fingerprint 'ci-scan-net11|net11.0|maui-pr-devicetests|device sample|assertion failed|android' `
-Disposition 'skipped' `
-SkipReason 'cap-reached')
))
(New-TestPipeline -Name 'maui-pr-uitests' -DefinitionId 313 -BuildId 123458)
)
}
{ Test-CiScanManifest `
-Manifest $manifest `
-TrustedEvidencePath (New-DefaultEvidenceRoot) } |
Should -Throw '*exactly 5 issues are filed*'
}
- It 'rejects a cap skip that appears before five later filed entries' {
+ It 'accepts a cap skip before five later filed entries' {
$earlySkip = New-TestSignature `
-Fingerprint 'ci-scan-net11|net11.0|maui-pr|early sample|assertion failed|windows' `
-Disposition 'skipped' `
-SkipReason 'cap-reached'
$filed = for ($i = 1; $i -le 5; $i++) {
$fingerprint = "ci-scan-net11|net11.0|maui-pr|later sample $i|assertion failed|windows"
New-TestSignature -Fingerprint $fingerprint -Body (
New-TestBody -Fingerprint $fingerprint
)
}
$manifest = New-CompleteManifest -MainSignatures (@($earlySkip) + @($filed))
- { Test-CiScanManifest -Manifest $manifest } |
- Should -Throw '*cannot use cap-reached before exactly 5 issues are filed*'
+ $plan = Test-CiScanManifest `
+ -Manifest $manifest `
+ -TrustedEvidencePath (New-DefaultEvidenceRoot)
+
+ $plan.filed_count | Should -Be 5
+ $plan.has_cap_skip | Should -BeTrue
+ }
+
+ It 'accepts a substantive skip after five filed entries' {
+ $filed = for ($i = 1; $i -le 5; $i++) {
+ $fingerprint = "ci-scan-net11|net11.0|maui-pr|filed sample $i|assertion failed|windows"
+ New-TestSignature -Fingerprint $fingerprint -Body (
+ New-TestBody -Fingerprint $fingerprint
+ )
+ }
+ $substantiveSkip = New-TestSignature `
+ -Fingerprint 'ci-scan-net11|net11.0|maui-pr|known infrastructure failure|assertion failed|windows' `
+ -Disposition 'skipped' `
+ -SkipReason 'infrastructure-noise'
+ $manifest = New-CompleteManifest -MainSignatures (@($filed) + @($substantiveSkip))
+
+ $plan = Test-CiScanManifest `
+ -Manifest $manifest `
+ -TrustedEvidencePath (New-DefaultEvidenceRoot)
+
+ $plan.filed_count | Should -Be 5
+ $plan.has_cap_skip | Should -BeFalse
+ $plan.pipelines[0].signatures[5].skip_reason | Should -BeExactly 'infrastructure-noise'
}
It 'rejects reordered configured pipelines' {
$manifest = New-CompleteManifest
$manifest.pipelines = @(
$manifest.pipelines[1],
$manifest.pipelines[0],
$manifest.pipelines[2]
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*must be maui-pr definition 302 in configured order*'
}
It 'rejects a skipped pipeline that still contains signatures' {
$manifest = New-CompleteManifest
$manifest.pipelines[1].status = 'skipped-no-recent-build'
$manifest.pipelines[1].signatures = @(
(New-TestSignature `
-Pipeline 'maui-pr-devicetests' `
-BuildId 123457 `
-Fingerprint 'ci-scan-net11|net11.0|maui-pr-devicetests|device sample|timeout|android')
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*must have an empty signatures array*'
}
}
Describe 'CI scanner issue payload gate' {
It 'rejects unsafe characters in a fingerprint' {
$fingerprint = 'ci-scan-net11|net11.0|maui-pr|sample "test"|assertion failed|windows'
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Fingerprint $fingerprint)
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*unsafe characters*'
}
It 'canonicalizes the exact production uppercase fingerprint before publication' {
$productionFingerprint = 'ci-scan|main|maui-pr|runoniOS_MauiReleaseTrimFull|ios-simulator-boot-timeout|ios-simulator-64'
$canonicalFingerprint = 'ci-scan|main|maui-pr|runonios_mauireleasetrimfull|ios-simulator-boot-timeout|ios-simulator-64'
$body = (New-TestBody).Replace('- **Branch**: net11.0', '- **Branch**: main')
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Fingerprint $productionFingerprint -Body $body)
)
$evidenceRoot = New-DefaultEvidenceRoot
$plan = Test-CiScanManifest `
-Manifest $manifest `
-ExpectedBuilds (New-ExpectedBuilds) `
-TrustedEvidencePath $evidenceRoot `
-ScannerId 'ci-scan'
$plan.pipelines[0].signatures[0].fingerprint | Should -BeExactly $canonicalFingerprint
$plan.issues[0].Fingerprint | Should -BeExactly $canonicalFingerprint
$plan.issues[0].Body |
Should -Match "(?m)^<!-- ci-scan-fingerprint: $([regex]::Escape($canonicalFingerprint)) -->$"
$plan.issues[0].Body.Contains('runoniOS_MauiReleaseTrimFull') | Should -BeFalse
}
It 'rejects fingerprints that collide after trusted case canonicalization' {
$productionFingerprint = 'ci-scan|main|maui-pr|runoniOS_MauiReleaseTrimFull|ios-simulator-boot-timeout|ios-simulator-64'
$canonicalFingerprint = $productionFingerprint.ToLowerInvariant()
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Fingerprint $productionFingerprint -Disposition 'existing' -IssueNumber 36827),
(New-TestSignature -Fingerprint $canonicalFingerprint -Disposition 'existing' -IssueNumber 36828)
)
{ Test-CiScanManifest -Manifest $manifest -ScannerId 'ci-scan' } |
Should -Throw "*Duplicate fingerprint '$canonicalFingerprint'*"
}
<#
A fingerprint is embedded verbatim in the canonical marker, but the marker is
matched AFTER ConvertTo-SafeIssueBody neutralizes the body. A GitHub issue/PR URL
passes the character-class check (every character is in the allowed set), so
without this gate the URL gets a zero-width space injected, the marker becomes
unmatchable, and the run dies blaming the BODY for a defect in the FINGERPRINT.
CI logs referencing a tracking issue by URL make this reachable in practice.
#>
It 'rejects a fingerprint containing a GitHub issue URL, naming the real cause' {
$fingerprint = 'ci-scan-net11|net11.0|maui-pr|see https://github.com/dotnet/maui/issues/12345|assertion failed|windows'
# Guard the premise: this really does survive the character-class check.
$fingerprint | Should -CMatch '^[a-z0-9][a-z0-9 ._:/+()\-|]*$'
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Fingerprint $fingerprint)
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*rewritten by notification neutralization*'
}
It 'rejects any fingerprint neutralization would rewrite, not just URL shapes' {
# Asserts the round-trip invariant itself, so a future neutralization rule is
# covered without editing Assert-ValidFingerprint.
$fingerprint = 'ci-scan-net11|net11.0|maui-pr|see https://github.com/dotnet/maui/pull/999|assertion failed|windows'
{ Assert-ValidFingerprint `
-Fingerprint $fingerprint `
-PipelineName 'maui-pr' `
-ScannerConfig $script:Net11Config } |
Should -Throw '*rewritten by notification neutralization*'
}
It 'accepts a fingerprint that neutralization leaves untouched' {
$fingerprint = 'ci-scan-net11|net11.0|maui-pr|see github.com/dotnet/maui issue 12345|assertion failed|windows'
{ Assert-ValidFingerprint `
-Fingerprint $fingerprint `
-PipelineName 'maui-pr' `
-ScannerConfig $script:Net11Config } |
Should -Not -Throw
}
It 'rejects a fingerprint for another scanner' {
$fingerprint = 'ci-scan-main|main|maui-pr|sample test|assertion failed|windows'
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Fingerprint $fingerprint)
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*does not match the ci-scan-net11 scanner*'
}
It 'rejects a fingerprint for another pipeline' {
$fingerprint = 'ci-scan-net11|net11.0|maui-pr-uitests|sample test|assertion failed|windows'
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Fingerprint $fingerprint)
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*does not match the ci-scan-net11 scanner*'
}
It 'rejects a fingerprint with the wrong field count' {
$fingerprint = 'ci-scan-net11|net11.0|maui-pr|sample test|assertion failed'
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Fingerprint $fingerprint)
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*exactly six non-empty pipe-delimited fields*'
}
It 'rejects the literal truncation placeholder in a title' {
$manifest = New-CompleteManifest -MainSignatures @(
(New-TestSignature -Title 'Sample failure [Content truncated due to length]')
)
{ Test-CiScanManifest -Manifest $manifest } |
Should -Throw '*forbidden truncation placeholder*'
}
+ It 'canonicalizes the production em-dash title before publication' {
+ $rawTitle = "Recurring Android device test failure $([char]0x2014) StatusBarThemeAppliesWhenHandlerConnects fails on Android CoreCLR and Mono (net11.0)"
+ $manifest = New-CompleteManifest -MainSignatures @(
+ (New-TestSignature -Title $rawTitle)
+ )
+
+ $plan = Test-CiScanManifest `
+ -Manifest $manifest `
+ -TrustedEvidencePath (New-DefaultEvidenceRoot)
+
+ $canonicalTitle = 'Recurring Android device test failure - StatusBarThemeAppliesWhenHandlerConnects fails on Android CoreCLR and Mono (net11.0)'
+ $plan.pipelines[0].signatures[0].title | Should -BeExactly $canonicalTitle
+ $plan.issues[0].Title | Should -BeExactly "[ci-scan-net11] $canonicalTitle"
+ }
+
+ It 'also canonicalizes an en-dash title separator' {
+ $rawTitle = "Recurring sample failure $([char]0x20
... [truncated]The diff was truncated to fit GitHub's review body limit.
This was referenced Aug 6, 2026
kubaflo
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Root Cause
The first post-merge dry runs after #36958 exposed two independent agent-manifest composition failures at
16671cadb63739c47da264953ee3e1267806cbb8:XHarness exit code: 1 (TESTS_FAILED)for the CarouselView leak signature but omitted that duplicated value from its issue body. The trusted validator correctly rejected the mismatch.Replaying the net11 artifact also exposed two bodies that paraphrased rather than copied an entire trusted evidence line, plus an order-dependent
cap-reachedclassification. Prompt compliance alone is not a reliable publication boundary for duplicated or normalized payload fields.Description of Change
-in issue titles before the existing printable-ASCII gate. Curly quotes, non-breaking spaces, emoji, controls, and all other non-ASCII remain rejected.match_patternagainst frozen same-run evidence before use. If an otherwise valid body omitted that exact pattern, append a bounded, safely renderedTrusted Match Patternexcerpt in trusted PowerShell. Hidden/control/marker content still fails closed.cap-reachedmay appear before or after the fifth filed entry, while substantive skip reasons remain valid regardless of position.What NOT to Do
Validation
.github/scriptsPester suite: 1,572 passed / 0 failed, 22 healthy containers, NUnit XML total parity.Safety and Rollout
Both production runs failed before staged publication, and every local replay stopped at trusted plan generation. No real
ci-scanorci-scan-net11issue was created, edited, labeled, closed, or reopened. No new production dry runs will be dispatched until this PR merges.Issues Fixed
N/A