Fix deterministic CI scan fingerprint publication - #36913
Conversation
GitHub Actions
Inactive
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 36913Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 36913" |
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
GitHub Actions
Inactive
GitHub Actions
Failure
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
There was a problem hiding this comment.
Pull request overview
This pull request hardens CI-scan issue publication by making fingerprint + match-count markers publisher-owned and deterministic, removing any reliance on agent prompt output for those markers. It also aligns the ci-status-main and ci-status-net11 workflows onto the same “frozen evidence + trusted validator + all-or-nothing publisher” architecture and adds extensive twin/mutation test coverage to keep the security controls from regressing.
Changes:
- Move canonical fingerprint and match-count marker injection/verification into the trusted validator + publisher boundary, and fail closed on any marker-like agent body content.
- Make
ci-status-mainuse the same deterministic publisher flow as net11 (custom safe-output job + trusted validator + preflight + immutable publisher checkout). - Add twin-aware Pester suites and mutation tests to enforce symmetry between the two scanner workflows and prove the controls are load-bearing.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/ci-status-main.md | Reworks main scanner to use the same trusted “submit-ci-scan” publisher architecture, including dry-run support and frozen evidence flow. |
| .github/workflows/ci-status-net11.md | Wires in trusted scanner identity env + adds publisher-side canonical marker validation and updates agent instructions to be marker-free. |
| .github/workflows/ci-status-net11.lock.yml | Regenerated compiled workflow with updated gh-aw tooling and the publisher-side marker validation logic wired into the lock. |
| .github/scripts/Validate-CiScanManifest.ps1 | Centralizes twin configuration, rejects marker-like content, recomputes match counts from frozen evidence, injects canonical markers, and validates the post-injection payload. |
| .github/scripts/Validate-CiScanManifest.Tests.ps1 | Updates/extends manifest tests for publisher-owned markers, evidence-required match counts, and twin configuration behavior. |
| .github/scripts/Validate-CiScanPublisher.Tests.ps1 | Adds twin discovery + anti-divergence checks and executes extracted compiled publisher code to validate boundary invariants. |
| .github/scripts/CiScanTwins.Helpers.ps1 | Adds helper functions to discover scanner twins and extract the compiled publisher script block for test execution. |
| .github/scripts/CiScanTrustedInventory.Tests.ps1 | Extends trusted inventory collector tests to run across both twins and validate failed-leaf coverage behavior. |
| .github/scripts/CiScanMutation.Tests.ps1 | Adds mutation testing to prove marker injection/rejection and twin discovery invariants fail closed when controls are removed. |
| .github/aw/actions-lock.json | Updates the pinned gh-aw setup action version used by agentic workflow compilation. |
Copilot's findings
- Files reviewed: 10/11 changed files
- Comments generated: 0
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
.github/workflows/ci-status-main.md:58
- The safe-outputs comment references gh-aw v0.82.14, but this workflow is now compiled with v0.83.4 (see ci-status-main.lock.yml metadata). Keeping a hard-coded old version here is misleading; either update the version or make the note version-agnostic.
# gh-aw v0.82.14 does not propagate staged mode into custom safe-output jobs.
# Keep this expression identical to GH_AW_SAFE_OUTPUTS_STAGED below; tests enforce it.
staged: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == true }}
- Files reviewed: 10/11 changed files
- Comments generated: 0 new
kubaflo
left a comment
kubaflo
left a comment
There was a problem hiding this comment.
Adversarial security review (round 1) — Opus 4.8 + GPT-5.6 Sol, verified against 9636664. This is a solid, well-architected trust-boundary fix: moving marker injection into the deterministic publisher (agent can no longer forge ci-scan-fingerprint/match-count markers) with a complete-manifest, frozen-evidence, all-or-nothing gate is the right design, and marker forgery is genuinely closed. Two residual agent-gaming vectors remain, though — both let an adversarial/prompt-injected agent suppress a real CI failure, which is the exact threat model this PR exists for.
❌ Match-count/coverage bypass via the universal evidence header (Medium)
ci-status-main.md:595 prepends a synthetic prefix to every frozen log: evidence = ['===== AzDO log ${buildId}/${logId} =====', azdoLog]. Validate-CiScanManifest.ps1:400 counts coverage with a substring test ($line.Contains($MatchPattern)) over that evidence — including the synthetic line. So an agent can submit a signature with match_pattern: "===== AzDO log" (14 chars, ≥8) and the required log IDs; it matches every file, granting "coverage" without any real failure-content proof — letting failed leaves be marked covered so the scan files nothing. (The 9636664 terminality hardening doesn't touch this.) Fix (Sol's): exclude the synthetic header from countable evidence and require match_pattern to come from raw failure content.
❌ match_pattern isn't marker-guarded → unrelated-issue replay (Medium)
Test-MarkerLikeContent is applied only to the agent rawBody (Validate-CiScanManifest.ps1:488), not to Get-ValidatedMatchPattern (:361, which checks only length + zero-width). Every ci-scan tracking issue body contains <!-- ci-scan-fingerprint: … -->, so if any scanned log line contains ci-scan-fingerprint, an agent can use it as match_pattern (passes the in-log check at :400) and copy an existing same-pipeline issue's full fingerprint + number (fingerprint binds only parts 0–2 = scanner/branch/pipeline at :338; identity/error/platform are agent-selected). The dedup path then finds the pattern in that issue's body (ci-status-main.md:268) and the exact marker (:271) → the current, unrelated failure is bound to it as canonical-fingerprint and suppressed. Fix (Sol's): apply the marker guard to match_pattern, ignore trusted marker lines during the recurrence body-search, and derive fingerprint identity from trusted evidence rather than agent-selected fields.
ℹ️ Lower-severity (Opus)
Test-MarkerLikeContentfolds to bare alphanumerics globally, so benign prose like "See ci scan fingerprint below" is rejected (verified) → aborts the all-or-nothing batch. Fail-closed (no spoofing) and real CI excerpts rarely trigger it, but a word-boundary check on the hyphenated token would reduce false batch-aborts.- The legacy (marker-less) dedup path adopts an already-
ci-scan-labeled issue on agent-chosen fingerprint identity/error tokens — bounded (needs a pre-labeled issue + exact-pipeline evidence) and transitional; the go-forward canonical path is unaffected.
Verdict: NEEDS_CHANGES — the marker-forgery hole is closed, but the two coverage/dedup vectors above still let an untrusted agent hide a real failure. Both have narrow, concrete fixes.
🤖 Adversarial ensemble — Opus 4.8 (READY, LOW-only) + GPT-5.6 Sol (found both Medium agent-gaming bypasses); mechanisms confirmed by reading the publisher + workflow at 9636664.
|
Addressed the adversarial review in
Additional producer/consumer caps are aligned at 200 raw segments, 25 MB structured evidence, and 200 distinct matching lines. Validation: strict compilation passed for both twins; focused scanner suites passed 229/229; full |
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (2)
.github/workflows/ci-status-main.md:45
- With
cancel-in-progress: false, multiple scanner runs can overlap and race between listing open tracking issues and creating new ones, which can still produce duplicate issues if two runs publish the same fingerprint concurrently. Keeping cancellation enabled preserves the single-writer invariant and reduces the chance of duplicate tracking issues.
cancel-in-progress: false
.github/workflows/ci-status-main.md:684
- The Helix terminality check no longer requires
Waiting/Runningto be zero. If Helix reports a transient inconsistency wheredetails.Finishedis present but there are still waiting/running work items (or those work items are omitted from the returned list), this could accept incomplete evidence and publish based on a partial job snapshot. Consider restoring a fail-closed check for waiting/running counts while still allowing cumulativeUnscheduledto remain non-zero.
terminalJob =
validCounts &&
Boolean(details?.Finished) &&
finishedCount > 0 &&
items.length >= finishedCount &&
- Files reviewed: 10/11 changed files
- Comments generated: 0 new
Inject canonical scanner fingerprints and trusted match counts in the deterministic publisher for both main and net11 scanners. Add fail-closed twin validation and mutation coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dbefc79-f6c9-422e-90c7-c94bb06d39f9
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dbefc79-f6c9-422e-90c7-c94bb06d39f9
Exclude synthetic provenance from evidence counts and bind canonical and legacy recurrence to publisher-derived raw evidence keys. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dbefc79-f6c9-422e-90c7-c94bb06d39f9
kubaflo
left a comment
kubaflo
left a comment
There was a problem hiding this comment.
Round 2 — trusted-evidence binding: 2 prior bypasses closed, 1 new residual
Re-reviewed at c8d17df with an independent adversarial pass (Opus + GPT‑5.6‑sol), then reproduced every claim against the validator myself.
✅ Both round‑1 MEDIUMs are fixed — nice rework
- Universal synthetic header (
===== AzDO log … =====asmatch_pattern): the countable/hashable corpus is now the separate*.evidence.json(ci-status-main.mdcollector:633‑637,:783‑800), and the validator counts/hashes only from it (Validate-CiScanManifest.ps1:560‑574). The synthetic=====headers live only in the human‑readable.log, so they no longer satisfy coverage. ThematchingLineHashes.Count -gt 200guard (:566‑568) additionally fail‑closes ultra‑generic patterns. Confirmed rejected. match_patternnot marker‑guarded / fingerprint replay:Get-ValidatedMatchPatternnow runsTest-MarkerLikeContent(:451‑453), the marker fold coversciscanevidencekey+ homoglyphs (:223‑272), and binding was upgraded from an 8‑char substring to a full normalized evidence‑line hash on both sides (:364‑384). Confirmed rejected (marker / homoglyph / evidence‑key variants all throw).
Also verified fail‑closed on malformed/missing/empty/oversized/null‑segment trusted evidence, the []→$null pipeline trap, ##[…] log‑command injection (ConvertTo-SafeLogValue), and pre‑agent TOCTOU (evidence uploaded at the lock step, publisher is a separate job).
🟡 New residual (MEDIUM) — publisher‑manufactured constant text becomes global dedup identity
The helix-deadletter-uri segment records a fixed placeholder URL as its entire evidence content:
// ci-status-main.md:748-751 (a deadletter's ConsoleOutputUri is the fixed
// https://dotnet.github.io/core-eng/helix-workitem-deadletter.txt placeholder)
rawSegments.push({ kind: 'helix-deadletter-uri',
source: `${jobId}/${workItem.Name}`, // run-specific…
content: deadletterUrl.toString() }); // …but only THIS is hashedGet-TrustedEvidenceMatchProof derives EvidenceKey/EvidenceLineHashes solely from segment.content (Validate-CiScanManifest.ps1:560, :581‑589) — the run‑specific source is never mixed in — and helix-deadletter-uri is an accepted identity kind (:517). So when the current build has any deadletter (which also forces the log into failed_leaf_log_ids, so a presence proof is required), an agent can pick match_pattern = 'helix-workitem-deadletter.txt' and the whole trusted‑evidence identity collapses to a global constant across every build, pipeline and run.
Independently reproduced at c8d17df (two unrelated failures — android‑emulator‑boot in build 900001 vs ios‑device‑lost in build 900002 — sharing only the constant URL):
build900001 key: sha256:eb6fecb501bb9f9a5a6bc7c327a598442300b8d3f8943a43e833719a5c2cb28c
build900002 key: sha256:eb6fecb501bb9f9a5a6bc7c327a598442300b8d3f8943a43e833719a5c2cb28c
IDENTICAL KEY ACROSS UNRELATED BUILDS: True
control (real run-specific line): keys differ = True
That constant key is what every recurrence/dedup consumer is keyed on (hasTrustedEvidenceLine ci-status-main.md:188‑198; existing preflight :304‑319; legacyEvidenceMatcher :270‑277), so the "bind dedup to trusted evidence" invariant degrades on this path to "some deadletter happened at some point":
- a stale canonical issue from an unrelated old build satisfies the current build's recurrence proof, and
- a genuinely new device‑test failure that routes through the deadletter path is silently adopted onto that unrelated issue (
legacy_dedup:true), so nothing is filed and the run reports complete coverage. This is exactly the mutationValidate-CiScanPublisher.Tests.ps1:385("pipeline alone suppresses a distinct failure") is meant to kill — the guard only holds while every evidence line is run‑specific, whichhelix-deadletter-uriis not.
Suggested fix (either):
- Exclude
helix-deadletter-urifrom the identity computation — still count it for coverage, but drop it frommatchingLineHashes/EvidenceKeyso a deadletter can't be the sole dedup identity; or - Make the segment content run‑specific, e.g.
content: \${jobId}/${workItem.Name} ${deadletterUrl}``, so the hashed identity is unique per build/job/work item.
Add a regression test that a deadletter‑only match_pattern cannot adopt an issue from a different build.
No other exploitable bypass was found in the +1259‑line rework.
Multi‑model adversarial review (Claude Opus 4.8 + GPT‑5.6‑sol); every finding independently reproduced against the validator before posting.
Bind Helix deadletter recurrence to stable work-item identity, reject non-terminal active work items, and align the merged reconciler invariants with publisher-owned fingerprint markers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dbefc79-f6c9-422e-90c7-c94bb06d39f9
c8d17df to
a535126
Compare
|
Addressed Round 2 and the two suppressed follow-up notes in
Validation on the rebased head:
No real |
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
.github/scripts/Validate-CiScanPublisher.Tests.ps1:62
- The failure message still refers to
legacyIdentityMatcher, but this helper now extractslegacyEvidenceMatcher+ evidence proof helpers. If this path fails, the current message will mislead debugging.
$helperEnd = $lock.IndexOf('// The plan is produced', $start)
$matcherStart = $lock.IndexOf('const legacyEvidenceMatcher', $helperEnd)
$end = $lock.IndexOf('const existingEntries', $matcherStart)
if ($helperEnd -lt 0 -or $matcherStart -lt 0 -or $end -lt 0) {
throw 'Could not find the end of the legacyIdentityMatcher block.'
}
- Files reviewed: 12/13 changed files
- Comments generated: 0 new
Normalize trusted AzDO transport timestamps and prevent markerless legacy issues from suppressing distinct failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dbefc79-f6c9-422e-90c7-c94bb06d39f9
|
Addressed the latest adversarial findings in
Canonical fingerprint drift continues to fall through to a bounded visible duplicate, contradictory canonical markers still fail closed, and the conservative marker-prose rejection remains unchanged. Validation on the pushed head:
No real |
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (2)
.github/workflows/ci-status-main.md:984
- This section says the publisher injects two hidden markers and only calls out
ci-scan-fingerprint/ci-scan-match-count, but the publisher/validator now also injects and enforcesci-scan-evidence-key. The prompt should reflect the third marker, and the “no marker content” rule should includeci-scan-evidence-keyso agent guidance matches the enforced contract.
The publisher injects two hidden HTML-comment markers at the top of every issue
it files: one carrying the fingerprint (taken from the validated manifest, not
from your body) and one carrying the match count (recomputed from the frozen
evidence, not from anything you report).
.github/workflows/ci-status-net11.md:984
- This section says the publisher injects two hidden markers and only calls out
ci-scan-fingerprint/ci-scan-match-count, but the publisher/validator now also injects and enforcesci-scan-evidence-key. The prompt should reflect the third marker, and the “no marker content” rule should includeci-scan-evidence-keyso agent guidance matches the enforced contract.
The publisher injects two hidden HTML-comment markers at the top of every issue
it files: one carrying the fingerprint (taken from the validated manifest, not
from your body) and one carrying the match count (recomputed from the frozen
evidence, not from anything you report).
- Files reviewed: 11/13 changed files
- Comments generated: 0 new
kubaflo
left a comment
kubaflo
left a comment
There was a problem hiding this comment.
Round 4 — LGTM ✅ (approving)
Followed this one across four rounds of adversarial review (Claude Opus 4.8 + GPT‑5.6‑sol), independently reproducing every finding and every fix against the validator. Everything raised is now resolved:
| Round | Finding | Status |
|---|---|---|
| r1 | Universal synthetic ===== AzDO log header usable as match_pattern (covers any signature) |
✅ fixed — countable corpus moved to *.evidence.json, synthetic headers excluded |
| r1 | match_pattern not marker‑guarded → fingerprint replay |
✅ fixed — Test-MarkerLikeContent + full‑line‑hash binding |
| r2 | helix-deadletter-uri content was a constant placeholder URL → global‑constant EvidenceKey → stale‑issue replay + silent suppression |
✅ fixed at a535126 |
| r3 (both models READY) / r4 | Recurrence‑identity hardening | ✅ clean |
Deadletter fix — independently reproduced at 133746fde
The segment content is now Helix work item ${workItemName} was deadlettered: … and the validator hashes the whole matching line, so identity is work‑item‑specific (empty/CRLF/>1000 names throw — no 'unknown' fallback). My repro:
distinct work items → DIFFERENT keys (e8e7499… vs 2f8b054…) ✅ collision closed
same work item / diff build → SAME key ✅ legitimate recurrence preserved
New 133746fde hardening — verified clean
StripAzdoTransportTimestamp(azdo‑log only): strips AzDO's run‑specific leading UTC timestamp so the same failure dedupes across builds. Confirmed it does not collapse distinct messages (same‑msg/diff‑ts → same identity; diff‑msg → different identity), retains the timestamp on the deadletter/helix path, and doesn't produce an empty identity. Sol's focused re‑verify (112 tool calls) came backdelta_clean: true, no findings.- Markerless legacy‑issue rejection tightens the exact adoption path the r2 exploit abused — legacy issues without publisher‑owned identity can no longer be adopted as recurrence evidence.
- Twin parity (
ci-status-net11.md) and lock‑file/JS integrity hold; the reconciler changes are comment‑only.
CI
Pester (.github/scripts) green (408 tests, 0 fail), license/cla green; maui-pr correctly skips for a .github‑only change.
Nicely done working through a genuinely subtle trust‑boundary. Two optional, non‑blocking follow‑ups for later: a dedicated unit test for the empty/CRLF/>1000 work‑item‑name guard, and the collector‑1000 vs validator‑source‑1000 length‑cap alignment (currently fail‑closed, so safe).
LGTM.
Multi‑model adversarial review (Claude Opus 4.8 + GPT‑5.6‑sol) across r1–r4; deadletter key‑collision fix and timestamp‑strip behavior reproduced with pwsh before approving.
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Description of Change Fixes two production dry-run failures discovered after #36913: - [main dry run 30574474829](https://github.com/dotnet/maui/actions/runs/30574474829) rejected `runoniOS_MauiReleaseTrimFull` because trusted validation depended on the agent lowercasing every fingerprint component. - [net11 dry run 30574476689](https://github.com/dotnet/maui/actions/runs/30574476689) failed while parsing a manifest serialized as JSON inside the `submit_ci_scan` string argument when multiline issue bodies contained literal newlines. The trusted PowerShell boundary now canonicalizes accepted ASCII fingerprint casing before structural validation, duplicate detection, marker injection, and plan publication. Unsafe characters still fail closed, and fingerprints that collide after canonicalization are rejected. The scanner twins now use a same-run artifact handoff instead of nested string transport. The agent writes one manifest to `/tmp/gh-aw/agent/manifest_final.json`, then calls the argument-free `submit_ci_scan` tool exactly once to authorize publication. gh-aw v0.83.4 uploads that fixed directory in the current run's fixed `agent` artifact; the custom safe job downloads that artifact and reads only `${{ runner.temp }}/gh-aw/safe-jobs/agent/manifest_final.json`. The file is treated as untrusted, must be a regular non-symlink file, and is capped at 500,000 bytes before all existing inventory, evidence, cap, staged-mode, all-or-nothing preflight, and post-write checks run. This artifact is only an intra-run handoff. It is not cross-run scanner state or persistence. The safe-output request cannot select another file path or carry manifest data. This PR intentionally does not add stale issue lifecycle or reset behavior. ### Review hardening Adversarial review found and closed additional gaps while preserving twin symmetry: - **Manifest bypassed sanitization and threat detection.** Both twins now stage the fixed manifest into the threat-detection directory, name it in the detection prompt, and fail closed if submission was authorized without it. The staging gate rejects symlinks/non-regular files, enforces `0 < size ≤ 500000` before copy, uses `cp --no-dereference`, and revalidates the staged file type and byte size. - **Exact-once gates ignored `.errors`.** Both the post-steps jq gate and `Assert-ScannerSubmissionFromAgentOutput` now fail closed on non-empty collector errors, and `max: 1` rejects duplicate custom safe-output calls. - **Hidden content could bypass scalar validation.** The trusted gate now walks Unicode scalar values, rejects unpaired surrogates, C0/DEL/C1 controls, bidi/invisible format characters, noncharacters, HTML comments, the full default-ignorable tag/variation-supplement plane, and reserved default-ignorable Specials. It rejects rather than strips so hash-verified evidence remains intact. - **Legitimate emoji presentation was over-blocked.** VS15/VS16 now pass only after an exact 16-base allowlist of common CI status/callout symbols. The allowlist includes the production `🛠️ Build Microsoft.Maui.sln` heading while selectors after ordinary text or arbitrary symbols still fail closed, so the fix does not reopen a generic variation-selector steganography channel. - **Threat detection contradicted the trusted selector rule.** Both scanner prompts now use that same exact 16-base allowlist and explicitly retain detection for isolated selectors or selectors after any other base. Tests parse the validator, both workflow sources, and both compiled runtime `CUSTOM_PROMPT` values so an omitted compile cannot silently restore the veto. - **A manifest test file was silently disabled.** A missing Pester `It` opener is restored, so `Validate-CiScanManifest.Tests.ps1` is discovered and executed again. - **Parameterized hidden-content tests broke NUnit XML export.** Their names now use the safe ASCII `<Case>` parameter instead of allowing raw XML-invalid values such as ESC U+001B into the NUnit test-name attribute. ### Validation - Focused scanner Pester suites with NUnit XML export and report-total parity guard: 302 passed - Full `.github/scripts` Pester 5.9.0 suite with NUnit XML export and report-total parity guard: 1,566 passed - `gh aw compile ci-status-main --strict` - `gh aw compile ci-status-net11 --strict` - Repeated strict compile produced byte-identical lock hashes - Replayed the exact main production artifact through full trusted validation; parsed the exact net11 `manifest_final.json` with all seven multiline bodies through the fixed-file reader - Replayed the exact production `🛠️ Build Microsoft.Maui.sln` text through trusted manifest validation while preserving arbitrary-symbol selector rejection - Mutation coverage rejects restoring nested manifest-string transport, agent-selected paths, unbounded/symlink threat-detection staging, disabled hidden-content rejection, removed or widened selector carve-outs, lost disallowed-selector rejection, and stale compiled selector policy - Final three-reviewer adversarial confirmation found no remaining blocking, warning, or actionable findings No real `ci-scan` or `ci-scan-net11` issue was created, updated, closed, reopened, or otherwise mutated during development or validation. ### Issues Fixed Follow-up to #36913; no separate tracking issue. --------- Co-authored-by: PureWeen <223556219+Copilot@users.noreply.github.com> Co-authored-by: Vally Fixture <vally-fixture@example.invalid> Copilot-Session: 41235a9a-79b4-433f-9e0d-7278c916a7c9
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Description of Change
ci-status-mainandci-status-net11azdo-logevidence, while preserving timestamps in non-AzDO failure messages(ID N), or live(definition N)syntax using the trusted configured pipeline definitionmatch_patternvariants; revalidate exact post-injection payloads at the GitHub write boundaryRoot cause
PR #36848 added fail-closed manifest validation to the net11 scanner and exposed a pre-existing repo-wide publication defect. In run 30413273824, the agent job succeeded, but
submit_ci_scanfailed before any issue write because the compiled prompt did not contain the authored HTML-comment marker template.Artifact
agent(8709769921) contained 16/16 signatures with zero fingerprint-marker-prefix and zero canonical-marker matches. gh-aw strips literal HTML comments while compiling the authored prompt, so regenerating the lock or strengthening prompt prose cannot make agent-side marker emission reliable. Output-side safe-output stripping is not needed to explain this incident.Main had the same silent blast radius: sampled issues #36858, #36779, #36709, and #36689 carry no fingerprint marker, but its permissive publisher did not validate the payload. Net11's all-or-nothing gate correctly prevented every write, so the first post-merge run published zero issues.
Architecture
The shared trusted validator resolves a hard-coded scanner configuration for
ci-scan|mainorci-scan-net11|net11.0. For each filed manifest entry it:match_patternvariants, including spacing, case, zero-width, separator, HTML-comment-like, and Unicode-homoglyph evasions.evidence.jsonraw segments; rendered.logfiles retain AzDO/Helix provenance for diagnosis, but synthetic headers are not countable evidenceAzDO's log API prepends a different UTC transport timestamp to each stored line on every build. PowerShell strips that prefix only when structured provenance says the segment is
azdo-log; Helix and other message timestamps remain identity-bearing. At the write boundary, publisher body matching computes both raw and AzDO-normalized candidates against the trusted plan hash. The same failure therefore keeps its evidence identity across builds while real non-AzDO timestamps remain distinct.Both compiled publisher jobs bind the plan to their trusted scanner ID, branch, and label; preflight every issue/reference before any mutation; preserve canonical marker retry/dedup; and revalidate GitHub's stored response. Canonical recurrence requires the exact fingerprint and evidence-key markers plus a current trusted evidence line.
Markerless legacy issues no longer provide authoritative coverage. Their exact pipeline/evidence shape is still recognized for a precise migration error, including no suffix,
(ID N), and the live(definition N)suffix with the correct configured definition. An explicit markerlessexistingreference aborts before any write, and afiledpayload never auto-adopts a markerless issue. It instead creates bounded visible canonical coverage. This is intentionally safer than silently merging two same-pipeline failures that share boilerplate such asBuild FAILED.The frozen evidence collector treats a Helix job as complete only when the job has a terminal
Finishedvalue,WaitingandRunningare zero, and every returned work item is terminal with valid completion evidence. Helix's cumulativeUnscheduledcounter may remain nonzero after completion and is validated but not treated as active work. AzDO build records with missing or invalidfinishTimefail closed. Structured evidence enforces matching producer/consumer caps of 200 segments, 25 MB, and 200 distinct matching lines.A deadletter placeholder URL contains no run-specific diagnostics and is constant across Helix. The countable evidence line includes the validated stable work-item name plus that URL. This distinguishes unrelated work items while deliberately excluding job/build IDs so recurrence for the same work item remains stable across builds. Deadletters still mark their AzDO submission log as a failed leaf, so absence-only coverage remains forbidden.
The branch is based on current
mainafter PR #36850. Its report-only reconciler asserts that both scanner twins compile trusted validation before publisher-side exact-marker checks, rather than expecting an agent marker template. The reconciler still has no production state-marker writer, so stale-issue closure candidates remain unreachable.Review findings resolved
azdo-logtransport timestamps are removed symmetrically from PowerShell proof generation and JavaScript body matching.(definition N)suffix: confirmed; exact no-suffix,(ID N), and(definition N)forms are recognized for all three configured pipelines and both twins, and a wrong definition is rejected.WaitingandRunningwhile allowing cumulativeUnscheduled.cancel-in-progress: falsepreserves the active publisher instead of allowing overlap.Tests
gh aw compilefor both twins: 0 errors, 0 warnings.github/scriptsPester: 1489/1489 passed(definition N)support, re-enabled markerless explicit coverage, re-enabled markerless auto-adoption, removed injection, untrusted fingerprint/count sourcing, pre-injection-only validation, duplicate rejection removal, synthetic framing, marker-pattern rejection, trusted-state recurrence, evidence-identity removal, constant deadletter identity, omitted twins, and empty discoveryThere is no scanner-specific gh-aw behavioral eval runner in this repository, so deterministic Pester, lock-extracted Node execution, strict compilation, and static anti-vacuity invariants provide behavioral regression coverage.
Residual risk
Disabling markerless adoption can produce a bounded visible duplicate for a legacy issue until canonical coverage exists. This is intentional: without a publisher-owned historical identity, silently reusing a markerless issue is not a trustworthy dedup decision. Conservative marker-content and evidence-size gates may also fail an entire scan rather than truncate or publish partial evidence. These behaviors fail closed and produce zero partial writes.
No real
ci-scanorci-scan-net11issue was mutated during development or validation.Issues Fixed
No scanner tracking issue is closed by this infrastructure correction. Related incident: PR #36848 and Actions run 30413273824.