Fix main CI failure attribution and classification

[Copilot]

What broke

Failed push runs on main could be blamed on the pull request that happened to trigger the build, even when the failure came from an earlier merge or the combined repository state.

The motivating failure was a semantic merge conflict on main:

error CS0117: 'AzureBicepResourceScope' does not contain a definition for 'ForSubscription'
error CS0117: 'AzureBicepResourceScope' does not contain a definition for 'ForTenant'

PR #19090 triggered the build, but the break came from the interaction between the earlier merges of #19084 and #18976.

Root cause

The workflow inferred attribution from associated pull requests instead of the failed run's immutable metadata. It also allowed incomplete history, ambiguous associations, and partial test evidence to reach downstream actions.

How this fixes it

The analyzer now derives run identity and candidate attribution from trusted GitHub metadata. Pull-request runs require one unambiguous subject PR; failed main runs consider every merge since the last successful run and withhold attribution when history is incomplete.

Agent output remains a proposal. Before any side effect, deterministic validation requires exact run and failed-job identity, complete test evidence with exact unique {test, job} pairs, compatible causes, and a valid rerun request. The operational contract is documented in docs/ci/analyze-ci-failure.md.

Important callouts

• Candidate comparison and last-successful-run history fail closed when GitHub returns malformed cardinality metadata or incomplete records.
• GitHub's artifact API does not identify the producer job. Test provenance follows the tests.yml and run-tests.yml naming contract and remains unavailable when expected artifacts are missing, ambiguous, malformed, incomplete, or contain no TRX files.
• PR comments and pull-request reruns require an unambiguous PR that is still open and unlocked immediately before the mutation. Run-scoped persistence can continue without an actionable PR.
• Missing test artifacts can block analysis and reruns for early infrastructure failures; this intentionally prefers withheld attribution over an unsupported empty result.

Testing

AnalyzeCiFailureWorkflowTests contains 243 passing tests covering attribution, evidence completeness, exact test/job binding, cause identity, publication, persistence, and rerun gates.

dotnet test --project tests/Infrastructure.Tests/Infrastructure.Tests.csproj --no-launch-profile -- --filter-class "*.AnalyzeCiFailureWorkflowTests" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
gh aw compile analyze-ci-failure --validate --actionlint --shellcheck

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • Threat model/security review completed
    • No formal security review

Fixes #19454

[Copilot]

Copilot wasn't able to review any files in this pull request.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

.github/workflows/analyze-ci-failure.md:169

• The main-run context records only each candidate PR's title/URL and then deletes the comparison payload. Because the agent is explicitly barred from querying GitHub, it receives no changed-file or patch data for these merges and cannot verify the semantic conflicts this change is intended to diagnose. Preserve and render the comparison's changed files/patches (or collect equivalent per-PR change data) so attribution can be based on code evidence rather than titles.

                    '. + [$commit + {pull_request: {
                      number: $pr.number,
                      title: $pr.title,
                      url: $pr.html_url,
                      merged_at: $pr.merged_at
                    }}]' ci-failure-data/candidate-merges.json \

.github/workflows/analyze-ci-failure.md:746

• For a main run this row contains main, but reused cause issues created by the old workflow still have a PR column header. The existing-issue path only appends this row and never migrates that header, so stable infra/flaky issues will misleadingly display main as a PR value. Rewrite the legacy header to Context before appending a main occurrence.

              if [ "$RUN_SCOPE" = "main" ]; then
                OCCURRENCE_CONTEXT="main"
              else
                OCCURRENCE_CONTEXT="#${PR_NUMBER}"
              fi
              NEW_OCCURRENCE_ROW="| ${OCC_DATE} | [${RUN_ID}](${RUN_URL}) | ${FIRST_JOB} | ${OCCURRENCE_CONTEXT} |"

.github/workflows/analyze-ci-failure.md:1189

• The schema now permits main-repository-breakage for jobs, but deterministic failed tests on main still have only flaky | code-issue available in both the JSON example and failed_tests[].classification details. That forces a main test regression to be mislabeled as a PR code issue or omitted. Add main-repository-breakage to the failed-test classification contract and example as well.

- `failed_jobs[].classification`: Per-job classification — one of `"transient-infra"`, `"flaky-test"`, `"code-issue"`, or `"main-repository-breakage"`.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19455

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19455"

[Copilot]

Pull request overview

Copilot reviewed 147 out of 1179 changed files in this pull request and generated 2 comments.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Suppressed comments (2)

.github/workflows/analyze-ci-failure.md:783

• TRUSTED_PR_NUMBERS is used for the final PR comment, but the cause persistence and issue-occurrence paths below still read .pr.number from the agent-authored analysis (lines 836 and 878). A mismatched agent PR number can therefore still be written to the memory branch and attributed in a cause issue, which leaves the trust-boundary problem this change is intended to close. Derive all occurrence PR metadata from this trusted value instead.

            TRUSTED_PR_NUMBERS=$(jq -r '.pr_numbers' "$RUN_CONTEXT_FILE")

.github/workflows/analyze-ci-failure.md:990

• The main-breakage issue still publishes agent-authored main_context and triggering_merge_pr values without comparing them to the downloaded trusted artifacts. An incorrect analysis can therefore put the wrong failed SHA or triggering PR into the privileged issue side effect. Read these fields directly from run-context.json, last-successful-main-run.json, and triggering-merge-pr.json.

                    LAST_SUCCESSFUL_SHA=$(jq -r '.main_context.last_successful_main_sha // "unknown"' "$ANALYSIS_FILE")
                    FAILED_SHA=$(jq -r '.main_context.failed_sha // "unknown"' "$ANALYSIS_FILE")
                    TRIGGERING_MERGE=$(jq -r 'if .triggering_merge_pr then "#\(.triggering_merge_pr.number) \(.triggering_merge_pr.title)" else "Not found" end' "$ANALYSIS_FILE")

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[Copilot]

🔵 Needs a closer look

Security-sensitive workflow attribution and publication changes require final human review.

Review details

• Files reviewed: 9/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Unresolved attribution, pagination, and workflow-command injection issues must be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

.github/workflows/analyze-ci-failure-validation.sh:170

• A cause filename is agent-controlled, but it is interpolated into a workflow-command line before the slug check at line 206. A filename containing a newline (for example, bad\n::warning::...json) emits a second runner command during validation. Validate the basename before any diagnostic uses it (and format the rejection with %q) so malformed agent output cannot forge Actions annotations.

  for CAUSE_FILE in "${CAUSE_FILES[@]}"; do
    if ! jq empty "$CAUSE_FILE" 2>/dev/null; then
      echo "::error::Invalid JSON in cause file: $(basename "$CAUSE_FILE")"

.github/workflows/analyze-ci-failure-validation.sh:219

• CAUSE_TYPE is also agent-controlled and is printed raw inside an Actions workflow command on the rejection path. A value containing a newline can forge a following runner command even though validation ultimately fails. Render the rejected value with a single-line-safe representation such as Bash %q.

        echo "::error::Cause ${CAUSE_BASENAME} type '${CAUSE_TYPE}' is not permitted for run scope ${TRUSTED_RUN_SCOPE}"

.github/workflows/analyze-ci-failure-validation.sh:255

• Prior-cause records are explicitly treated as untrusted historical data, but their raw type is interpolated into a runner command here. A malformed stored value containing a newline can inject a second workflow command before this fail-closed exit. Avoid including the raw value in ::error:: output or escape it first.

      PRIOR_CAUSE_TYPE=$(jq -r '.type // ""' "$PRIOR_CAUSE_FILE")
      if [ "$PRIOR_CAUSE_TYPE" != "$CAUSE_TYPE" ]; then
        echo "::error::Cause ${CAUSE_BASENAME} cannot change type from '${PRIOR_CAUSE_TYPE}' to '${CAUSE_TYPE}'"

• Files reviewed: 9/10 changed files
• Comments generated: 5
• Review effort level: Balanced

[Ankit Jain (radical)]

[automated] Addressed the three suppressed workflow-command findings in 4fed63ad6d.

• Cause filenames are rendered with Bash %q before every pre-validation diagnostic, including invalid JSON, invalid schema, and ID/filename mismatch paths.
• Rejected agent cause types and untrusted prior-cause types are rendered safely before entering ::error:: output.
• The equivalent stored-cause type diagnostic in the publisher is hardened as well.

The regressions assert that newline-bearing rejected values produce exactly one Actions workflow-command line.

[Copilot]

🟡 Changes recommended

Critical validation and moderate fail-closed attribution and publication issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

.github/workflows/analyze-ci-failure.md:540

• When candidate history is unavailable or incomplete, this still exposes the triggering PR to the agent. That contradicts the PR's fail-closed guarantee to withhold triggering-merge context unless every commit association is complete, and the validator does not prevent the agent from copying that PR into a published cause title or diagnostic. Read the history state before this section and render the triggering merge only when it is available.

              echo "Triggering merge PR (context only, not necessarily causal):"
              echo ""
              bash .github/workflows/analyze-ci-failure-persistence.sh \
                render-untrusted-json ci-failure-data/triggering-merge-pr.json

.github/workflows/analyze-ci-failure.md:562

• The candidate script intentionally preserves partial results while setting the state to incomplete, but this unconditional length check still sends those partial candidates to the agent. This defeats the stated fail-closed behavior and can lead to incomplete PR attribution in published cause text. Only render candidates when the history state is available.

              if [ "$(jq 'length' ci-failure-data/candidate-merges.json)" -gt 0 ]; then
                echo ""
                bash .github/workflows/analyze-ci-failure-persistence.sh \
                  render-untrusted-json ci-failure-data/candidate-merges.json
              fi

• Files reviewed: 9/10 changed files
• Comments generated: 3
• Review effort level: Balanced

[Ankit Jain (radical)]

[automated] Addressed the two suppressed Copilot findings in 362864b:

• Triggering-merge context is omitted from the agent summary unless candidate history is available.
• Candidate merges are rendered only for available history, so incomplete or unavailable collections cannot expose a partial candidate list.

[Copilot]

🟡 Changes recommended

One critical and four moderate findings remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 9/10 changed files
• Comments generated: 5
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Exact trusted test-to-job attribution is not currently enforced.

Review details

Suppressed comments (1)

.github/workflows/analyze-ci-failure-validation.sh:143

• The trusted test and job are validated independently here: $matches is selected only by test name, while .job merely has to equal any failed-job name. The collected test-failures.json records contain no job identity, so a real TRX failure can be assigned to a different failed job and then accepted as that job's flaky cause. Preserve job identity while collecting each TRX record and require one trusted {test, job} match before binding diagnostics; otherwise the claimed exact test/job attribution is not enforced.

      all(.failed_tests[];
        . as $reported |
        ([$trusted_tests[0][] | select(.test == $reported.name)]) as $matches |
        ($matches | length) == 1 and
        any($trusted_jobs[0][]; .name == $reported.job))

• Files reviewed: 9/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Incomplete trusted evidence can still permit incorrect publication or reruns, and recurring flaky-cause identity is insufficiently validated.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

.github/workflows/analyze-ci-failure-validation.sh:125

• Trusted TRX evidence is loaded only when the agent reports at least one failed test. If the agent omits all—or only some—trusted failures, validation accepts the result; for example, a transient-infra analysis with failed_tests: [] passes even when ci-failure-data/test-failures.json contains a failed test. The rerun handler likewise checks only the agent array, so this can publish or automatically rerun a real test failure as infrastructure. Sanitize trusted evidence unconditionally and require the reported {name, job} set to equal the trusted {test, job} set before either side effect path proceeds.

FAILED_TEST_COUNT=$(jq '[.failed_tests[]?] | length' "$ANALYSIS_FILE")
if [ "$FAILED_TEST_COUNT" -gt 0 ]; then

• Files reviewed: 9/10 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Missing artifact and test-evidence checks can permit unsafe classification or reruns, and the main rerun contract is inconsistent.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

.github/workflows/analyze-ci-failure-validation.sh:125

• The trusted-test comparison runs only when the agent reports at least one test. An analysis with failed_tests: [] therefore passes even when ci-failure-data/test-failures.json contains a real TRX failure (the validator exits 0 for that fixture), allowing the rerun path to treat a test failure as pure infrastructure. Inspect trusted test evidence independently of the agent array and reject a transient-infra analysis/rerun whenever trusted failed tests were omitted.

FAILED_TEST_COUNT=$(jq '[.failed_tests[]?] | length' "$ANALYSIS_FILE")
if [ "$FAILED_TEST_COUNT" -gt 0 ]; then

• Files reviewed: 11/12 changed files
• Comments generated: 2
• Review effort level: Balanced

[Ankit Jain (radical)]

[automated] Addressed in f32078f: trusted test evidence is now loaded independently of the agent's failed_tests array. Publication requires exact unique {test, job} equality, and reruns read the trusted evidence state and reject any complete evidence containing failed tests.

[Copilot]

🟡 Changes recommended

Three moderate fail-closed validation and PR-state issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 10/12 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Empty TRX evidence can be accepted, and PR comments can still be published to closed pull requests.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

.github/workflows/analyze-ci-failure-persistence.sh:145

• A selected logs artifact can contain only binlogs/logs and no TRX file (for example, when a test job fails before producing results). In that case the loop is empty, jq -sc writes [], and collection marks the evidence complete, allowing publication or rerun without any test evidence. Fail this command when no TRX file is present so the caller keeps the evidence unavailable.

  json_lines=$(mktemp)

• Files reviewed: 10/12 changed files
• Comments generated: 1
• Review effort level: Balanced

[github-actions]

Tests selector

1 / 99 PR test projects · 0 PR jobs · 0 advisory-only targets, from 10 changed files.

Selected PR test projects (1 / 99)

Infrastructure.Tests

Selected PR jobs (0)

none

Advisory workflow impact (0)

none

---

How these were chosen — grouped by what changed

📄 .github/workflows/analyze-ci-failure-candidates.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-comment.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-history.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-issue.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-persistence.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-validation.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure.lock.yml (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure.md (changed)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/AnalyzeCiFailureWorkflowTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/analyze-ci-failure-rerun.harness.js (changed test)
→ 1 directly: Infrastructure.Tests

Job reasons

none

---

Selection computed for commit 7a7a05e.

[Copilot]

🟡 Changes recommended

Unresolved attribution and validation flaws can blame unrelated PRs and omit flaky failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 10/12 changed files
• Comments generated: 3
• Review effort level: Balanced