Skip to content

Fix release-readiness tracker warnings: filter cross-repo PR cross-references and drop invalid authorAssociation gh field - #36029

Merged
PureWeen merged 3 commits into
mainfrom
pureween-fix-tracker-warnings
Jun 19, 2026
Merged

Fix release-readiness tracker warnings: filter cross-repo PR cross-references and drop invalid authorAssociation gh field#36029
PureWeen merged 3 commits into
mainfrom
pureween-fix-tracker-warnings

Conversation

@PureWeen

Copy link
Copy Markdown
Member

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Problem

The [Release Readiness] tracker issues were rendering gh warnings embedded in their bodies, two distinct root causes:

1. Cross-repo timeline cross-references (e.g. SR8 tracker #35876). Get-IssueTimelinePrs walked a regression issue's cross-referenced timeline events and collected the source PR number without checking which repository the PR lives in. Real examples found live:

Looked up against dotnet/maui these 404 (gh pr view 24 ... Could not resolve to a PullRequest), and the warning gets written into the tracker issue. Worse, a foreign PR whose number happens to exist in dotnet/maui (e.g. #877 from DIPSAS/DIPS.Mobile.UI) silently mis-matches an unrelated dotnet/maui PR as a "fix candidate" with no warning at all.

2. Invalid gh pr list field (e.g. SR9 tracker #35867). Get-CandidatePrChecks requested gh pr list --json number,title,author,authorAssociation,updatedAt,url, but authorAssociation is not a valid gh pr list projection field (it's REST-only). The whole query failed (Unknown JSON field: "authorAssociation"), surfacing a warning and silently degrading the maintainer spoof-gate.

Fix

  • Get-IssueTimelinePrs: drop any cross-reference whose repository.full_name ≠ the target repo. The timeline API populates repository.full_name for same-repo references too, so legitimate dotnet/maui fix PRs are preserved. This fixes both the noisy 404 warnings and the silent wrong-repo mis-matches.
  • Get-CandidatePrChecks: remove authorAssociation from the gh pr list projection and fetch author_association per title-matched candidate via gh api repos/{repo}/pulls/{n} (cheap — candidate matches are almost always 0–1). Fail closed: an unreadable association excludes the PR so a Candidate-titled spoof can't slip through.

Tests

Adds 8 deterministic, offline (gh-stubbed) regression assertions: cross-repo timeline filtering (keeps same-repo, drops foreign #24/#102 and non-PR events; foreign-only → 0 candidates) and the REST author-association spoof-gate (MEMBER accepted, CONTRIBUTOR excluded, spoof-only reports exclusion). Offline suite: 492 passed / 0 failed.

Scope check: the sibling Preview engine (Get-PreviewReadiness.ps1) has neither code path, so no change is needed there.

Get-IssueTimelinePrs collected cross-referenced PR numbers without checking
the source repository, so PRs from forks/unrelated repos (e.g. #24 in a fork,
#102 in an unrelated project) were looked up against dotnet/maui — 404ing
(surfacing a 'gh pr view' warning embedded in the tracker issue) or silently
matching an unrelated same-numbered PR. Filter cross-references to $Repo.

Get-CandidatePrChecks queried 'gh pr list --json ...,authorAssociation' but
authorAssociation is not a valid gh pr list field, so the query failed (warning
in the tracker) and the maintainer spoof-gate degraded. Fetch author_association
per title-matched candidate via 'gh api repos/{repo}/pulls/{n}' instead.

Adds 8 deterministic gh-stubbed regression assertions for both fixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 36029

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 36029"

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Skill Validation Results

@PureWeen — new skill validation results are available based on this last commit: 1a5a2c0.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Passed Static Passed LLM Skipped Skills 20 Agents 6

Skill Validation Results1a5a2c0 · Fix release-readiness tracker warnings: filter cross-repo PR cross-references and drop invalid authorAssociation gh field · 2026-06-19T18:45:14Z

✅ Static Checks Passed

Skills: 20 | Eval specs linted: 7

Full lint output
── .github/skills/agentic-labeler/tests/eval.vally.yaml
npm warn deprecated prebuild-install@7.1.3: No longer maintained. Please contact the author of the relevant native addon; alternatives are available.
✔ .github/skills/agentic-labeler/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/eval.capability.vally.yaml
✔ .github/skills/code-review/tests/eval.capability.vally.yaml is valid
── .github/skills/code-review/tests/eval.vally.yaml
✔ .github/skills/code-review/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/hermeticity.vally.yaml
✔ .github/skills/code-review/tests/hermeticity.vally.yaml is valid
── .github/skills/evaluate-pr-tests/tests/eval.vally.yaml
✔ .github/skills/evaluate-pr-tests/tests/eval.vally.yaml is valid
── .github/skills/try-fix/tests/eval.vally.yaml
✔ .github/skills/try-fix/tests/eval.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml is valid

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

@github-actions github-actions Bot added the area-infrastructure CI, Maestro / Coherency, upstream dependencies/versions label Jun 19, 2026
PureWeen and others added 2 commits June 19, 2026 12:31
…n candidate gate

Adversarial multi-model review of the candidate spoof-gate surfaced a real
defect (3/3 consensus after a follow-up round): the per-title-match REST
author_association lookup added in the prior commit fails closed correctly,
but on a *transient* failure (rate-limit, 5xx, network blip, or a race where
the PR is transferred between the gh pr list and the gh api call) it produced
three misleading downstream effects for a LEGITIMATE maintainer Candidate PR:

  1. The candidate vanished from the report (count -> 0).
  2. The WATCH detail asserted it was a 'non-maintainer PR titled Candidate
     excluded as not real cut PR' — mislabeling a transient API failure as a
     CONFIRMED spoofer, which would tell a release captain their own PR isn't
     from a maintainer during an actual SR cut.
  3. Invoke-Gh's non-zero-exit Write-Warn embedded a raw 'gh ... exited'
     line into $Script:Warnings, which is rendered into the tracker issue
     body — reintroducing the exact warning noise this PR set out to remove.

Fix (security posture unchanged — still fail-closed; a spoof PR is never
accepted): replace the Where-Object gate with an explicit loop that classifies
each title match as maintainer / confirmed-spoofer / unverifiable, and report
unverifiable PRs honestly ('could not have their author association verified —
rerun to re-check') instead of as spoofers. Add a -Quiet switch to Invoke-Gh
so the transient association lookup doesn't leak a gh warning into the tracker
body; the structured WATCH note carries that signal instead.

Adds a deterministic offline regression test for the unverifiable path and a
guard asserting confirmed spoofers are not reported as unverifiable. Offline
suite: 522 passed / 0 failed; full E2E: 588 / 0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Round-2 adversarial review follow-ups (both 💡, low-risk, no behavior
regression):

- Get-CandidatePrChecks: when a valid maintainer Candidate PR is accepted
  but a *second* title-matched PR is a confirmed spoofer or an unverifiable
  (transient gh REST failure) lookup, the accepted check now surfaces that
  excluded sibling in its Details and, for unverifiable cases, appends a
  "rerun to re-check" hint to NextAction. Previously those counts were
  silently dropped on the accepted path, so a transient blip on a co-existing
  Candidate PR was invisible to the release captain. Security posture
  unchanged — still fail-closed; only OWNER/MEMBER/COLLABORATOR are accepted.

- Tests: add case (d) (accepted maintainer PR + co-existing unverifiable
  sibling must be surfaced) and a dedicated Invoke-Gh -Quiet test that
  exercises the REAL Invoke-Gh against a simulated failing gh to prove the
  warning-suppression contract directly (warns without -Quiet, silent with
  it) — the classification tests shadow Invoke-Gh and cannot observe it.
  Reword case (c)'s comment so it no longer claims to verify suppression.

Offline suite: 530 passed / 0 failed. Full E2E: 596 passed / 0 failed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@PureWeen
PureWeen merged commit 4567a05 into main Jun 19, 2026
17 of 18 checks passed
@PureWeen
PureWeen deleted the pureween-fix-tracker-warnings branch June 19, 2026 20:08
@github-actions github-actions Bot added this to the .NET 10 SR9 milestone Jun 19, 2026
@MauiBot MauiBot added s/agent-fix-pr-picked AI could not beat the PR fix - PR is the best among all candidates s/agent-reviewed PR was reviewed by AI agent workflow (full 4-phase review) labels Jun 19, 2026
MauiBot

This comment was marked as outdated.

@MauiBot MauiBot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 1 findings

See inline comments for details.

# matches an unrelated $Repo PR that happens to share the number. Filter
# to $Repo. The timeline API populates `repository.full_name` for both
# same-repo and cross-repo references, so this is reliable.
if (-not $iss.PSObject.Properties['repository']) { continue }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] GitHub API shape handling — This drops every cross-referenced PR unless the timeline payload contains source.issue.repository.full_name. The REST timeline issue payloads also expose repository_url, which identifies the source repo. Accept both repository.full_name and repository_url and add a repository_url-only same-repo regression fixture so same-repo PR references are not silently skipped if the nested repository object is absent.

@MauiBot MauiBot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

@PureWeen — new AI review results are available based on this last commit: 1a5a2c0. To request a fresh review after new comments or commits, comment /review rerun.

Gate Skipped Code Review In Review Confidence Low Platform Infrastructure

Review Sessions — click to expand
Gate — Test Before & After Fix

Gate Result: ⚠️ SKIPPED

No tests were detected in this PR.

Recommendation: Add tests to verify the fix using the write-tests-agent.


Pre-Flight — Context & Validation

Issue: #36029 - PR-only infrastructure fix (no linked issue)
PR: #36029 - Fix release-readiness tracker warnings: filter cross-repo PR cross-references and drop invalid authorAssociation gh field
Platforms Affected: infrastructure / release-readiness tooling (Windows PowerShell validation)
Files Changed: 1 implementation, 1 test

Key Findings

  • PR fixes two release-readiness tracker warning sources: foreign repository timeline cross-references being treated as dotnet/maui PR candidates, and an invalid gh pr list --json authorAssociation projection.
  • The implementation filters timeline PR references by repository.full_name before collecting candidate PR numbers, removes the invalid list projection, and fetches REST author_association for title-matched Candidate PRs.
  • Gate was already completed separately and skipped because no tests were detected by the gate script; the PR does include offline PowerShell regression assertions in .github/skills/release-readiness/tests/Test-ReleaseReadiness.ps1.
  • GitHub CLI is unauthenticated in this environment, so PR checks and authenticated review surfaces could not be queried via gh; public REST data and the local merge checkout were used instead.

Code Review Summary

Verdict: NEEDS_DISCUSSION
Confidence: low
Errors: 0 | Warnings: 0 | Suggestions: 0

Key code review findings:

  • No actionable ❌ Error or ⚠️ Warning findings were identified by the independent code-review sub-agent.
  • CI status is undetermined because gh pr checks --required requires authentication in this environment; confidence is capped low.
  • Failure-mode probes confirmed the PR handles foreign low-number PRs, foreign same-number PRs, contributor Candidate PR spoofing, unverifiable maintainer Candidate PRs, and valid Candidate PRs with unverifiable siblings.

Fix Candidates

# Source Approach Test Result Files Changed Notes
PR PR #36029 Filter timeline cross-references by repository.full_name; fetch author_association per Candidate-titled PR using REST; suppress handled lookup warnings via Invoke-Gh -Quiet. ⚠️ SKIPPED (Gate) .github/skills/release-readiness/scripts/Get-ReleaseReadiness.ps1, .github/skills/release-readiness/tests/Test-ReleaseReadiness.ps1 Original PR; gate skipped by prior step.

Code Review — Deep Analysis

Code Review — PR #36029

Independent Assessment

What this changes: Release-readiness tooling now filters timeline cross-referenced PRs to the configured repo, removes unsupported authorAssociation from gh pr list --json, fetches REST author_association only for Candidate-titled PRs, and adds offline regression tests.

Inferred motivation: Prevent tracker-body warning spam and false fix-candidate detection from foreign PR references or invalid gh projections.

Reconciliation with PR Narrative

Author claims: Fixes cross-repo timeline leakage and invalid Candidate PR author-association lookup; adds deterministic tests.

Agreement/disagreement: Agree. The implementation matches the stated defects and tests the relevant failure modes.

Prior Review Reconciliation

Prior ❌ Error Finding Source Status Evidence
None found MauiBot review, inline comments API, issue comments API MauiBot had no ❌ findings; inline comments were empty; issue comments had dogfood + skill-validation only.

gh is unauthenticated, so public REST endpoints and the local merge checkout were used. No GitHub comments were posted.

Blast Radius Assessment

  • Runs for all instances: No — release-readiness script only.
  • Startup impact: None.
  • Static/shared state: Existing $Script:Warnings; new -Quiet suppresses warnings only at selected handled lookup sites.

CI Status

  • Required-check result: undetermined.
  • Classification: gh pr checks 36029 --required could not run because gh auth login is required. Public status for head 1a5a2c0 shows skill-validation success, but required checks could not be confirmed.
  • Action taken: confidence capped low; no posting.

Local validation observed by the sub-agent: Test-ReleaseReadiness.ps1 -SkipE2E = 528 passed / 2 failed. Baseline parent also fails the same two truncation-marker assertions, so failures appear pre-existing/unrelated.

Findings

No actionable ❌ Error or ⚠️ code findings.

Failure-Mode Probing

  • Foreign timeline PR with low/nonexistent dotnet/maui number: filtered by repository.full_name before Get-PrInfo.
  • Foreign PR number exists in dotnet/maui: also filtered, preventing silent wrong-PR match.
  • Contributor opens “Candidate” PR: excluded via REST author association.
  • Maintainer Candidate association lookup fails: excluded fail-closed and surfaced as unverifiable with rerun guidance.
  • Valid Candidate plus unverifiable sibling: valid PR remains reported; sibling is surfaced.

Verdict: NEEDS_DISCUSSION

Confidence: low
Summary: Code is targeted and appears sound, with relevant regression coverage. LGTM cannot be used because required CI status is undetermined due unauthenticated gh.


Fix — Analysis & Comparison

Fix Candidates

# Source Approach Test Result Files Changed Notes
1 try-fix-1 REST pulls-list for Candidate PRs plus canonical timeline pull_request.url filtering. ❌ FAILED 2 files Failed under PowerShell StrictMode because optional author_association was accessed directly when absent.
2 try-fix-2 GraphQL-first: timeline CrossReferencedEvent.source.repository.nameWithOwner filtering plus Candidate PR PullRequest.authorAssociation in one GraphQL query. ⚠️ PARTIAL 2 files PR-specific assertions passed; full suite retained known unrelated 528/2 baseline failures. More complex than PR fix.
PR PR #36029 Filter REST timeline cross-references by repository.full_name; remove invalid gh pr list projection; fetch author_association only for Candidate-titled PRs via REST with quiet handled failures. ⚠️ SKIPPED (Gate) 2 files Original PR. Gate was skipped by prior step; code-review found no actionable code issues.

Cross-Pollination

Model Round New Ideas? Details
gpt-5.5 / maui-expert-reviewer 2 Yes Proposed GraphQL-first approach after try-fix-1 StrictMode failure.
gpt-5.5 / maui-expert-reviewer 3 No No meaningfully different third approach remains; other options are field/fallback variants or less authoritative/brittle mechanisms.

Exhausted: Yes
Selected Fix: PR #36029 — The best alternative (GraphQL-first) is broader and more complex, while the PR fix is smaller, targeted, authoritative, and avoids the invalid gh pr list field without changing the data-access architecture. No candidate passed all tests and was demonstrably better than the PR fix.


Recommended PR Title & Description

Assessment: ✏️ Recommend updating — the provided current title is unknown and the current description is empty, so the PR metadata does not describe the actual release-readiness tooling fix.

Recommended title

[Infrastructure] Release Readiness: Filter cross-repo PR references and fix Candidate author lookup

Recommended description

### Description of Change

Fixes two sources of warning noise and candidate misclassification in the release-readiness tracker tooling:

- `Get-IssueTimelinePrs` now keeps only timeline cross-referenced PRs from the target repo, preventing foreign PRs from being looked up as dotnet/maui fix candidates. This avoids both noisy 404 warnings for low foreign PR numbers and silent same-number mismatches against unrelated dotnet/maui PRs.
- `Get-CandidatePrChecks` no longer requests unsupported `authorAssociation` from `gh pr list --json`. It lists open PRs on the main branch using valid fields, then fetches REST `author_association` only for Candidate-titled PRs and fails closed when the association cannot be verified.

The fix also suppresses handled transient REST lookup failures from the tracker body via `Invoke-Gh -Quiet`, while still surfacing unverifiable Candidate-titled PRs as structured WATCH details.

### Key Technical Details

- Timeline cross-reference filtering uses the source issue/PR repository identity so only same-repo PRs are considered release fix candidates.
- Candidate PR spoof protection accepts only maintainer associations: `OWNER`, `MEMBER`, or `COLLABORATOR`.
- Unverifiable author-association lookups are excluded fail-closed and reported separately from confirmed non-maintainer spoofers.

### Tests

Adds deterministic offline PowerShell regression assertions for:

- Keeping same-repo timeline PR references while dropping foreign PR references and non-PR events.
- Returning no candidates for foreign-only timeline references.
- Accepting a maintainer-authored Candidate PR and excluding contributor-authored Candidate PRs.
- Reporting unverifiable Candidate-titled PRs without labeling them as confirmed non-maintainers.
- Suppressing handled `Invoke-Gh -Quiet` warnings from tracker output.

### Issues Fixed

PR-only infrastructure fix; no linked product issue.

Report — Final Recommendation

Comparative Report — PR #36029

Candidates Compared

Candidate Approach Test Result Assessment
pr Filters timeline PR cross-references by repository.full_name; removes invalid gh pr list --json authorAssociation; fetches REST author_association only for Candidate-titled PRs with quiet handled failures. ⚠️ Gate skipped; PR-specific offline assertions present. Small, targeted, and matches the observed public timeline payload shape. Expert review found one hardening gap around accepting repository_url if nested repository.full_name is absent.
pr-plus-reviewer Same as pr, plus a fallback from repository_url to source repo full name and an offline repository-url-only regression fixture. ⚠️ Relevant assertions passed; full -SkipE2E suite reports known unrelated 529/2 baseline. Best balance: preserves the PR's targeted fix while addressing the expert reviewer's only actionable concern with a small, local change.
try-fix-1 REST pulls-list for Candidate PRs plus canonical timeline pull_request.url filtering. ❌ Failed. Must rank lower than candidates without regression failure. It throws under PowerShell StrictMode when optional author_association is absent.
try-fix-2 GraphQL-first timeline and Candidate PR queries, including authorAssociation in one GraphQL flow. ⚠️ Relevant assertions passed; full suite retained known unrelated 528/2 baseline. Viable but broader and more complex than necessary. Adds GraphQL pagination/query plumbing for a localized release-readiness script fix.

Ranking

  1. pr-plus-reviewer — wins. It is the raw PR's targeted fix plus the only expert-review hardening suggestion, with relevant regression coverage.
  2. pr — acceptable and targeted, but slightly less robust than pr-plus-reviewer for timeline payload shape variation.
  3. try-fix-2 — relevant assertions passed, but it is a larger GraphQL rewrite with more moving parts and no clear benefit over the PR-lineage fix.
  4. try-fix-1 — failed regression validation under StrictMode and must rank below candidates whose relevant assertions passed.

Winning Candidate

Winner: pr-plus-reviewer

pr-plus-reviewer should be selected because it keeps the PR's minimal REST/CLI architecture, avoids the invalid gh pr list field, filters foreign timeline PRs, and adds a low-risk repository_url fallback for the expert reviewer's API-shape concern. Candidates with failed regression tests were ranked lower as required.


Future Action — review latest findings

No alternative fix was selected for this run. Review the session findings and CI results before merging.

@github-actions github-actions Bot locked and limited conversation to collaborators Jul 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-infrastructure CI, Maestro / Coherency, upstream dependencies/versions s/agent-fix-pr-picked AI could not beat the PR fix - PR is the best among all candidates s/agent-reviewed PR was reviewed by AI agent workflow (full 4-phase review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants