Skip to content

fix(scheduler): resolve live refs before cancelling runs - #1348

Merged
seonghobae merged 28 commits into
mainfrom
fix/queue-hygiene-live-ref-race
Sep 1, 2026
Merged

fix(scheduler): resolve live refs before cancelling runs#1348
seonghobae merged 28 commits into
mainfrom
fix/queue-hygiene-live-ref-race

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Root cause

The organization queue sweep treated the SHA in the open-PR list as the
authoritative branch head. During a synchronize-event propagation window,
LineageWeave #667 already had current-head runs for 37cc9ab1, while the PR
payload still exposed its preceding head. Queue hygiene cancelled the valid new
runs as superseded.

Repair

  • enumerate open PR head repositories and branch names
  • resolve each current SHA through GitHub's Get a reference endpoint
  • compare queued/in-progress runs with that live ref
  • fail closed for the whole repository when any ref is inaccessible or malformed
  • retain immediate cancellation of genuinely superseded heads without a time heuristic

Evidence

  • uv run --group dev pytest -q — 1,437 passed, 1 skipped, 16 subtests passed
  • coverage run -m pytest -q tests && coverage report --fail-under=100 — 100% statements and branches
  • interrogate --fail-under=100 scripts/ci — 100%
  • actionlint .github/workflows/pr-review-merge-scheduler.yml
  • git diff --check

Cross-repository incident: ContextualWisdomLab/LineageWeave#667.


Open in Devin Review

Summary by CodeRabbit

  • 버그 수정

    • PR의 최신 브랜치 상태를 실시간으로 확인한 후 오래된 워크플로 실행을 취소합니다.
    • 브랜치 참조를 확인할 수 없거나 유효하지 않으면 안전을 위해 해당 저장소의 실행 취소를 중단합니다.
    • 오래된 PR 정보로 인해 현재 커밋의 검사가 잘못 취소되는 문제를 방지합니다.
  • 문서

    • 관련 장애 원인, 처리 규칙 및 검증 방법을 문서화했습니다.
    • 변경 사항을 미출시 항목에 기록했습니다.
  • 테스트

    • 최신 PR 헤드 확인 및 참조 조회 실패 시 동작을 검증합니다.

@coderabbitai

coderabbitai Bot commented Aug 26, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: dde04617-a7e2-4fa5-b9aa-371748c171b6

📥 Commits

Reviewing files that changed from the base of the PR and between dca874b and 79d4461.

📒 Files selected for processing (6)
  • .github/workflows/pr-review-merge-scheduler.yml
  • docs/doctoring/queue-hygiene-live-ref-race.md
  • scripts/ci/revalidate_queue_cancellation.sh
  • tests/test_queue_cancellation_open_pr_revalidation.py
  • tests/test_queue_cancellation_revalidation.py
  • tests/test_queue_cancellation_scheduler_contract.py
📝 Walkthrough

Walkthrough

조직 큐 스윕이 열린 PR의 head SHA를 live Git ref에서 다시 조회합니다. ref 조회 실패나 잘못된 SHA가 있으면 실행 취소를 중단하고 queue_hygiene_ready=false로 처리합니다. 테스트와 문서가 이 계약과 장애 사례를 반영합니다.

Changes

실시간 PR 헤드 검증

Layer / File(s) Summary
Live Git ref 조회와 안전 중단
.github/workflows/pr-review-merge-scheduler.yml, tests/test_required_workflow_queue_contract.py
조직 큐 스윕이 head 저장소와 브랜치의 live Git ref를 조회합니다. 조회 결과가 없거나 SHA 형식이 잘못되면 실행 취소를 중단하고 오류를 기록합니다. 검증 계약이 ref 조회와 jq 기반 SHA 추출을 확인합니다.
변경 기록과 장애 기준
CHANGELOG.md, docs/doctoring/queue-hygiene-live-ref-race.md, docs/product-technical-gap-baseline.md
변경 기록과 장애 문서가 오래된 PR 페이로드와 현재 Git ref의 불일치, 안전 중단 조건, 이전 head 실행 취소 규칙을 기록합니다.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: 🟡 Moderate · up to dca87

Queue cleanup can still cancel valid active runs when an open PR has a missing or malformed head reference because that PR is silently omitted from live-head resolution. Fix the fail-closed handling before merging.

Sequence Diagram(s)

sequenceDiagram
  participant QueueSweep as 조직 큐 스윕
  participant PullRequestsAPI as GitHub Pull Requests API
  participant GitRefAPI as GitHub Git ref API
  participant ActionsAPI as GitHub Actions API
  QueueSweep->>PullRequestsAPI: 열린 PR의 head 저장소와 브랜치 조회
  QueueSweep->>GitRefAPI: head 브랜치의 live Git ref 조회
  GitRefAPI-->>QueueSweep: 현재 SHA 반환 또는 조회 실패
  QueueSweep->>ActionsAPI: 유효한 이전 head 실행 취소
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 실행 취소 전에 live Git ref를 조회하도록 스케줄러를 수정하는 PR의 핵심 변경을 정확하고 간결하게 설명합니다.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. (4 skipped: 4 …
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. (4 skipped: 4 unsupported.)

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/queue-hygiene-live-ref-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor Author

Fresh current-head cancellation canary after a normal non-force ref update: PR #1158 moved to exact head 54f5bbe73a292aa11d920efc263f88870acde4b9 (parents 39643cca... and protected main@f6c2a270...). All hosted runs for that exact head were immediately terminal cancelled, including Security Scan 33146873036, SAST 33146872997, CodeQL 33146873013, OSV Direct Source Quality 33146873078, OSV-Scanner 33146873336, SBOM 33146873004, Python Security 33146872982, Secret Scan 33146873046, and both readiness/attestation quality lanes. Local focused evidence is 90 passed, 54 subtests; full is 1732 passed, 1 skipped, 71 subtests, but cancelled hosted runs remain non-passing.

Please use #1158@54f5bbe7 as a live regression fixture for the ref-resolution repair. Acceptance should prove the canceller compares each candidate against the current PR head/ref immediately before cancellation, preserves same-head runs, and only cancels a proven predecessor/closed-PR run. Once protected main contains the fix, rerun the unchanged #1158 head rather than counting these cancelled attempts.

Fix the empty-open-PR inventory diagnostic without weakening live-ref cancellation evidence.
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Renumber this branch's new Gap G-15 entry to G-16 to avoid colliding
with main's existing G-15 (attachment processing), which merged
independently while this branch was open.
devin-ai-integration[bot]

This comment was marked as resolved.

seonghobae added a commit that referenced this pull request Aug 30, 2026
* docs(gaps): restore section dropped by #1423's merge, correct stale claim

PR #1423 merged (ebd1b8b) using a merge-conflict resolution that kept
only its own branch's dated section and silently dropped main's
already-published '2026-08-30 sidecar pin staleness recurrence'
section (the #1422 fix's own record) from the file entirely. That
merge also left #1423's own section claiming #1413/#1422 'have not
merged' even though both are its own git ancestors.

Restore the dropped section verbatim, annotate the stale section as
superseded, and append an accurate dated entry recording this cycle's
backlog-refresh work: 15 PRs merged/retriggered against the new main,
8 left untouched for real conflicts, one independent sidecar/review-
dispatch defect newly observed on #1420, and a Gap ID collision
(G-15) renumbered to G-16 on #1348.

Verified: git diff against origin/main shows this change is purely
additive (no main content removed); pytest
tests/test_product_technical_gap_baseline.py
tests/test_pr_governance_audit_contract.py (8 passed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw

* docs(gaps): fix Devin-flagged count/reference errors in #1424

- Clean-merge subtotal said 5 but named 6 PRs; corrected to 6, and added
  the omitted #1391 to the trivial-conflicts list so 6+7+2=15 reconciles
  with the stated total.
- Compact cross-repo references to the owner/repo#num binding convention:
  ContextualWisdomLab/contextual-orchestrator#857 and
  ContextualWisdomLab/LineageWeave#667.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Resolved the stale merge conflict against current main and pushed a merge commit (af519b79) onto this PR's branch (fix/queue-hygiene-live-ref-race).

Conflict scope: only CHANGELOG.md's ## [Unreleased] section conflicted. This PR's entry (live-ref resolution before queue-hygiene cancellation) sat adjacent to several independent main entries added since this PR opened (the ORCHESTRATOR_CATALOG_FAMILY_CAP default bump, the Strix orchestrator/autoorchestrator/free switch, and the zdr_policy.py NVIDIA ToS citation strengthening). All four entries are additive and unrelated in substance, so I kept this PR's entry first (chronologically first Unreleased addition) followed by main's three entries, in original order — no content was dropped.

.github/workflows/pr-review-merge-scheduler.yml, docs/product-technical-gap-baseline.md, and tests/test_required_workflow_queue_contract.py all auto-merged cleanly with no marker conflicts — main's unrelated Strix/contextual-orchestrator sidecar changes and this PR's queue-hygiene live-ref logic touch disjoint regions.

Test evidence on the merged tree:

  • tests/test_required_workflow_queue_contract.py: 71 passed
  • Full suite (PYTHONPATH=. python3 -m pytest tests -q): 1900 passed, 1 skipped, 21 subtests passed
  • coverage run -m pytest tests -q && coverage report --show-missing: scripts/ci TOTAL 9966 stmts / 3926 branches — 100%
  • interrogate -v scripts/ci: TOTAL 887/887 — 100.0%
  • bash -n on all touched .sh files (contextual_orchestrator_review_sidecar.sh, strix_quick_gate.sh, strix_required_workflow_smoke.sh, test_strix_quick_gate.sh): all OK
  • python3 -c "import yaml; yaml.safe_load(open('.github/workflows/pr-review-merge-scheduler.yml'))": parses cleanly
  • git diff --check: clean, no leftover conflict markers or whitespace errors

No product/design decision was in question here — this was a mechanical changelog merge plus disjoint-region auto-merges. The branch is now conflict-free against main and ready for review/CI.


Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

seonghobae pushed a commit that referenced this pull request Sep 1, 2026
…llision

Devin correctly flagged that G-16 was already allocated in this same
document (line ~633) to #1348's own queue-hygiene gap, which is still
open and unmerged, so reusing it here would collide once #1348 lands.
Renumbered the three 2026-09-01-directive gaps to G-19/G-20/G-21
(verified free via a PR body search) and fixed the timeout-floor
cross-reference to point at the owner-directive section rather than a
Gap ID that does not itself define the 3-hour floor.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 1 new potential issue.

🐛 1 issue in files not directly in the diff

🐛 Final cancellation check never runs

revalidate_queue_cancellation.sh has no production caller after this workflow is removed. A synchronized PR can therefore lose its current-head checks to stale cancellation.

Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@seonghobae
seonghobae merged commit 7683f1d into main Sep 1, 2026
14 of 33 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in naruon Platform Roadmap Sep 1, 2026
@seonghobae
seonghobae deleted the fix/queue-hygiene-live-ref-race branch September 1, 2026 21:14

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 4 new potential issues.

Devin Review


case "$event" in
pull_request|pull_request_target)
pr_number="$(jq -r '.pull_requests[0].number // empty' <<<"$run_json")"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Secondary pull requests lose current checks

For runs associated with several pull requests, pull_requests[0] validates only the first. Another open pull request can lose its current-head check.

Prompt for agents
Update scripts/ci/revalidate_queue_cancellation.sh to evaluate every pull request listed in the live workflow-run metadata, not only pull_requests[0]. Before cancellation, preserve the run if any associated open PR resolves to the run head. Fail closed if any relevant PR or head ref cannot be read or validated. Add regression coverage for a run associated with multiple PRs where the first is closed or superseded and a later open PR has the run's authoritative current head.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +176 to +178
if ! gh api -X POST "/repos/${repo_full_name}/actions/runs/${run_id}/cancel" >/dev/null; then
echo "Could not cancel ${cancellation_mode} run ${run_id} in ${repo_full_name}; it may have started or finished already."
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Cancellation remains best effort

A failed cancellation returns success after logging. The next hourly sweep can retry the still-active candidate without failing unrelated repository processing.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +134 to +136
if [ "$live_head_sha" != "$classified_sha" ] || [ "$final_ref_sha" != "$classified_sha" ]; then
warn_preserve "PR ${pr_number} moved after queue classification."
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Head movement defers cleanup safely

Any post-classification head mismatch preserves the candidate. A genuinely obsolete run can survive one sweep, avoiding cancellation from inconsistent intermediate evidence.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +67 to +106
if ! fresh_open_pr_refs_json="$(
gh api \
-H "Accept: application/vnd.github+json" \
"/repos/${repo_full_name}/pulls?state=open&per_page=100" \
--paginate \
| jq -sc '[.[] | .[] | {
repo: (.head.repo.full_name // null),
ref: (.head.ref // null)
}]'
)"; then
warn_preserve "open PR heads could not be re-fetched for an unassociated PR run."
fi
if ! jq -e '
all(.[];
(.repo | type) == "string" and (.repo | length) > 0 and
(.ref | type) == "string" and (.ref | length) > 0
)
' <<<"$fresh_open_pr_refs_json" >/dev/null; then
warn_preserve "fresh open PR head evidence is malformed."
fi
if jq -e \
--arg repo "$run_head_repo" \
--arg ref "$run_branch" \
'any(.[]; .repo == $repo and .ref == $ref)' \
<<<"$fresh_open_pr_refs_json" >/dev/null; then
encoded_run_ref="$(encode_ref_path "$run_branch")"
if ! final_ref_sha="$(
gh api \
-H "Accept: application/vnd.github+json" \
"/repos/${run_head_repo}/git/ref/heads/${encoded_run_ref}" \
--jq '.object.sha // empty'
)"; then
warn_preserve "live ref for newly associated PR head could not be re-fetched before cancellation."
fi
if ! [[ "$final_ref_sha" =~ ^[0-9a-fA-F]{40}$ ]]; then
warn_preserve "live ref for newly associated PR head is malformed."
fi
if [ "$run_head" = "$final_ref_sha" ]; then
warn_preserve "run became associated with an open PR at its authoritative current head after queue classification."
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Unassociated runs avoid stale payloads

Late association matches repository and branch identity, then resolves the Git ref. A stale pull-request list SHA cannot authorize cancellation.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants