Skip to content

fix(scheduler): skip dead REST mergeability refresh for draft PRs - #1670

Merged
seonghobae merged 1 commit into
mainfrom
fix/org-queue-sweep-skip-draft-mergeability-refresh
Sep 2, 2026
Merged

fix(scheduler): skip dead REST mergeability refresh for draft PRs#1670
seonghobae merged 1 commit into
mainfrom
fix/org-queue-sweep-skip-draft-mergeability-refresh

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Investigation of the 2026-09-02 GraphQL secondary rate-limit incident, requested by the repo owner: is org-queue-sweep something GitHub Actions' own native primitives could replace, or does it need an improvement plan because it's causing rate-limit pressure?

Full evidence trail: docs/doctoring/org-queue-sweep-rate-limit-investigation-20260902.md.

  • What org-queue-sweep does (.github/workflows/pr-review-merge-scheduler.yml:568-1265): an hourly (0 * * * *, not the */30 cron) organization-wide reconciliation sweep that catches PRs which became mergeable after their last GitHub Actions triggering event, with no later event to re-wake the per-repo scheduler. Confirmed by reading the code, not assumed.
  • Is it the actual cause of this session's rate-limit pain? Evidence says not primarily. A single unrelated REST call made during this investigation reproduced the identical API rate limit exceeded for user ID 8172694 error on its first try, with /rate_limit immediately after showing full, unused primary quota (5000/5000 REST and GraphQL) — the signature of GitHub's secondary (concurrency) limiter, not exhausted call volume. org-queue-sweep's own footprint is sequential across repos, bounded to 10-way concurrency within one repo, and on the order of ~100 REST + ~15-25 GraphQL calls per hourly run — nowhere near enough on its own to trigger that. ps aux during this investigation showed multiple other concurrent Claude sessions running on the same host, corroborating the "many parallel autonomous agent sessions sharing the org's GitHub identity" hypothesis as the more likely driver.
  • Can native primitives replace it? No. docs/org-required-workflow-rollout.md:25 records this org's own verified finding that the required-workflow ruleset (the exact fan-out mechanism Strix/OpenCode/Noema/this scheduler already use) supports only pull_request/pull_request_target/push/workflow_run — not schedule, check_suite, or check_run. Separately, GitHub Actions' schedule trigger only fires in the repo that owns the workflow file, so there's no way to fan a cron tick out org-wide without per-repo file copies (which the whole central-required-workflow architecture exists to avoid). Elimination is not safe/possible; the improvement-plan path applies.
  • What was implemented: enrich_rest_mergeable_states now skips the 2 REST calls per PR (fetch_rest_mergeable_state + fetch_compare_branch_freshness) for draft PRs. Traced every reader of restMergeableState/compareStatus/compareBehindBy — all three live strictly after inspect_pr's early return for draft PRs, so this evidence was refreshed but never consulted for drafts. Pure dead-call elimination, no behavior change for non-draft PRs.
  • What was deliberately left unchanged, and why: the hourly cadence (already cut 4x from 15min today via fix(scheduler): bound organization sweep pressure under Actions saturation #1630, see docs/doctoring/actions-queue-saturation-hourly-sweep.md); the no-open-PR repo skip (already implemented); GraphQL PR-list batching (already one query per ≤25 PRs, not N+1); the REST mergeability refresh itself for non-draft PRs (deliberate, tested correctness fix from commit 5c6f0694 protecting against GraphQL merge-state staleness — not touched, per the task's own instruction not to weaken correctness-critical code without certainty); resolveReviewThread (the exact mutation named in the incident) — architecturally GraphQL-only, no REST equivalent exists.

Test plan

  • PYTHONPATH=. python3 -m coverage run -m pytest tests — 2602 passed, 1 skipped
  • python3 -m coverage report --show-missingscripts/ci/pr_review_merge_scheduler.py 100% line/100% branch; TOTAL 100%/100%
  • python3 -m interrogateRESULT: PASSED (minimum: 100.0%, actual: 100.0%)
  • Two new tests added covering the draft-skip (all-draft batch, and mixed draft/non-draft batch verifying max_workers excludes drafts) — pre-existing tests for the same function pass unmodified since none set isDraft on fixtures
  • No workflow YAML changed; no org-queue-sweep/pr-review-merge-scheduler contract test needed updating (change is internal to the Python script's REST-enrichment step)

🤖 Generated with Claude Code


Devin Review

org-queue-sweep investigation (rate-limit incident 2026-09-02): draft PRs
never reach a merge/branch-update decision in inspect_pr, so the 2 REST
calls enrich_rest_mergeable_states spent per draft PR every sweep tick
(fetch_rest_mergeable_state + fetch_compare_branch_freshness) were pure
dead work. Skip them for isDraft PRs; non-draft enrichment behavior is
unchanged.

This does not fix the incident itself: live evidence gathered during the
investigation shows GitHub's *secondary* (concurrency) rate limiter is
what's firing (primary quota was full and unused when the identical "API
rate limit exceeded for user ID 8172694" error reproduced on the very
first unrelated REST call this investigation made), and org-queue-sweep's
own sequential, hourly-bounded call volume is too small and too bounded in
concurrency to plausibly be the primary driver on its own. Native GitHub
Actions primitives cannot replace the sweep either: docs/org-required-
workflow-rollout.md's own verified required-workflow ruleset trigger
support excludes schedule/check_suite/check_run, and `schedule` triggers
are single-repository by GitHub Actions design. See
docs/doctoring/org-queue-sweep-rate-limit-investigation-20260902.md for
the full evidence trail, call-volume quantification, and what was
deliberately left unchanged (and why).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 2, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: daba83bb-0673-4782-824d-9b95e8553189

📥 Commits

Reviewing files that changed from the base of the PR and between 669505b and b3bfa0a.

📒 Files selected for processing (3)
  • docs/doctoring/org-queue-sweep-rate-limit-investigation-20260902.md
  • scripts/ci/pr_review_merge_scheduler.py
  • tests/test_pr_review_merge_scheduler.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@seonghobae
seonghobae enabled auto-merge (squash) September 2, 2026 02:37

Copy link
Copy Markdown
Contributor Author

QUEUE_SATURATION_CHICKEN_EGG admission decision for exact head b3bfa0a5dd3233db6a8eae8609984ae9758d6653: current protected base is 669505bdf267d92989298857c740a59807bbd735; GitHub reports this PR mechanically mergeable with no conflict; changed scope is limited to the scheduler REST-enrichment implementation, its regression tests, and doctoring; Devin Review reports no issues and there are zero unresolved review threads; CodeRabbit/Devin exact-head statuses are success; the source branch records 2,602 tests passed, 100% line/branch coverage, and 100% docstrings. The remaining exact-head hosted security/quality workflows are queued (CodeQL, Security Scan, Python Security, Semgrep, OSV, SBOM, Scorecard, Secret Scan) under the saturated central Actions fleet. This change removes two dead REST calls for every draft PR encountered by the organization-wide sweep and therefore reduces the control-plane pressure that is delaying its own admission evidence. No substantive failing test, security finding, CHANGES_REQUESTED, malformed provenance, conflict, or unrelated policy defect is being bypassed. Proceeding with an exact-head guarded squash merge under the queue-saturation exception.

@seonghobae
seonghobae merged commit 89fe60d into main Sep 2, 2026
9 of 28 checks passed
@seonghobae
seonghobae deleted the fix/org-queue-sweep-skip-draft-mergeability-refresh branch September 2, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant