Skip to content

ci(janitor): re-run CLA checks the spent GITHUB_TOKEN budget failed, once the budget is back - #25326

Merged
teamleaderleo merged 3 commits into
mainfrom
gt-quota-cla-heal
Oct 26, 2026
Merged

teamleaderleo merged 3 commits into
mainfrom
gt-quota-cla-heal

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

The required checks CLA Assistant and CLA policy guard both run on the repository's GITHUB_TOKEN. When the installation budget runs out (2026-10-06, and since 21:10Z on 2026-10-25), both fail outright and stay red until someone re-runs them by hand, often while the budget is still spent. This currently blocks #25172 and #25302, among others.

Why CLA policy guard says "rejected the proposed policy" on PRs that touch no CLA file. validate-cla-policy.rb reads the PR, its reviews, its compare and snapshots through gh api. A failed read raises PolicyError ("GitHub API request failed…"), and the rescue at the end turns every PolicyError into the same generic annotation. So a quota 403 looks exactly like a policy rejection. That explains the pass-then-fail pattern on unrelated PRs.

Why the fix is not inside the two jobs. cla.yml and the guard are a trusted control plane. The guard requires an exact-head review from a trusted maintainer (TRUSTED_REVIEWER_IDS) for any change to cla.yml or to its own bytes (EXPECTED_GUARD_SCRIPT_DIGEST). In-job backoff is therefore a trust-policy change and is left to that review.

What this adds instead is a self-healing re-run in the janitor's scheduled sweep (every 10 minutes, no new schedule):

  1. One rate_limit read of GITHUB_TOKEN, which costs no quota. If less than 20% (minimum 500) is left, it stops: a re-run would fail the same way.
  2. Using the janitor's App token (its own budget, actions: write), it takes each PR head's newest run of each workflow, across all statuses. It acts only when that run failed at the head the PR still has.
  3. It re-runs a CLA Assistant failure whose job log shows the rate limit. A genuine "not signed" failure is left alone.
  4. It re-runs a CLA policy guard failure while its attempt is under 8, because the guard's reason is hidden. A genuine rejection is deterministic and stops there.
  5. Bounds per sweep: 12 log reads (rotated, so genuine failures can't starve the scan) and 8 re-runs, within 24 hours. The step never fails the sweep.

Cost: about 34 App calls per sweep at most, and 0 GITHUB_TOKEN calls.

Tests: tests/test_cla_quota_rerun.py, 7 cases against a fake GitHub, all pass. actionlint is clean.

Review: independent Codex (gpt-6-astra, xhigh). The first pass returned BLOCK with three findings (lost guard retries, scan starvation, replaying superseded runs). All three are fixed, and the second pass returned SHIP.

🤖 Generated with Claude Code

Agent session: agent session


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by CodeRabbit

  • Bug Fixes
    • Recent CLA check failures caused by API rate limits can now be automatically rerun, helping eligible checks complete without manual intervention.
    • Reruns are limited to failures on the pull request’s current revision and are capped to avoid excessive retries. Other failures and outdated checks are excluded.

teamleaderleo and others added 2 commits October 25, 2026 21:32
…once the budget is back

CLA Assistant and CLA policy guard (required on main) run on GITHUB_TOKEN and
fail outright when the installation budget is spent (since 21:10Z 2026-10-25),
then stay red until someone re-runs them by hand. Both are a trusted control
plane whose change needs a trusted maintainer's review, so the retry lives in
the janitor's scheduled sweep: one free rate_limit read of GITHUB_TOKEN, and
when at least 20% is left, re-run (on the App's budget) each CLA Assistant run
at its PR's current head whose log says the rate limit was exceeded, plus that
head's policy guard run (whose reason is hidden by design; a first-attempt
guard failure also gets one retry). Bounded: 12 log reads and 8 re-runs a
sweep, 24 h, attempt < 20. Never fails the sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nded scan, re-runs failed guards up to 8 attempts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@blacksmith-sh

blacksmith-sh Bot commented Oct 26, 2026

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_a_requested_ci_run_refreshes_a_stale_snapshot (main.WorkflowShapeTests.test_a_
requested_ci_run_refreshes_a_stale_snapshot)/
test_a_requested_ci_run_refreshes_a_stale_snapshot (main.WorkflowShapeTests.test_a_
requested_ci_run_refreshes_a_stale_snapshot)
View Logs
test_bot_jobs_mint_and_use_the_scoped_app_token (main.CmuxBotTokenWorkflowShape.tes
t_bot_jobs_mint_and_use_the_scoped_app_token) (workflow='ci-queue-janitor.yml', job='sw
eep')/
test_bot_jobs_mint_and_use_the_scoped_app_token (main.CmuxBotTokenWorkflowShape.tes
t_bot_jobs_mint_and_use_the_scoped_app_token) (workflow='ci-queue-janitor.yml', job='sw
eep')
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Walkthrough

Adds a scheduled workflow step and a script that checks API quota, identifies eligible recent CLA failures, and submits bounded reruns. The script supports dry runs and reports handled errors as warnings.

Changes: CLA quota failure reruns

Layer / File(s) Summary
Quota checks and run eligibility
scripts/ci/cla_quota_rerun.py
The script checks the core API quota and filters workflow runs by completion, failure status, recency, current pull request head, and attempt count.
Failure selection and scheduled execution
scripts/ci/cla_quota_rerun.py, .github/workflows/ci-queue-janitor.yml, tests/test_cla_quota_rerun.py
The script checks failure logs for rate-limit messages, rotates candidate scans, and caps reruns. The scheduled workflow invokes it with separate tokens. Offline tests cover selection, limits, and dry-run behavior.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant GitHubActions as GitHub Actions
  participant RerunScript as cla_quota_rerun.py
  participant GitHubAPI as GitHub API
  GitHubActions->>RerunScript: Run scheduled sweep with App token and GITHUB_TOKEN
  RerunScript->>GitHubAPI: Check core quota using RATE_TOKEN
  RerunScript->>GitHubAPI: List workflow runs and inspect eligible failure logs using GH_TOKEN
  RerunScript->>GitHubAPI: Submit selected failed-job reruns using GH_TOKEN
Loading

Suggested labels: risk:high

Merge Risk: Minimal · up to 37196

This adds a bounded, non-failing retry for CLA checks that failed because the token budget was spent. No concrete merge-blocking risk was established. The unpaginated run query may miss failures only if run volume gets very high.

Pre-merge checks: Passed 24 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 19.05% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 21 functions across 2 files. (1 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 24 passed
Check name Status Explanation
Title check Passed The title clearly identifies the janitor change that reruns CLA checks after the GITHUB_TOKEN budget recovers.
Description check Passed The description clearly explains the problem, implementation, limits, testing, and review results. It omits the template headings for Changelog, Proof, and Checklist, but the missing content is non-cr…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input Passed PASS: The pull request changes only the CI janitor workflow, a GitHub CLA quota re-run script, and its tests. The diff introduces no Cloud terminal creation, cmux-tui client or transport, manual rende…
Cmux Swift Actor Isolation Passed The pull request changes only a GitHub Actions workflow, Python code, and Python tests. The authoritative diff contains no Swift files or Swift declarations, so it introduces no Swift 6 actor-isolatio…
Cmux Swift Blocking Runtime Passed PASS: The pull request changes only a GitHub workflow, Python code, and Python tests. It introduces no production Swift files or Swift blocking/timing primitives.
Cmux Browser Automation Off-Main Passed PASS: The PR changes only the janitor workflow, a Python CLA quota script, and its tests. It does not change browser socket automation, WebKit/AppKit routing, or policy tests. The governed Swift files…
Cmux Expensive Synchronous Load Passed The pull request changes only one workflow YAML file and two Python files. The reviewed diff contains no Swift production changes, so the expensive synchronous Swift load condition does not apply.
Cmux Cache Substitution Correctness Passed PASS: The pull request changes only a GitHub Actions workflow, Python code, and Python tests. It introduces no production Swift, TypeScript, or JavaScript change, and no persistence, history, undo, or…
Cmux No Hacky Sleeps Passed PASS: The pull request introduces no fixed sleep, timer, polling loop, or wall-clock wait used to hide a lifecycle or readiness race. The production Python script uses a 30-second network request time…
Cmux Algorithmic Complexity Passed No algorithmic-complexity violation is introduced. The production script processes at most one API page of workflow runs (per_page=100), uses a dictionary for newest-run selection, and bounds log re…
Cmux Swift Concurrency Passed The pull request changes only one workflow file and two Python files. The reviewed diff contains no Swift files or Swift concurrency patterns, so it does not introduce or expand the specified legacy a…
Cmux Swift @Concurrent Passed The pull request changes only a workflow, a Python script, and Python tests. It introduces no Swift changes, so the Swift @concurrent rule does not apply.
Cmux Swift Package Boundaries Passed The reviewed diff changes only one YAML workflow and two Python files. It contains no production Swift changes, so the Swift package boundary rule does not apply.
Cmux Swiftpm Lockfiles Passed The PR changes one workflow plus Python and test files. The workflow only checks out and runs scripts/ci/cla_quota_rerun.py; it does not change SwiftPM dependencies, Xcode package references, `.giti…
Cmux Swift Logging Passed PASS: The pull request changes only a workflow, Python code, and Python tests. It adds no Swift files or Swift logging. The added Python print calls are CLI output, which is outside this Swift-speci…
Cmux User-Facing Error Privacy Passed The changed files add a scheduled GitHub Actions janitor step and an internal CI script. Its output is CI/operator diagnostic text, including GitHub and token-budget terms, not an app UI, product CLI,…
Cmux Full Internationalization Passed The diff changes only a scheduled GitHub Actions janitor, a Python CI helper, and offline tests. Added text is CI step metadata, operational comments, script diagnostics, and test fixtures; it is not …
Cmux Swiftui State Layout Passed The check is not applicable. The pull request changes one YAML workflow and two Python files. The diff contains no SwiftUI files or SwiftUI state/layout constructs, so it does not introduce any listed…
Cmux Architecture Rethink Passed PASS: The pull request changes only a GitHub Actions workflow, Python CI logic, and Python tests. The diff contains no Swift files or Swift architecture changes, so the Swift-specific failure conditio…
Cmux Swift Auxiliary Window Close Shortcuts Passed The pull request changes only a workflow, a Python script, and Python tests. It adds or materially changes no Swift-owned window code, so the auxiliary-window close-shortcut rule does not apply.
Cmux Source Artifacts Passed The diff changes only an existing workflow, a hand-written CI script, and its unit test. These are intentional source, configuration, and test files. No logs, screenshots, recordings, temp directories…
Cmux No Test Or Debug Seam In Production Source Passed The pull request changes only a workflow, a Python script, and a Python test. It changes no Swift file under a production Sources/ path, so this check does not apply.
Full details: Description check

Explanation

The description clearly explains the problem, implementation, limits, testing, and review results. It omits the template headings for Changelog, Proof, and Checklist, but the missing content is non-critical for this CI-only change.

Full details: Cmux Cloud Persistent Session And Early Input

Explanation

PASS: The pull request changes only the CI janitor workflow, a GitHub CLA quota re-run script, and its tests. The diff introduces no Cloud terminal creation, cmux-tui client or transport, manual renderer, PTY readiness gate, input routing, snapshot, auth, revision, or lease behavior covered by the rule.

Full details: Cmux Swift Actor Isolation

Explanation

The pull request changes only a GitHub Actions workflow, Python code, and Python tests. The authoritative diff contains no Swift files or Swift declarations, so it introduces no Swift 6 actor-isolation issue.

Full details: Cmux Browser Automation Off-Main

Explanation

PASS: The PR changes only the janitor workflow, a Python CLA quota script, and its tests. It does not change browser socket automation, WebKit/AppKit routing, or policy tests. The governed Swift files Sources/TerminalController.swift and Packages/macOS/CmuxControlSocket/Sources/CmuxControlSocket/Wire/ControlCommandExecutionPolicy.swift are unchanged.

Full details: Cmux Cache Substitution Correctness

Explanation

PASS: The pull request changes only a GitHub Actions workflow, Python code, and Python tests. It introduces no production Swift, TypeScript, or JavaScript change, and no persistence, history, undo, or snapshot cache substitution is present.

Full details: Cmux No Hacky Sleeps

Explanation

PASS: The pull request introduces no fixed sleep, timer, polling loop, or wall-clock wait used to hide a lifecycle or readiness race. The production Python script uses a 30-second network request timeout, a 24-hour eligibility window, and a ten-minute timestamp-based scan rotation. These bound requests, select recent runs, and provide fair scan coverage; they do not delay execution or paper over readiness. The GitHub Actions timeout and existing ten-minute schedule are workflow orchestration, which the rule excludes.

Full details: Cmux Algorithmic Complexity

Explanation

No algorithmic-complexity violation is introduced. The production script processes at most one API page of workflow runs (per_page=100), uses a dictionary for newest-run selection, and bounds log reads and reruns with MAX_LOG_READS = 12 and MAX_RERUNS = 8. Its sorting and per-candidate job/log requests run in a scheduled janitor sweep, not a hot UI, socket, search, or process path, and do not rescan one full collection for each target.

Full details: Cmux Swift Concurrency

Explanation

The pull request changes only one workflow file and two Python files. The reviewed diff contains no Swift files or Swift concurrency patterns, so it does not introduce or expand the specified legacy async patterns.

Full details: Cmux Swiftpm Lockfiles

Explanation

The PR changes one workflow plus Python and test files. The workflow only checks out and runs scripts/ci/cla_quota_rerun.py; it does not change SwiftPM dependencies, Xcode package references, .gitignore, or any Package.resolved file. The SwiftPM lockfile failure conditions are therefore not triggered.

Full details: Cmux Swift Logging

Explanation

PASS: The pull request changes only a workflow, Python code, and Python tests. It adds no Swift files or Swift logging. The added Python print calls are CLI output, which is outside this Swift-specific check.

Full details: Cmux User-Facing Error Privacy

Explanation

The changed files add a scheduled GitHub Actions janitor step and an internal CI script. Its output is CI/operator diagnostic text, including GitHub and token-budget terms, not an app UI, product CLI, or product API path to a cmux end user. The repository rule expressly allows provider and configuration names in internal CI diagnostics. No user-facing error path is introduced.

Full details: Cmux Full Internationalization

Explanation

The diff changes only a scheduled GitHub Actions janitor, a Python CI helper, and offline tests. Added text is CI step metadata, operational comments, script diagnostics, and test fixtures; it is not Swift UI, web UI, API response copy, rendered markdown, changelog, or app metadata. No localization or locale-backed production surface is introduced.

Full details: Cmux Swiftui State Layout

Explanation

The check is not applicable. The pull request changes one YAML workflow and two Python files. The diff contains no SwiftUI files or SwiftUI state/layout constructs, so it does not introduce any listed violation.

Full details: Cmux Architecture Rethink

Explanation

PASS: The pull request changes only a GitHub Actions workflow, Python CI logic, and Python tests. The diff contains no Swift files or Swift architecture changes, so the Swift-specific failure conditions do not apply.

Full details: Cmux Source Artifacts

Explanation

The diff changes only an existing workflow, a hand-written CI script, and its unit test. These are intentional source, configuration, and test files. No logs, screenshots, recordings, temp directories, caches, build output, dependency checkout, or broad scratch directory enters source control.

Full details: Docstring Coverage

Explanation

Docstring coverage is 19.05% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 21 functions across 2 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 2
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔇 Additional comments: 1
scripts/ci/cla_quota_rerun.py-95-96: 1

95-96: Major · Functional Correctness · Heavy lift

⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Scan beyond the first page of workflow runs.

The API returns at most 100 runs on this page. If more than 100 newer runs precede an eligible failure, that failure never enters rotated() and can age out before any sweep examines it. Paginate through the recent-run window, or rotate the page queried across sweeps. (docs.github.com)


Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 1b5a4b7 and 3719687.

Files selected for processing: 3
  • .github/workflows/ci-queue-janitor.yml
  • scripts/ci/cla_quota_rerun.py
  • tests/test_cla_quota_rerun.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

…r and freshness gate stay as they are

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@teamleaderleo
teamleaderleo merged commit 72a8fe5 into main Oct 26, 2026
47 checks passed
@teamleaderleo
teamleaderleo deleted the gt-quota-cla-heal branch October 26, 2026 05:21
@github-actions

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Merge receipt for 2b078082b5: every check was green at merge (13 verified; 22 skipped by policy). Full suite runs on main after merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant