Conversation
comment-cop took its file list from pulls.listFiles. Right after a rebase or a merge from main, that list can still be relative to the base the PR was opened against, so it contains every file main has touched since then and the workflow flags comments the PR never added. It posted one review comment per group with no bound, so each such run kept posting until the 5 minute job timeout cancelled it (about 120 comments per push). Take the diff from repos.compareCommitsWithBasehead(base.ref...head.sha), which GitHub computes from the current merge base at request time, and post at most 25 new comments per run, logging a warning when capped.
|
Status: reproduced and fixed.
|
|
Warning Review limit reached
Next review available in: 4 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
Comment |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — targeted CI tooling fix that swaps the stale PR file cache for a live compare and caps per-run posts.
What was reviewed:
compareCommitsWithBaseheadresponse shape (files[].filename/status/patch) matches what the existing parser and dedup key consume, so downstream logic is unchanged.- Fork-PR head SHAs are reachable in the base repo via
refs/pull/N/head, so the compare againstoven-sh/bunresolves them; base refs containing/are path-encoded by octokit. per_page: 1only pages the commit list; the ≤300 files all arrive on the first page, and the 300-file ceiling is acceptable for a best-effort lint on claude-labeled PRs.- The
.slice(0, MAX_COMMENTS_PER_RUN)cap composes correctly with the existing content-hash dedup, so overflow groups are picked up on the next push rather than lost.
Extended reasoning...
Overview
Single-file change to .github/workflows/comment-cop.yml, the internal bot that flags multi-line code comments on claude-labeled PRs. Two functional edits: (1) source the changed-file list from repos.compareCommitsWithBasehead(${base.ref}...${head.sha}) instead of pulls.listFiles, because the latter can return a stale, massively inflated diff right after a rebase/merge-from-main; (2) cap new review comments to 25 per run with a core.warning when truncating. A diagnostic core.info line logs the compare's file count and merge base.
Security risks
None introduced. The workflow already runs on pull_request_target with pull-requests: write and never checks out PR code. The only new input read from the event payload is base.ref, which is controlled by the base repo, not the PR author. No shell interpolation, no new tokens, no widened permissions.
Level of scrutiny
Low. This is repo-internal CI tooling with a bounded, reversible effect (posting/resolving review comments). The failure mode being fixed is comment spam; the worst case for the new code is under-posting (300-file compare ceiling, 25-comment cap) or a failed API call, both of which degrade to "fewer nag comments" rather than anything user-facing or build-affecting. It cannot be exercised by this PR's own CI (pull_request_target runs base-branch code), and the PR description documents dry-runs against several live PRs — including one with a currently-stale cached diff — showing identical output on accurate PRs and correct output on the pathological ones.
Other factors
The response-shape compatibility (filename, status, patch) means groupsFromPatch, keyFor, and the auto-resolve logic are untouched and continue to work; compare.files ?? [] guards the one field that can be absent. The 25-per-run cap interacts safely with the existing seenKeys dedup: unposted groups remain in the diff and are posted on the next synchronize. No prior reviewer comments to address; the bug-hunting pass found nothing.
|
Updated 3:05 AM PT - Aug 13th, 2026
❌ @robobun, your commit 3259e6f has some failures in 🧪 To try this PR locally: bunx bun-pr 38127That installs a local version of the PR into your bun-38127 --bun |
…austed (#39183) ### Problem - The `Comment Cop` check is red on every claude-labeled PR whenever the repo's GraphQL quota is used up. The step dies before it scans anything: ``` GraphqlResponseError: Request failed due to following response errors: - API rate limit already exceeded for site ID installation. ##[error]Unhandled error: GraphqlResponseError: ... ``` with `x-ratelimit-resource: graphql`, `x-ratelimit-limit: 10000`, `x-ratelimit-remaining: 0` (run [31898901053](https://github.com/oven-sh/bun/actions/runs/31898901053) on #39139). All 33 comment-cop failures on Aug 15 are this error (197 runs succeeded, 171 were skipped); they hit unrelated PRs at the same time because the quota is shared by every workflow run in the repo, and they come back whenever PR volume is high. - Cause, `.github/workflows/comment-cop.yml:124` on main: the step reads the PR's existing review threads with `github.graphql()`, outside any `try`, and dedup and posting both sit behind that call. The REST call just before it (`pulls.listFiles`) had succeeded in each failing run, so the step had the diff; it only lacked the dedup data, which it was reading from the exhausted quota. - The same thread data also fed an auto-resolve of stale threads. That part does not work: under the Actions `GITHUB_TOKEN` every `resolveReviewThread` mutation fails with `Resource not accessible by integration`, after which the step still logs `Resolved N stale comment-cop thread(s).` Two successful runs from today: [31900717034](https://github.com/oven-sh/bun/actions/runs/31900717034) (20 attempted, 20 failed, "Resolved 20") and [31900354924](https://github.com/oven-sh/bun/actions/runs/31900354924) (4 attempted, 4 failed, "Resolved 4"). #36959 documents the same thing and is the PR that moves resolution to a token that can do it. So the GraphQL query was paying for one thing REST can provide and one thing that does not happen. ### Fix - Dedup reads the PR's review comments with `pulls.listReviewComments` (REST) and collects the `<!-- comment-cop:KEY -->` markers from them. Every comment the step posts starts with that marker, so the review comments carry the same keys the thread roots did; REST is the quota the step already needs for `listFiles` and `createReviewComment`, and it was available in every failing run. - The GraphQL query and the resolve loop are removed, so the step makes no GraphQL request at all; the failure in the Problem section cannot happen, rather than being caught. Nothing observable is lost: the mutations the loop issued all fail today, and the only other thing it did was log `Resolved N stale comment-cop thread(s).` after they had failed. Resolving stale threads stays with #36959, which will also need to move its thread lookup into its token step, since this PR removes the `GITHUB_TOKEN` lookup it currently reuses (noted there). - What the step posts is unchanged: the script on main and the REST dedup were dry-run (real reads, writes recorded) against 9 PRs carrying existing comment-cop threads (#35988, #36956, #36713, #35635, #33632, #35596, #39139, #30609, #36959) and chose the same comments to post on every one of them; the canned scenarios below show the same thing with GraphQL working and with it exhausted. - Test: `test/internal/source-lints/comment-cop.test.ts` extracts the script from the workflow and runs it the way `actions/github-script` does, against a fake `github` whose GraphQL requests all fail with the rate limit error above. It checks that groups not yet flagged are posted with the right line ranges, that a group already flagged (and a stale marker) are left alone, that a second run recognizes the comments the first run posted and posts nothing, and that no GraphQL request is made. Both tests fail against the workflow on main (the script throws the error above) and pass with this change. `source-lints.yml` now also triggers on changes to `comment-cop.yml`, so the test runs whenever the script is edited; it is excluded from the Buildkite shards like the rest of that directory. - Also ran: `bun bd test test/internal/source-lints/comment-cop.test.ts`, `bun test test/internal/source-lints/` (whole directory green), prettier on the three files. The comment-cop run on this PR itself still executes the script from main (`pull_request_target`), so the `Source lints` job is the one that exercises the change here. - Does not overlap with #37948 (which groups are flagged) or #38127 (where the file list comes from); both touch other parts of the script. #36959 rewrites the block this PR deletes and will need a rebase either way. ### Background - Comment Cop (`.github/workflows/comment-cop.yml`): on each push to a claude-labeled PR it reads the PR diff, finds multi-line comments added under `src/`, and posts one review comment per comment block. Each bot comment starts with `<!-- comment-cop:KEY -->`, KEY being the file path plus a hash of the block's text; a block whose KEY is already on the PR is not posted again. The check is advisory (not required for merge). - GitHub API quotas: REST and GraphQL requests count against separate hourly quotas (`x-ratelimit-resource` is `core` for REST and `graphql` for GraphQL). For the `GITHUB_TOKEN` Actions hands out, each quota is per repository, so every workflow run in oven-sh/bun draws on the same two pools, and GraphQL-heavy automation (the `gh pr` / `gh issue` / `gh search` commands used by other workflows go through GraphQL) empties the GraphQL pool for everything else when PR volume is high. - Review threads vs review comments: a review thread is GraphQL's grouping of a line comment with its replies, and is the only place a thread's id (what `resolveReviewThread` takes) and its resolved flag exist. `GET /repos/{owner}/{repo}/pulls/{n}/comments` returns every review comment on the PR, including each thread's root comment, so the markers are reachable from REST; only resolving needs GraphQL, and under `GITHUB_TOKEN` GitHub refuses that mutation regardless of the `pull-requests: write` permission. <details> <summary>Script on main vs this branch against a fake github (the fake's resolve mutation fails the way GITHUB_TOKEN's does)</summary> ``` main | quota ok, stale threads present | exit 0 | posts ["src/foo.ts:5-6"] | resolve attempts ["T_STALE_OPEN"] | graphql ["query","query","mutation"] | warnings 1 main | graphql quota exhausted, stale threads present | step fails (unhandled rate limit error) | posts [] | resolve attempts [] | graphql ["query"] | warnings 0 main | graphql quota exhausted, nothing stale | step fails (unhandled rate limit error) | posts [] | resolve attempts [] | graphql ["query"] | warnings 0 main | graphql quota exhausted, PR has no review comments yet | step fails (unhandled rate limit error) | posts [] | resolve attempts [] | graphql ["query"] | warnings 0 fixed | quota ok, stale threads present | exit 0 | posts ["src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0 fixed | graphql quota exhausted, stale threads present | exit 0 | posts ["src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0 fixed | graphql quota exhausted, nothing stale | exit 0 | posts ["src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0 fixed | graphql quota exhausted, PR has no review comments yet | exit 0 | posts ["src/foo.ts:2-3","src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0 ``` </details> <details> <summary>Dry-run against live PRs: comments the script on main would post vs the REST dedup (real reads through a user token, writes recorded)</summary> ``` PR #35988: same posts (0 vs 0) 245 threads on the PR are stale; main attempts to resolve them, this branch does not PR #36956: same posts (0 vs 0) PR #36713: same posts (1832 vs 1832) the stale cached file list that #38127 fixes; identical on both PR #35635: same posts (0 vs 0) PR #33632: same posts (0 vs 0) PR #35596: same posts (0 vs 0) PR #39139: same posts (1 vs 1) the comment the failing run above did not get to post PR #30609: same posts (0 vs 0) PR #36959: same posts (0 vs 0) ``` </details> <details> <summary>Headers from the failing run</summary> ``` errors: [ { type: 'RATE_LIMIT', code: 'graphql_rate_limit', message: 'API rate limit already exceeded for site ID installation.' } ] variables: { owner: 'oven-sh', repo: 'bun', pr: 39139, after: null } 'x-ratelimit-limit': '10000' 'x-ratelimit-remaining': '0' 'x-ratelimit-resource': 'graphql' 'x-ratelimit-used': '10000' ``` </details>
This branch has not been deployed
2 participants
What does this PR do?
Problem
If you need a paragraph-long comment to justify why the workaround is OK, the code is wrongreview threads on files a PR does not touch. On fetch: reject network errors as TypeError('fetch failed' / 'terminated') with the system error as cause #35988 two runs (31678330598, 31680070931) posted 240 threads across 67 files, none of them in the PR's 20-file diff. bundler: merge duplicate external ESM imports across the files of a chunk #35635 (6-file PR, 172 such threads), js_parser: lower undecorated auto-accessors to a native #-private storage field #35708 (2 files, 154), bundler: resolve require('bindings')(name) to the package's .node addon at build time #36713 (11 files, 123), watcher(windows): watch workspace packages outside the project root #35596 (8 files, 65) and transpiler: compile zod v4 schemas into lazy wrappers with flat validators #36956 (21 files, 59) got the same treatment..github/workflows/comment-cop.yml:92: the file list comes frompulls.listFiles. The workflow runs onsynchronize, and right after a rebase or a merge from main the PR's cached diff can still be relative to the base the PR was opened against, so it lists every file main touched since then. fetch: reject network errors as TypeError('fetch failed' / 'terminated') with the system error as cause #35988 reportedchanged_files: 3133against a base from July 25 (git diff 44f6469e..90ef0cdcis exactly 3133 files) for over an hour; bundler: resolve require('bindings')(name) to the package's .node addon at build time #36713 still reports 1946 files against an August 1 base while its live diff is 11 files..github/workflows/comment-cop.yml:170: onecreateReviewCommentper group with nothing bounding the count, so every such run posted until the job'stimeout-minutes: 5cancelled it (about 120 threads per push, both runs above endedcancelled). Because the inflated list is walked in path order, the run was usually cancelled before reaching the files the PR did change, so the comments it exists to post were not posted either.Fix
repos.compareCommitsWithBaseheadwithbase.ref...head.sha. GitHub computes that from the current merge base at request time, so it is the PR's actual diff no matter what the PR's cached diff says; the file entries have the same shape (filename,status,patch) aspulls.listFiles, so the parser and the dedup keys are unchanged. Requested withper_page: 1: the changed files all come on the first page (at most 300),per_pageonly pages the commit list.MAX_COMMENTS_PER_RUN), with a warning annotation when the cap is hit, so a run's write volume is bounded by design instead of by the job timeout. Groups left over are still in the diff on the next push and get posted then (dedup is per group, unchanged). Of about 30 claude PRs I scanned with the new script, all but one have 12 or fewer groups in their real diff; the exception, watcher(windows): watch workspace packages outside the project root #35596, has 40 (33 of them in one file) and would get 25 on the first run and the other 15 on the push after.presentKeysnow comes from the real diff, the existing auto-resolve step identifies the bogus threads as stale (240 on fetch: reject network errors as TypeError('fetch failed' / 'terminated') with the system error as cause #35988) and resolves them once ci: use a PAT to resolve stale comment-cop review threads #36959 gives it a token that can.How did you verify your code works?
The workflow runs on
pull_request_targetfrom the base branch, so CI on this PR does not exercise it. I dry-ran the embedded script (extracted from the YAML, real GitHub reads,createReviewCommentand the resolve mutation recorded instead of sent) against live PRs:changed_files: 1946, live compare 11 files): the script on main would post 1832 comments across 396 files; this branch's script posts 7 comments across 5 files, all in the PR's diff.repos.compareCommitsWithBaseheadis present in the pinnedactions/github-scriptbundle (GET /repos/{owner}/{repo}/compare/{basehead}); a fork PR's head SHA works as the head of the compare (checked against fix(types): correct FormData values()/entries() iterator types #34264 and install: serve lifecycle node-gyp from the install cache after EXDEV #38080), and octokit's percent-encoding of a base branch containing/is accepted by the endpoint. Prettier passes on the file.Does not overlap with #37948 (which groups count as a comment) or #36959 (token used to resolve threads); both touch other parts of the same script.
Per-PR audit of existing Comment Cop threads against each PR's live compare
For #35635, #35708 and #36713, 43 of 45, 53 of 54 and 37 of 40 of the commented-on paths were never touched by any non-merge commit on the branch (
git log --no-merges --name-only origin/main..<head>), and the threads were posted in one burst per push lasting until the 5 minute timeout (for example #35635: 160 threads between 03:08 and 03:13 UTC on Aug 13, right after a rebase).pr.changed_filesfor most of these has since been recomputed to the right number; the threads were posted while it was not.