Skip to content

ci: let comment-cop resolve its stale review threads - #40468

Closed
robobun wants to merge 1 commit into
mainfrom
farm/150beb6e/comment-cop-resolve-permission
Closed

robobun wants to merge 1 commit into
mainfrom
farm/150beb6e/comment-cop-resolve-permission

Conversation

@robobun

@robobun robobun commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Comment Cop never resolves its own review threads after the flagged comment block leaves the diff. Every mutation fails with resolveReviewThread failed for PRRT_...: Resource not accessible by integration, then the step logs Resolved 10 stale comment-cop thread(s). anyway (run 32848460003 on css: keep incompatible selectors that fail the same way in one rule #40464). Someone has to resolve the stale threads by hand.
  • Cause: GitHub gates resolveReviewThread on the token's contents permission, not on pull-requests. .github/workflows/comment-cop.yml:12 grants contents: read.

Fix

  • Grant contents: write. The job checks out no code and runs a fixed script from the base branch, so PR content never reaches the wider token.
  • Count the mutations that succeed and log Resolved X of Y. A failure is a warning, not a resolved thread.
  • Correct because the same mutation was probed under both grants with one script, a minute apart: contents: read fails with this exact error, contents: write succeeds (open-code-review#27, run links in Notes). This supersedes the PAT in ci: use a PAT to resolve stale comment-cop review threads #36959: the Actions token is not blocked, it is under-scoped.
  • Verified: test/internal/source-lints/comment-cop.test.ts runs the embedded script against a fake GitHub and pins the permission. All 3 tests fail on main.

Background

  • pull_request_target runs the base branch's workflow file with a write-capable GITHUB_TOKEN. The permissions block sets that token's scope. The risk with this event is to run PR code under that token. This job never does.
  • A review thread is GraphQL's grouping of a line comment and its replies. Its id and resolved flag exist only there, so resolving needs the GraphQL mutation.
  • source-lints.yml runs test/internal/source-lints/ against a released bun. It now also triggers on comment-cop.yml changes.
Notes
  • Probe runs: 31245782000 (contents: read + pull-requests: write, RESOLVE_RESULT=FAILED ... Resource not accessible by integration) and 31245826522 (contents: write + pull-requests: write, RESOLVE_RESULT=OK). Community discussion 44650 reports the same requirement for GitHub App tokens.
  • The probe also reported viewerCanResolve: false in both runs, including the one that succeeded, so that field cannot serve as a pre-flight check.
  • pull_request_target runs the base branch's copy of the workflow, so the Comment Cop run on this PR still executes the script from main. The change takes effect after merge. The Source lints job is what exercises the new test here.
  • Related open PRs on this workflow: ci: use a PAT to resolve stale comment-cop review threads #36959 (moves resolution to ROBOBUN_TOKEN, a long-lived PAT with push access to other oven-sh repos), ci: keep comment-cop working when GITHUB_TOKEN's GraphQL quota is exhausted #39183 (drops the resolve loop because it always failed, and moves dedup to REST to survive GraphQL quota exhaustion). With this change the loop works, so ci: keep comment-cop working when GITHUB_TOKEN's GraphQL quota is exhausted #39183 would need to keep it and guard the reviewThreads query instead of removing it.
  • Fail-before: with the workflow as on main, the test reports Expected: "write", Received: "read" and Received: "Resolved 2 stale comment-cop thread(s).".
  • Also ran bun test test/internal/source-lints/ (168 pass) and bun bd test test/internal/source-lints/comment-cop.test.ts.

GitHub gates the resolveReviewThread GraphQL mutation on the token's
contents permission, not on pull-requests. The workflow granted
contents: read, so every mutation failed with "Resource not accessible
by integration" and stale comment-cop threads stayed open on every
claude-labeled PR. Grant contents: write. The job checks out no code.

Count the mutations that succeed and log "Resolved X of Y" instead of
reporting every attempt as resolved.

Add a source lint that runs the embedded script against a fake GitHub
and pins the permission, and run the lints when the workflow changes.
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

On-demand reviews are free for the next 26 days. After that, they cost $0.25 per reviewed file.

Or wait 7 minutes for your next included review.

View limit details

Limit details: You’ve used all 5 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a85672ef-c459-4562-81ac-46449dc88400

📥 Commits

Reviewing files that changed from the base of the PR and between adc354d and 4485e44.

📒 Files selected for processing (3)
  • .github/workflows/comment-cop.yml
  • .github/workflows/source-lints.yml
  • test/internal/source-lints/comment-cop.test.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: closed, superseded by #39183.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this and didn't find any bugs. The logic and tests check out, but since it widens GITHUB_TOKEN to contents: write on a pull_request_target workflow, a maintainer should sign off on the permission change.

What was reviewed:

  • Confirmed the job has no actions/checkout step and the embedded script runs only GitHub API calls from the base-branch workflow file, so the write token is not exposed to PR-controlled code.
  • Traced the new resolved counter and log message — accurate on both the all-succeed and partial-reject paths.
  • Walked the test's fake-GitHub harness against the embedded script: presentKeys/seenKeys/toResolve derive as expected, and all three tests fail on main's workflow.
Extended reasoning...

Overview

This PR touches CI configuration only: it flips permissions.contents from read to write in .github/workflows/comment-cop.yml, adds an accurate success counter to the resolve loop's log line, wires comment-cop.yml into the source-lints.yml path triggers, and adds test/internal/source-lints/comment-cop.test.ts which parses the workflow YAML, extracts the embedded github-script body, and executes it against a mock github/context/core to pin the permission and the resolve behavior. No runtime or product code is affected.

Security risks

The substantive risk is granting contents: write to a pull_request_target workflow. That combination is the classic vector for fork-PR privilege escalation when the workflow checks out and runs PR head code. Here the job has no checkout step at all — its only step is actions/github-script with a script literal that lives in the base-branch workflow file, and it interacts with the PR solely via REST/GraphQL. The if: guard also pins github.repository == 'oven-sh/bun'. Under those constraints the widened token cannot be reached by untrusted code. That said, the blast radius if this reasoning were wrong (or if a later edit added a checkout) is write access to the repo, so this is exactly the kind of change a human maintainer should explicitly approve rather than an automated reviewer.

Level of scrutiny

High, because it is a permission grant on a fork-triggerable workflow. The code change itself is trivial (one YAML value, one counter, two path-filter lines) and the test is well-constructed — it actually executes the embedded script the same way actions/github-script does and asserts exact resolve attempts and log output, and it fails on main. But per the approval guidelines, security-sensitive permission changes should not be auto-approved regardless of how sound the accompanying analysis is.

Other factors

The PR description cites an external probe demonstrating the contents gating and a community discussion corroborating it, and notes two overlapping open PRs (#36959, #39183) that take different approaches — a maintainer will want to reconcile those anyway. No prior human review comments exist on the timeline.

@robobun

robobun commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Agreed that the permission grant is the part that needs a human decision, which is why the PR is assigned to @Jarred-Sumner rather than self-merging.

For that decision, the relevant facts in one place: the job has no checkout step and runs a fixed script from the base branch. GITHUB_TOKEN is scoped to this repository and expires when the job ends. The alternative on the table, #36959, puts ROBOBUN_TOKEN (a long-lived PAT that also pushes to other oven-sh repos) into the same pull_request_target workflow, so this change is the smaller exposure of the two ways to make the resolve path work.

@robobun

robobun commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 6:27 AM PT - Aug 25th, 2026

❌ @robobun, your commit 4485e44 has 3 failures in Build #105660 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 40468

That installs a local version of the PR into your bun-40468 executable, so you can run:

bun-40468 --bun

@robobun

robobun commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this PR. A self-review of the change made a case against it that holds up, even though the diagnosis (the mutation needs contents: write) is correct.

  • The stale threads already get resolved without this loop. On the 15 most recent merged claude PRs, all 61 comment-cop threads are resolved, 60 by robobun and 1 by a maintainer. A wider sample of about 300 threads across 110 PRs shows the same. The 10 threads from the run in the description were resolved by robobun within the hour. The set this loop would clear (unresolved and outdated) is empty on every PR sampled. The only residue is threads on abandoned branches.
  • The grant goes against the repo's convention. The other three pull_request_target workflows run with contents: read or permissions: {}, and close-linked-issues.yml spells out why read-only matters for that trigger. A standing contents: write on a workflow that five open PRs are editing is a poor trade for clearing threads nobody waits on.
  • It works against ci: keep comment-cop working when GITHUB_TOKEN's GraphQL quota is exhausted #39183. That PR removes the GraphQL query and the resolve loop so the check survives GraphQL quota exhaustion, which is a real failure (33 red runs on Aug 15). Keeping the loop alive would force that PR to keep a GraphQL dependency it is better off without.

#39183 is the direction to take for this workflow. It already includes the source-lints.yml trigger for comment-cop.yml and a test for the embedded script. If in-workflow cleanup is still wanted later, pulls.deleteReviewComment on the bot's own stale, reply-less comments needs only pull-requests: write and no GraphQL.

@robobun robobun closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants