ci: publish an honest automated review status - #3991
Conversation
|
Warning Review limit reached
Next review available in: 22 minutes Limit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📦 Client bundle boundary
A server module in a client graph aborts hydration in the browser. New leaks fail CI; known leaks are tracked in |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d748d92f81
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Resolve the deno.json test:scripts conflict by keeping the main branch entry and re-adding scripts/ci/automated-review-gate.test.ts to the script suite.
Read the gate from the default branch instead of the pull request base commit. Every pull request opened before this gate lands has a base commit that predates scripts/ci/automated-review-gate.mjs, so the checkout produced ERR_MODULE_NOT_FOUND and the job crashed before publishing any status. A crash publishes nothing, which is the same false green the gate exists to remove. The default branch is equally trusted under pull_request_target and always carries the current gate. Publish a pending status for draft pull requests instead of returning early. Review bots skip drafts, so a draft has no verdict yet. Returning early left no "Automated review" context at all, which is indistinguishable from a workflow that never ran. Pending says "not decided yet" and can never read as a pass. Rerun the gate when a review is dismissed. The classifier rejects DISMISSED reviews, so dismissing a qualifying bot review used to leave a stale success status in place until the next push. Skip the pull_request_review trigger on fork pull requests. GitHub gives that event a read-only GITHUB_TOKEN for pull requests from forks, so createCommitStatus returns 403 and the run dies without updating anything. The trusted pull_request_target and issue_comment runs still publish the decision, so the fork case stays fail-closed rather than crashing. Add tests for the fail-closed transport-error path and the draft pending path, and pin the workflow contract to the new checkout ref, the dismissed trigger, the fork guard, and the draft flag.
The gate is imported from the default branch checkout. If that file is missing or broken, the dynamic import threw and the job died before publishing anything, so the pull request showed no `Automated review` context at all. No context looks the same as a skipped review, which is the exact confusion this workflow exists to remove. Catch the import failure, publish `failure` with a description that names the cause, and mark the job failed. The workflow contract test asserts the fallback, so removing it turns the suite red.
2 participants
Summary
Automated reviewcommit status on the exact pull request headpending, so "not reviewed yet" never renders as a pass and never as a missing statuspull_request_targetauthorityThe states
successsuccessfailurefailure, recomputed on thedismissedeventfailurepending"No findings" and "did not run" never render the same, and the lookup fails closed: a throw inside the review lookup still publishes
failure, it does not skip the status.Red and green
The workflow contract test initially failed because no review gate existed. Live reproduction confirmed the false-green condition: CodeRabbit can report success while posting only a rate-limit comment and no review.
The classifier was verified against live API payloads:
CodeRabbitcheck reports pass, but the only comment is a rate-limit warning, so the gate failsFork pull requests
GitHub gives
pull_request_reviewa read-onlyGITHUB_TOKENfor pull requests from forks, socreateCommitStatuswould return 403 there. The job skips that trigger when the head repo is not this repo. Fork pull requests still get their status frompull_request_targetandissue_comment, which both keep write authority, so the fork path stays fail-closed rather than crashing.Verification
deno task typecheckdeno task lint:cideno task fmt:checkdeno task test:file scripts/ci/automated-review-gate.test.tsnode --check scripts/ci/automated-review-gate.mjssuccessturns the test suite redAfter merge, repository rules can require the
Automated reviewstatus context if enforcement at merge time is desired. Even without that rule, skipped reviews are no longer indistinguishable from green reviews in the pull request status surface.Refs veryfront/veryfront-issue-inbox#724