feat(code reviewer): tell customers when their own provider key is rate limited - #4806
Merged
Conversation
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryThe new commit adds the previously-requested GitLab commit-status test coverage for Files Reviewed (1 files)
Previous Review Summary (commit 9b085bc)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 9b085bc)Status: 1 Issue Found | Recommendation: Address before merge Executive SummaryThe new BYOK rate-limit customer-copy logic is correctly gated and covered by tests for the GitHub check-run path, but the equivalent GitLab commit-status branch lacks test coverage. Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (7 files)
Reviewed by claude-sonnet-5 · Input: 26 · Output: 4.6K · Cached: 525.8K Review guidance: REVIEW.md from base branch |
pandemicsyn
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A code review that failed because the customer's own provider API key hit its
rate limit showed them a generic "Kilo Code Review failed" with the raw internal
message "Assistant request was rate limited". Nothing indicated the cause was on
their side rather than ours.
This adds customer-facing copy for
assistant_rate_limited_byokand uses iteverywhere that failure surfaces. It is notification only: no PR comment, no
email, no disabling of Code Reviewer.
The terminal reason it keys on only started being recorded in #4805, which is
what made this possible.
Changes
terminal-reason-copy.ts, mapping a terminal reason to customer-facingcopy. Only
assistant_rate_limited_byokis in it, with the message "Yourprovider API key hit its rate limit."
supersededpatternof replacing the label and message. Renders muted rather than destructive red,
since nothing is broken on our side.
error_messageverbatim.status description.
hasKnownUnretryableTerminalReason.(Kilo key)".
Verification
No manual testing. Reproducing this needs a real provider rate limit against a
customer's own key, which I cannot trigger on demand. Covered by tests instead:
terminal-reason-copysuite, 5 tests, including that managed andunqualified rate limits get no copy, and that the map does not overlap the
action-required reasons
code-review-statusroute suite, 119 tests, including that a byok ratelimit gets the new check run copy without an
action_requiredconclusion,that it is not auto-retried, and that a managed one still is
Visual Changes
Not captured. Two app surfaces change, but reproducing them needs a real byok
rate limit failure, so I could not take before and after screenshots.
What changes, for review by reading:
Error: Assistant request was rate limited, destructive redRate limited: Your provider API key hit its rate limit., mutedError: Assistant request was rate limited, destructive redRate limited: Your provider API key hit its rate limit., mutedKilo Code Review failed/Review failed: Assistant request was rate limitedKilo Code Review rate limited/Your provider API key hit its rate limit.Rate Limited (managed key)Rate Limited (Kilo key)Reviewer Notes
hasKnownUnretryableFailureMessagealready tried to catch byok rate limits,but only by matching the raw
[BYOK] Your API key has hit its rate limit...text. Safe-failure projection rewrites that to
Assistant request was rate limitedbefore the callback ever sees it, so that check has never fired.Someone had already decided these should not be retried; the structured reason
makes that decision take effect. Managed key rate limits stay retryable, since
our own capacity can free up.
CODE_REVIEW_ACTION_REQUIRED_REASONSalso disables Code Reviewer for thatowner, which is wrong for a transient failure and would hit many customers at
once during a provider wide event. There is precedent: the auto-disable in
disableCodeReviewForRepeatedCloneTimeoutsTodayis commented out for exactlythat reason.
assistant_rate_limited_managedgets no customer-facing copy on purpose. Itmeans the request used Kilo's credentials, which currently conflates our
upstream quota running out with our own abuse rules throttling the request.
Those mean different things to a customer, so any single sentence would be
wrong in one of the two cases. Splitting them is a separate change in
cloud-agent-next, reading the structured
error_typethe abuse responsealready carries, with no change to abuse behavior.
terminal_reasonvalues areunchanged. "managed key" required knowing that
managedis the internal namefor our own credential, which is the wrong thing to have to recall while
triaging, given one of the two should page us and the other should not.