Skip to content

feat(code reviewer): tell customers when their own provider key is rate limited - #4806

Merged
St0rmz1 merged 2 commits into
mainfrom
feat/byok-rate-limit-customer-copy
Jul 27, 2026
Merged

feat(code reviewer): tell customers when their own provider key is rate limited#4806
St0rmz1 merged 2 commits into
mainfrom
feat/byok-rate-limit-customer-copy

Conversation

@St0rmz1

@St0rmz1 St0rmz1 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

A code review that failed because the customer's own provider API key hit its
rate limit showed them a generic "Kilo Code Review failed" with the raw internal
message "Assistant request was rate limited". Nothing indicated the cause was on
their side rather than ours.

This adds customer-facing copy for assistant_rate_limited_byok and uses it
everywhere that failure surfaces. It is notification only: no PR comment, no
email, no disabling of Code Reviewer.

The terminal reason it keys on only started being recorded in #4805, which is
what made this possible.

Changes

  • Add terminal-reason-copy.ts, mapping a terminal reason to customer-facing
    copy. Only assistant_rate_limited_byok is in it, with the message "Your
    provider API key hit its rate limit."
  • Use it on the review detail page, following the existing superseded pattern
    of replacing the label and message. Renders muted rather than destructive red,
    since nothing is broken on our side.
  • Use it in the Code Reviews list, which previously rendered the raw
    error_message verbatim.
  • Use it for the GitHub check run title and summary, and for the GitLab commit
    status description.
  • Stop auto-retrying byok rate limits by adding the reason to
    hasKnownUnretryableTerminalReason.
  • Rename the admin chart label "Rate Limited (managed key)" to "Rate Limited
    (Kilo key)".

Verification

No manual testing. Reproducing this needs a real provider rate limit against a
customer's own key, which I cannot trigger on demand. Covered by tests instead:

  • terminal-reason-copy suite, 5 tests, including that managed and
    unqualified rate limits get no copy, and that the map does not overlap the
    action-required reasons
  • code-review-status route suite, 119 tests, including that a byok rate
    limit gets the new check run copy without an action_required conclusion,
    that it is not auto-retried, and that a managed one still is
  • Affected apps/web suites, 477 tests across 30 files

Visual Changes

Not captured. Two app surfaces change, but reproducing them needs a real byok
rate limit failure, so I could not take before and after screenshots.

What changes, for review by reading:

Surface Before After
Review detail page Error: Assistant request was rate limited, destructive red Rate limited: Your provider API key hit its rate limit., muted
Code Reviews list Error: Assistant request was rate limited, destructive red Rate limited: Your provider API key hit its rate limit., muted
GitHub check run Kilo Code Review failed / Review failed: Assistant request was rate limited Kilo Code Review rate limited / Your provider API key hit its rate limit.
Admin error analysis Rate Limited (managed key) Rate Limited (Kilo key)

Reviewer Notes

  • The retry change is a behavior change worth its own look.
    hasKnownUnretryableFailureMessage already tried to catch byok rate limits,
    but only by matching the raw [BYOK] Your API key has hit its rate limit...
    text. Safe-failure projection rewrites that to Assistant request was rate limited before the callback ever sees it, so that check has never fired.
    Someone had already decided these should not be retried; the structured reason
    makes that decision take effect. Managed key rate limits stay retryable, since
    our own capacity can free up.
  • Deliberately not action required. Adding a reason to
    CODE_REVIEW_ACTION_REQUIRED_REASONS also disables Code Reviewer for that
    owner, which is wrong for a transient failure and would hit many customers at
    once during a provider wide event. There is precedent: the auto-disable in
    disableCodeReviewForRepeatedCloneTimeoutsToday is commented out for exactly
    that reason.
  • assistant_rate_limited_managed gets no customer-facing copy on purpose. It
    means the request used Kilo's credentials, which currently conflates our
    upstream quota running out with our own abuse rules throttling the request.
    Those mean different things to a customer, so any single sentence would be
    wrong in one of the two cases. Splitting them is a separate change in
    cloud-agent-next, reading the structured error_type the abuse response
    already carries, with no change to abuse behavior.
  • The admin label rename is display only. Stored terminal_reason values are
    unchanged. "managed key" required knowing that managed is the internal name
    for our own credential, which is the wrong thing to have to recall while
    triaging, given one of the two should page us and the other should not.

@kilo-code-bot

kilo-code-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The new commit adds the previously-requested GitLab commit-status test coverage for assistant_rate_limited_byok, correctly asserting the customer-facing description; no new issues found.

Files Reviewed (1 files)
  • apps/web/src/app/api/internal/code-review-status/[reviewId]/route.test.ts
Previous Review Summary (commit 9b085bc)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 9b085bc)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The new BYOK rate-limit customer-copy logic is correctly gated and covered by tests for the GitHub check-run path, but the equivalent GitLab commit-status branch lacks test coverage.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1
Issue Details (click to expand)

SUGGESTION

File Line Issue
apps/web/src/app/api/internal/code-review-status/[reviewId]/route.ts 888 New getGitLabStatusDescription branch for assistant_rate_limited_byok has no GitLab-focused test, unlike the GitHub check-run path
Files Reviewed (7 files)
  • apps/web/src/app/(app)/code-reviews/[reviewId]/CodeReviewDetailClient.tsx - 0 issues
  • apps/web/src/app/api/internal/code-review-status/[reviewId]/route.test.ts - 0 issues
  • apps/web/src/app/api/internal/code-review-status/[reviewId]/route.ts - 1 issue
  • apps/web/src/components/code-reviews/CodeReviewJobsCard.tsx - 0 issues
  • apps/web/src/lib/code-reviews/terminal-reason-copy.test.ts - 0 issues
  • apps/web/src/lib/code-reviews/terminal-reason-copy.ts - 0 issues
  • apps/web/src/routers/admin-code-reviews-router.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by claude-sonnet-5 · Input: 26 · Output: 4.6K · Cached: 525.8K

Review guidance: REVIEW.md from base branch main

@St0rmz1
St0rmz1 merged commit 84e1ff1 into main Jul 27, 2026
20 checks passed
@St0rmz1
St0rmz1 deleted the feat/byok-rate-limit-customer-copy branch July 27, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants