Skip to content

Feat(code review) keep and map error codes - #4788

Merged
St0rmz1 merged 3 commits into
mainfrom
feat/code-review-failure-classification
Jul 27, 2026
Merged

Feat(code review) keep and map error codes#4788
St0rmz1 merged 3 commits into
mainfrom
feat/code-review-failure-classification

Conversation

@St0rmz1

@St0rmz1 St0rmz1 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

cloud-agent-next already sends a structured failure object on the code review
status callback ({ stage, code, subtype, attempts, message }), but the
receiver only read failure.code for a single case (workspace_setup_failed
combined with sandbox_storage_full) and ignored it otherwise. Everything else
was stored with a NULL terminal_reason and only a human-readable sentence in
error_message, so the admin Error Analysis had to recover the category by
pattern-matching that sentence. Roughly a quarter of failures matched nothing
and fell into "Other".

This maps every CloudAgentFailureCode and WorkspaceFailureSubtype onto a
terminal reason at the callback boundary, so classification no longer round
trips through English. terminal_reason is an untyped text column with no
constraint, so widening the allowlist needs no migration.

Receiver side only. cloud-agent-next is unchanged. Existing rows keep their NULL
terminal_reason and continue to use the message fallback; there is no backfill.

Changes

  • Add apps/web/src/lib/code-reviews/terminal-reason-from-failure.ts, mapping
    all 20 failure codes and 13 workspace subtypes to a terminal reason. Both
    tables use satisfies Record<...>, so adding a code upstream without
    categorizing it here is a type error rather than a silent fall through to
    "Other".
  • Split assistant_error by exact match on the safe messages produced by
    classifyAssistantFailure. That code covers every provider failure including
    rate limiting, which is the largest single bucket, so a 1:1 mapping would have
    merged rate limits into a generic category and made the chart worse. Backed by
    a Map rather than an object literal because the key is caller supplied text
    and an object lookup for constructor or __proto__ returns a truthy
    inherited member.
  • Add 18 terminal reasons to CODE_REVIEW_TERMINAL_REASONS. Only
    setup_command_failed is marked benign, since that is the customer's own
    setup script. The rest stay as system failures so they still count toward
    error spike alerting.
  • Replace the CloudAgentTerminalReason type union in worker-utils with a
    runtime CLOUD_AGENT_TERMINAL_REASONS array and derive the type from it, then
    build the zod enum in services/code-review-infra/src/types.ts from that
    import. That schema ends in .catch(undefined), so any reason missing from
    its hand-maintained copy was silently coerced away and lost from the
    orchestrator's in-memory state while the DB row kept the correct value.
  • Re-apply the billing and model-not-found status normalization when the reason
    comes from the structured payload. Both existing checks key off error text,
    and a structured model_missing failure carries the generic "No model was
    selected", so status would stay failed instead of cancelled and the
    customer would see generic copy instead of the actionable message.
  • Collapse the two duplicated 20 line SQL CASE expressions in the admin router
    into a single buildErrorCategoryExpr(terminalReasonColumn, errorMessageColumn),
    following the existing excludeBillingErrorCondition pattern. It reads
    terminal_reason first and keeps the message patterns underneath for older
    rows.

Verification

No manual testing. The change sits on the internal status callback, which needs
a real cloud-agent-next failure of each code to exercise end to end. Covered by
unit tests instead:

  • apps/web affected suites (460 tests), including 4 new callback tests
    asserting wrapper_failed, assistant_rate_limited, cancelled status for
    model_missing, and failed status for payment_required
  • Exhaustiveness tests over every failure code and workspace subtype
  • Parity test between CODE_REVIEW_TERMINAL_REASONS and
    CLOUD_AGENT_TERMINAL_REASONS, which live in packages that cannot import
    each other
  • worker-utils (323) and code-review-infra (57) suites

Visual Changes

N/A. No component or layout changes. The admin Error Analysis chart will show
new category names once failures start arriving with a structured reason, but
that needs production data to screenshot.

Reviewer Notes

  • sandbox_error and interrupted are matched last in the SQL, after the
    message patterns. Both are generic buckets rather than causes, so matching
    them earlier would mask the specific category on existing rows. This is the
    behavior the admin router tests cover.
  • Label and reason strings in the SQL are inlined as escaped literals rather
    than bound parameters. The expression appears in both SELECT and GROUP BY, and
    parameter placeholders are renumbered by position, so Postgres would stop
    recognizing the two as the same expression.
  • The assistant message table is exact whole string matching, not substring. If
    a message in classifyAssistantFailure is reworded, the map stops matching and
    falls back to the generic assistant_failed, which is no worse than current
    behavior. The cleaner fix is to carry the assistant reason in the structured
    failure itself, which would be a sender side change.

Comment thread apps/web/src/app/api/internal/code-review-status/[reviewId]/route.test.ts Outdated
@kilo-code-bot

kilo-code-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The test now omits errorMessage so the structured payment_required failure path is genuinely exercised instead of being short-circuited by the pre-existing billing text heuristic, resolving the previously flagged issue; no new issues found in the incremental diff.

Files Reviewed (1 file)
  • apps/web/src/app/api/internal/code-review-status/[reviewId]/route.test.ts
Previous Review Summary (commit 68034d6)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 68034d6)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

One new test (payment_required structured-failure case) doesn't actually exercise the code path it claims to verify, since a pre-existing message heuristic already sets the terminal reason first.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1
Issue Details (click to expand)

SUGGESTION

File Line Issue
apps/web/src/app/api/internal/code-review-status/[reviewId]/route.test.ts 1377 Test's errorMessage matches the pre-existing billing text heuristic, so terminalReasonFromCloudAgentFailure is never actually invoked, and the intended structured payment_required path is untested
Files Reviewed (8 files)
  • apps/web/src/app/api/internal/code-review-status/[reviewId]/route.ts - 0 issues
  • apps/web/src/app/api/internal/code-review-status/[reviewId]/route.test.ts - 1 issue
  • apps/web/src/lib/code-reviews/terminal-reason-from-failure.ts - 0 issues
  • apps/web/src/lib/code-reviews/terminal-reason-from-failure.test.ts - 0 issues
  • apps/web/src/routers/admin-code-reviews-router.ts - 0 issues
  • packages/db/src/schema-types.ts - 0 issues
  • packages/worker-utils/src/cloud-agent-next-client.ts - 0 issues
  • services/code-review-infra/src/types.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by claude-sonnet-5 · Input: 24 · Output: 3.1K · Cached: 425.3K

Review guidance: REVIEW.md from base branch main

@St0rmz1
St0rmz1 merged commit 7bde012 into main Jul 27, 2026
65 checks passed
@St0rmz1
St0rmz1 deleted the feat/code-review-failure-classification branch July 27, 2026 14:30
iscekic added a commit that referenced this pull request Jul 27, 2026
…-audit-w1-pr-safety

WORKFLOW_LEARNINGS.md: both sides appended Orchestrator entries; kept all
three (PR-review E2E env traps + EXITCODE false-trigger + Kilobot
no-findings reading). #4788's code-review error-code changes touch only
the review-status route, disjoint from this PR's surface — behavior-neutral
resolution, no reruns per the base-advance rule.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants