CodeRouter: hold capacity errors on the same model instead of failing fast

[teamleaderleo]

A capacity blip ended autonomous agent turns. When every Claude or Codex account was cooling, or four accounts had failed, CodeRouter answered 503 overloaded_error. Codex and Claude Code show that as a final error and stop until a human types "continue" (#15301).

CodeRouter now holds the request and retries the same model instead of failing fast:

• When a routing round ends on a transient failure (429, 5xx/529, an overloaded or "model at capacity" SSE event before any output, a transport error), the proxy waits and replays the same body. Waits use jittered exponential backoff (0.5 s → 15–30 s) and never end before the soonest account cooldown, which already carries the upstream retry-after.
• The hold lasts up to CODEROUTER_CAPACITY_HOLD_MS (default 20 minutes). It is capped by the existing 25-minute header budget, so the function still answers before its 30-minute maxDuration.
• It never holds once response bytes reached the client, and it never substitutes a model.
• It fails at once, as today, when no account recovers within the budget: every account has a revoked credential, or a Codex workspace quota resets after the budget. Claude's selector skips invalid_credential cooldowns when computing the recovery time; Codex gets a small nextCapacityAvailableAt query for the same purpose.
• route_events gains held_ms and hold_count, and the PostHog trace carries coderouter_held_ms / coderouter_hold_count, so capacity pain shows up as latency.

Deploy order: ClickHouse migration web/db/clickhouse/006_route_events_capacity_hold.sql (additive ADD COLUMN IF NOT EXISTS ... DEFAULT 0) must be applied to coderouter_dev and coderouter before merge. Otherwise route-event inserts carry unknown columns.

Part 2 of #15301 (agents auto-resume after a retryable failure) is a separate PR.

Validation

• Commit 1 adds the tests only; commit 2 adds the change.
• bun test tests/coderouter-*.test.ts*: 506 pass, 0 fail (rebased on main). CI: commit 1 red (web typecheck and web tests), commit 2 green (59/59). New tests: a 529 → 429 → transport → 529 → 200 storm on one Claude account returns 200 with four holds and the same model on every replay; the same for Codex, with each wait honoring the recorded cooldown; the budget cap; no hold for a revoked credential or an hour-long quota; no replay after output started.
• tsc --noEmit clean; lint:complexity clean.
• Production build against a mocked upstream (local next build + next start; this machine cannot reach cmux's Vercel previews). A throwaway, unpushed branch swaps in stub auth, one stub account with in-memory cooldowns, and a mock upstream, then drives the real /v1/messages and /v1/responses routes. Every storm shorter than the budget reached the client as a 200 on the same model; the same build with CODEROUTER_CAPACITY_HOLD_MS=0 returns the 529 in 2.3 s, which is the old behavior.

Transcript

$ # Claude: 3 x 529, then 200 (hold on, default 20 min budget)
=== claude-529 (3 x 529) after 94s
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"held and served"}}
HTTP 200 in 93.024501s

$ # Claude: 2 x 429 retry-after 5, then 200 (same account, concurrent)
=== claude-429 (2 x 429) after 68s
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"held and served"}}
HTTP 200 in 67.095940s

$ # Codex: 2 x "Selected model is at capacity" SSE event, then 200
data: {"type":"response.output_text.delta","delta":"held and served"}
HTTP 200 in 121.558265s
elapsed 122s

# server log (mock upstream)
[mock-upstream +   0.0s] claude-529 client request received (failures=3, kind=529)
[mock-upstream +   1.7s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +   1.7s] claude-529 attempt 1 -> 529
[mock-upstream +   1.7s] claude-529 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +   0.0s] claude-429 client request received (failures=2, kind=429)
[mock-upstream +  20.7s] claude-429 upstream POST model=claude-sonnet-4-5
[mock-upstream +  20.7s] claude-429 attempt 1 -> 429
[mock-upstream +  20.7s] claude-429 cooldown verify-claude-1 5000 ms (rate_limited)
[mock-upstream +  25.9s] claude-429 upstream POST model=claude-sonnet-4-5
[mock-upstream +  25.9s] claude-429 attempt 2 -> 429
[mock-upstream +  25.9s] claude-429 cooldown verify-claude-1 5000 ms (rate_limited)
[mock-upstream +  36.1s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +  36.1s] claude-529 attempt 2 -> 529
[mock-upstream +  36.1s] claude-529 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +  61.6s] claude-429 upstream POST model=claude-sonnet-4-5
[mock-upstream +  61.6s] claude-429 attempt 3 -> 200 stream
[mock-upstream +  61.8s] claude-429 client response status=200
[mock-upstream +  65.5s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +  65.5s] claude-529 attempt 3 -> 529
[mock-upstream +  65.5s] claude-529 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +  89.0s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +  89.0s] claude-529 attempt 4 -> 200 stream
[mock-upstream +  89.0s] claude-529 client response status=200
[mock-upstream +   0.0s] codex-capacity client request received (failures=2, kind=529)
[mock-upstream +   0.0s] codex-capacity upstream POST model=gpt-5.6-sol
[mock-upstream +   0.0s] codex-capacity attempt 1 -> capacity SSE event
[mock-upstream +   0.2s] codex-capacity cooldown verify-codex-1 60000 ms (model_capacity)
[mock-upstream +  60.5s] codex-capacity upstream POST model=gpt-5.6-sol
[mock-upstream +  60.5s] codex-capacity attempt 2 -> capacity SSE event
[mock-upstream +  60.5s] codex-capacity cooldown verify-codex-1 60000 ms (model_capacity)
[mock-upstream + 121.1s] codex-capacity upstream POST model=gpt-5.6-sol
[mock-upstream + 121.1s] codex-capacity attempt 3 -> 200 stream
[mock-upstream + 121.1s] codex-capacity client response status=200

$ # Same build, CODEROUTER_CAPACITY_HOLD_MS=0 (old fail-fast), warm server, 1 x 529
run 2 HTTP 529 in 2.254531s
[mock-upstream +   0.0s] nohold-2 client request received (failures=1, kind=529)
[mock-upstream +   0.0s] nohold-2 upstream POST model=claude-sonnet-4-5
[mock-upstream +   0.0s] nohold-2 attempt 1 -> 529
[mock-upstream +   0.5s] nohold-2 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +   0.7s] nohold-2 client response status=529

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: dce72fc7-5d70-46e2-ad45-bb4aa6bcf695

📥 Commits

Reviewing files that changed from the base of the PR and between 1755ea8 and 874d45e.

📒 Files selected for processing (17)

• docs/coderouter-operations.md
• web/.env.example
• web/db/clickhouse/006_route_events_capacity_hold.sql
• web/services/coderouter/README.md
• web/services/coderouter/capacityHold.ts
• web/services/coderouter/claudeProxy.ts
• web/services/coderouter/claudeUpstream.ts
• web/services/coderouter/codexProxy.ts
• web/services/coderouter/faultClassification.ts
• web/services/coderouter/repository.ts
• web/services/coderouter/requestTelemetry.ts
• web/services/coderouter/usageLedger.ts
• web/tests/coderouter-capacity-hold.test.ts
• web/tests/coderouter-claude-proxy.test.ts
• web/tests/coderouter-claude-upstream.test.ts
• web/tests/coderouter-responses-proxy.test.ts
• web/tests/coderouter-usage-ledger.test.ts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[github-actions]

CI failure attribution

CI passes on 874d45eaec (run 36413333748 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's.

[github-actions]

Merge receipt for 874d45eaec: every check was green at merge (19 verified; 17 skipped by policy). Full suite runs on main after merge.