Skip to content

fix(resilience): escalate per-kind breaker cooldown on repeated open cycles (#14960) - #15001

Merged
diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.51from
afonsoft:fix/14960-breaker-override-escalation
Sep 29, 2026
Merged

diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.51from
afonsoft:fix/14960-breaker-override-escalation

Conversation

@afonsoft

@afonsoft afonsoft commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #14960.

  • With useUpstream429BreakerHints on, quota_exhausted maps to a fixed cooldownByKind override (3.6M ms) — but CircuitBreaker._effectiveCooldown() returned that override before the openCycleCount escalation ran, so failed HALF_OPEN probes kept bumping the cycle count while the next cooldown stayed 1h. A provider whose quota resets once a day got probed every hour all day.
  • The per-kind override now receives the same 2^(openCycleCount - backoffEscalationCount) escalation the base resetTimeout gets, capped at override * maxBackoffMultiplier (so quota_exhausted backs off 1h → 2h → 4h … up to 16h). Kindless failures are unchanged — they still escalate the base resetTimeout only.
  • providerCircuitOpenResponse accepts an optional failureKind and surfaces it as error.failure_kind on the 503 body; the three chatHelpers call sites pass breaker.getStatus().lastFailureKind. Operators can now tell a quota_exhausted breaker (long upstream reset) from a generic rate_limit one without reading server logs.

Test plan

  • tests/unit/circuit-breaker-kind-cooldown-escalation-14960.test.ts — 3/3 (override escalation curve + cap, kindless path unchanged, failure_kind on/off)
  • tests/unit/circuit-breaker-failure-kind.test.ts — 15/15 (existing [BUG] Circuit breaker: differentiate 429 rate-limit vs quota-exhausted #2100 per-kind cooldown coverage still green)
  • npm run typecheck:core — clean
  • eslint on changed files — no new errors (5 pre-existing on base)

⚠️ base-red inherited: #14963 (release/v3.8.51 tip has unrelated HARD gate failures; no overlap with this change)

Maintainer rework (merge-batch 2026-09-28 (release drain))

  • Merged release/v3.8.51 into the branch (real merge, no rewrite).
  • Prettier on circuitBreaker.ts / chatHelpers.ts.
  • The three circuit-open call sites in chatHelpers.ts now go through one breakerOpenResponse(provider, breaker, retryAfterSec) helper so each stays on a single line.
  • Recorded the remaining own growth (error.ts 1219->1221, chatHelpers.ts 1279->1285) as a dated frozen entry in config/quality/file-size-baseline.json.

…cycles (cheaper-inference#14960)

With useUpstream429BreakerHints on, cooldownByKind overrode
_effectiveCooldown() before openCycleCount escalation applied — a
quota_exhausted provider re-probed at the same fixed 1h interval
forever instead of backing off. The override now gets the same
2^(cycle - backoffEscalationCount) doubling the base resetTimeout
gets, capped at override * maxBackoffMultiplier (1h -> up to 16h).
providerCircuitOpenResponse also surfaces the classified
failure_kind on the 503 body so operators can tell a quota_exhausted
breaker from a generic rate_limit one without reading server logs.
@afonsoft

Copy link
Copy Markdown
Contributor Author

Detalhamento da correção (#14960)

Root cause

Com useUpstream429BreakerHints ligado, quota_exhausted mapeia para um cooldown fixo de 3_600_000 ms via cooldownByKind (chatHelpers.ts:362, accountFallback.ts:1163). O problema: CircuitBreaker._effectiveCooldown() (src/shared/utils/circuitBreaker.ts) retornava o override antes de aplicar o escalonamento — cada probe HALF_OPEN que falha incrementa openCycleCount, mas o próximo cooldown continuava 1h. Um provider com quota diária era sondado de hora em hora o dia inteiro.

Mudanças

  1. circuitBreaker.ts::_effectiveCooldown() — o override per-kind agora recebe o mesmo escalonamento do resetTimeout base: 2^(openCycleCount - backoffEscalationCount), com cap em override * maxBackoffMultiplier (quota_exhausted: 1h → 2h → 4h → … → 16h máx). Escolhi a primeira opção sugerida na issue (escalonar o override, não seguir o hint upstream verbatim) porque um probe que falha prova que o hint estava errado — backoff exponencial é a resposta correta. Falhas sem kind continuam no path antigo (_effectiveResetTimeout).
  2. open-sse/utils/error.ts::providerCircuitOpenResponse — aceita failureKind opcional e emite error.failure_kind no body do 503 (ponto levantado na issue).
  3. chatHelpers.ts — os 3 call sites passam breaker.getStatus().lastFailureKind.

Testes

tests/unit/circuit-breaker-kind-cooldown-escalation-14960.test.ts (3 casos): curva de escalonamento do override (flat até backoffEscalationCount, dobra depois, cap), path kindless inalterado, failure_kind presente/ausente. Os 15 testes de circuit-breaker-failure-kind.test.ts (#2100) seguem verdes.

cc @diegosouzapw — pode revisar?

…heaper-inference#15001

Formats circuitBreaker.ts/chatHelpers.ts with Prettier, folds the three
circuit-open call sites into one breakerOpenResponse helper and records the
remaining own growth (error.ts +2, chatHelpers.ts +6) as a dated frozen
file-size entry.
@diegosouzapw
diegosouzapw merged commit 60663bb into cheaper-inference:release/v3.8.51 Sep 29, 2026
6 of 7 checks passed
diegosouzapw added a commit that referenced this pull request Sep 29, 2026
…#14190, proxyLogs import side effect (#15116)

Release-captain base-red fix (v3.8.51 release PR #11442, unit shards 1-2 + integration): production defect — OmniRoute's own local-cooldown 429 was classified as CLIProxyAPI quota exhaustion since #14190 and made combos skip same-provider targets (now tagged X-OmniRoute-Local-Cooldown); the proxyLogHost extraction reconciled with #15109; contract propagations for #14627/#14873/#13888/#14876/#15001 and two harness isolation fixes. Focused suites green on the reconciled tip, typecheck clean.
@afonsoft
afonsoft deleted the fix/14960-breaker-override-escalation branch October 20, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(resilience): circuit breaker cooldownByKind override bypasses openCycleCount escalation (quota_exhausted retries every 1h forever)

2 participants