Repository navigation
fix(resilience): escalate per-kind breaker cooldown on repeated open cycles (#14960) - #15001
Conversation
…cycles (cheaper-inference#14960) With useUpstream429BreakerHints on, cooldownByKind overrode _effectiveCooldown() before openCycleCount escalation applied — a quota_exhausted provider re-probed at the same fixed 1h interval forever instead of backing off. The override now gets the same 2^(cycle - backoffEscalationCount) doubling the base resetTimeout gets, capped at override * maxBackoffMultiplier (1h -> up to 16h). providerCircuitOpenResponse also surfaces the classified failure_kind on the 503 body so operators can tell a quota_exhausted breaker from a generic rate_limit one without reading server logs.
Detalhamento da correção (#14960)Root causeCom Mudanças
Testes
cc @diegosouzapw — pode revisar? |
…heaper-inference#15001 Formats circuitBreaker.ts/chatHelpers.ts with Prettier, folds the three circuit-open call sites into one breakerOpenResponse helper and records the remaining own growth (error.ts +2, chatHelpers.ts +6) as a dated frozen file-size entry.
60663bb
into
cheaper-inference:release/v3.8.51
…#14190, proxyLogs import side effect (#15116) Release-captain base-red fix (v3.8.51 release PR #11442, unit shards 1-2 + integration): production defect — OmniRoute's own local-cooldown 429 was classified as CLIProxyAPI quota exhaustion since #14190 and made combos skip same-provider targets (now tagged X-OmniRoute-Local-Cooldown); the proxyLogHost extraction reconciled with #15109; contract propagations for #14627/#14873/#13888/#14876/#15001 and two harness isolation fixes. Focused suites green on the reconciled tip, typecheck clean.
Summary
Closes #14960.
useUpstream429BreakerHintson,quota_exhaustedmaps to a fixedcooldownByKindoverride (3.6M ms) — butCircuitBreaker._effectiveCooldown()returned that override before theopenCycleCountescalation ran, so failed HALF_OPEN probes kept bumping the cycle count while the next cooldown stayed 1h. A provider whose quota resets once a day got probed every hour all day.2^(openCycleCount - backoffEscalationCount)escalation the baseresetTimeoutgets, capped atoverride * maxBackoffMultiplier(soquota_exhaustedbacks off 1h → 2h → 4h … up to 16h). Kindless failures are unchanged — they still escalate the baseresetTimeoutonly.providerCircuitOpenResponseaccepts an optionalfailureKindand surfaces it aserror.failure_kindon the 503 body; the threechatHelperscall sites passbreaker.getStatus().lastFailureKind. Operators can now tell aquota_exhaustedbreaker (long upstream reset) from a genericrate_limitone without reading server logs.Test plan
tests/unit/circuit-breaker-kind-cooldown-escalation-14960.test.ts— 3/3 (override escalation curve + cap, kindless path unchanged,failure_kindon/off)tests/unit/circuit-breaker-failure-kind.test.ts— 15/15 (existing [BUG] Circuit breaker: differentiate 429 rate-limit vs quota-exhausted #2100 per-kind cooldown coverage still green)npm run typecheck:core— cleanMaintainer rework (merge-batch 2026-09-28 (release drain))
breakerOpenResponse(provider, breaker, retryAfterSec)helper so each stays on a single line.frozenentry in config/quality/file-size-baseline.json.