Skip to content

fix(domain): bound quota park windows and add healthy override (#14359) - #14555

Merged
diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.51from
texastoland:fix/14359-quota-park-cap
Sep 24, 2026
Merged

diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.51from
texastoland:fix/14359-quota-park-cap

Conversation

@texastoland

Copy link
Copy Markdown
Contributor

Fixes #14359.

Problem

qwen-cloud-token-plan stopped receiving traffic with
All qwen-cloud-token-plan accounts have exhausted their quota while the same upstream key
kept serving: the plan was topped up with a credit pack, which the console window percentage
does not reflect. Root cause chain (all read at release/v3.8.51):

  1. src/domain/quotaCache.ts — an exhausted entry with a nextResetAt is never TTL-expired;
    the park holds until that timestamp, however far out.
  2. open-sse/services/qwenTokenPlanQuotaFetcher.ts — the 5-hour window is "Temporarily
    Removed" from the console, so the fetcher reports the weekly window and its reset
    (~4 days) as the worst case -> setQuotaCache parks the connection on the weekly reset.
  3. src/sse/services/auth.ts — with zero quota candidates, the router refuses pre-dispatch
    with the synthetic message; no call_logs row is written (log-visibility half: fix(sse): requests skipped by the quota-parking path leave no call_logs row #14360).

Measured on the reporter's install: last real dispatch 2026-09-21T06:26Z, then ~12.5 h of
zero real dispatches while hourly connection-test probes returned 200, breaker CLOSED,
failureCount: 0, dashboard "Weekly window 0% left, 40,000 / 40,000".

Fix

"Upstream facts outrank console scrapes — but only within a freshness window."

  • Park cap (EXHAUSTED_MAX_PARK_MS, 30 min): every park writer (setQuotaCache,
    snapshot hydration, Codex hydration) caps the deadline at observation + park window, so
    nextResetAt becomes "when we stop trusting the park", not "when quota resets". Display
    paths read per-window resetAt, so dashboards stay truthful.
  • Streak preservation: while the same exhausted streak continues (prior entry exhausted
    with a future deadline), setQuotaCache keeps the existing capped deadline instead of
    re-anchoring at Date.now(). Without this the quota monitor's periodic rewrites re-anchor
    the park every tick and it never expires (this is exactly why a cap-only first attempt
    failed its live test).
  • Healthy override (healthyUntil map on the #8065 shared globalThis state): a
    successful upstream dispatch (chat.ts success hook, beside clearModelLock) arms
    now + 30 min, sliding; both predicates stand down while armed; a genuine 429
    (markAccountExhaustedFrom429) clears it and re-parks on the existing 5-minute TTL.
    The map lives on the shared state, not module scope, because standalone builds duplicate
    this module across webpack chunks (fix(backend): quota cache keeps renewed Codex accounts exhausted after restart #8065) — per-entry fields silently no-op on the
    apikey + hydration path, which is precisely the qwen path.
  • Preflight parity: evaluateQuotaCutoff reads fetcher data independently of the cache,
    so it honoured neither the cap nor the override; it now accepts
    QuotaCutoffScope.connectionId and has the same healthy escape, beside the existing
    isClaudeExtraUsageAllowed escape.
  • Refusal diagnostics: the all-accounts-skipped message appends (cached quota state, no upstream attempt; earliest reset ...) after the existing text — the literal
    have exhausted their quota substring is preserved because classify429 matches on it
    (fix(backend): auto/coding zero-config routes do not fall back when LKGP-sticky provider is quota-exhausted (429) #9269); only the reset hint travels in Retry-After.

Behaviour change: Retry-After

Far-reset providers now advertise at most the park window; clients retry sooner. Extra 429
traffic is bounded: genuinely exhausted accounts are re-probed at most once per park window,
and the first real 429 re-parks them on the existing 5-minute TTL.

Scope

Standard providers only. Codex/antigravity per-window branches are intentionally untouched
beyond the shared writer caps (their parks are released via the capped nextResetAt through
advancedWindowResetAt, but getQuotaWindowStatus().reachedThreshold still reads raw window
data once dispatched).

Prior art

#8065 (chunk-duplicated quota cache state), #12860 (Codex preflight cooldown never lifted),
#12451 / #12452 (healthy accounts persisted as exhausted), #14072 (Antigravity stuck 11+ h).

Tests

New (fail on ea3c12264b = current release/v3.8.51, pass here):

  • tests/unit/quota-park-cap-14359.test.ts — writer/hydration caps, streak preservation,
    stale-snapshot hydration unblocks, fresh snapshot still parks, expired park unblocks both
    predicates.
  • tests/unit/quota-healthy-override-14359.test.ts — override arms/stands down both
    predicates, 429 clears it, expiry after the park window.
  • tests/unit/quota-preflight-healthy-override-14359.test.ts — evaluateQuotaCutoff
    honours the override; still cuts off without it.
  • qwen-token-plan-quota-fetcher — weekly-only, fully-used case pins the parked-state
    precondition (limitReached, weekly resetAt).

Existing suites re-run green: quota-cache-hydrate-5015, repro-8065-quota-cache-cross-instance,
13601-quota-far-reset-skip, quota-preflight, account-fallback-anthropic-quota,
classify429, tests/integration/combo-matrix/quota-aware. npm run typecheck:core clean.

Live validation (reporter's install, 3.8.51-dev dist)

The same design, hot-patched into the running gateway's compiled chunks, was validated
end-to-end: a routed combo test dispatched the previously-parked qwen-cloud-token-plan
member (HTTP 200 "pong", 1.5 s) while the other members reached their real upstreams
(command-code 400 insufficient-credits, opencode-go 429 weekly) — no synthetic pre-dispatch
refusals anywhere.

Out of scope

#14360 (the skip writes no call_logs row) — separate log-visibility issue.

…souzapw#14359)

A connection whose cached quota entry carried a far-future nextResetAt
(e.g. qwen-cloud-token-plan, where the console currently reports only the
weekly window, reset ~4 days out) was skipped pre-dispatch until that date
even though the upstream was serving fine - `All <provider> accounts have
exhausted their quota` with zero call_logs rows (diegosouzapw#14359).

Changes:

- quotaCache: new EXHAUSTED_MAX_PARK_MS (30 min). Every park writer
  (setQuotaCache, snapshot hydration, Codex hydration) caps its deadline at
  observation + park window; setQuotaCache additionally preserves the prior
  deadline while the same exhausted streak continues, so the quota monitor's
  periodic rewrites cannot re-anchor (and thereby extend) the park.
- quotaCache: healthyUntil override on the shared (diegosouzapw#8065) state, armed per
  successful dispatch (chat.ts success hook, beside clearModelLock) and
  cleared by markAccountExhaustedFrom429; both predicates stand down while
  armed.
- quotaPreflight: evaluateQuotaCutoff honours the same override via
  QuotaCutoffScope.connectionId, next to the isClaudeExtraUsageAllowed
  escape.
- auth: the all-accounts-skipped refusal now appends cached-state
  diagnostics (earliest reset) after the existing message; the literal
  `have exhausted their quota` substring is preserved for classify429.

Retry-After for far-reset providers now advertises at most the park window;
genuinely exhausted accounts re-probe at most once per park window and a
real 429 re-parks them on the existing 5-minute TTL.

Scoped to standard providers; codex/antigravity per-window branches are
intentionally untouched beyond the shared writer caps.

Tests: new quota-park-cap-14359 / quota-healthy-override-14359 /
quota-preflight-healthy-override-14359 (RED on ea3c122, GREEN here),
weekly-100% precondition added to the qwen fetcher suite; existing
5015/8065/13601/quota-preflight/account-fallback/classify429 suites and
combo-matrix quota-aware pass; typecheck:core clean.
@texastoland
texastoland force-pushed the fix/14359-quota-park-cap branch from 3d054e5 to e5a6ed6 Compare September 22, 2026 21:16
@diegosouzapw

Copy link
Copy Markdown
Owner

Nicely instrumented root-cause writeup with real production timestamps, and the fix design
("upstream facts outrank console scrapes, but only within a freshness window") is sound —
verified the cap is applied at all three park-writing call sites and the healthy override is
correctly armed on real success / disarmed on a genuine 429 in both directions. The streak-
preservation detail (and the note that a cap-only first attempt failed live) is a good sign
this was validated against the real upstream, not just theorized. Looks merge-ready.

@diegosouzapw
diegosouzapw merged commit e693a7e into diegosouzapw:release/v3.8.51 Sep 24, 2026
3 checks passed
SCys pushed a commit to SCys/OmniRoute that referenced this pull request Sep 25, 2026
Root cause: diegosouzapw#14555 added `import { isQuotaHealthy } from "@/domain/quotaCache"`
to open-sse/services/quotaPreflight.ts. quotaCache already imports
open-sse/services/usage.ts -> usage/openrouter.ts -> openrouterQuotaFetcher.ts
-> quotaPreflight.ts, so the new edge closed an ESM cycle. In the esbuild
MCP bundle every module is an async __esm initializer; quotaPreflight's init
awaited quotaCache's still-pending init promise, the chain never settled and
`import(server.js)` exited 13 with "Detected unsettled top-level await"
(tests/unit/build/mcp-bundle-startup.test.ts).

Fix: move the shared globalThis state (diegosouzapw#8065), the entry types,
EXHAUSTED_MAX_PARK_MS and the healthy-override helpers into a runtime
import-free leaf, src/domain/quotaCacheState.ts. quotaCache imports and
re-exports them (chat.ts and tests unchanged); quotaPreflight imports the
leaf directly. Same `__omnirouteQuotaCacheState.healthyUntil` map, so the
diegosouzapw#14555 park cap and healthy override behave exactly as before.

Refs diegosouzapw#14547
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(domain): qwen-cloud-token-plan parked as "all accounts exhausted" after a credit-pack top-up

2 participants