fix(domain): bound quota park windows and add healthy override (#14359) - #14555
Merged
diegosouzapw merged 1 commit intoSep 24, 2026
Merged
diegosouzapw merged 1 commit into
diegosouzapw merged 1 commit into
Conversation
texastoland
force-pushed
the
fix/14359-quota-park-cap
branch
from
September 22, 2026 19:59
08d11a5 to
3d054e5
Compare
…souzapw#14359) A connection whose cached quota entry carried a far-future nextResetAt (e.g. qwen-cloud-token-plan, where the console currently reports only the weekly window, reset ~4 days out) was skipped pre-dispatch until that date even though the upstream was serving fine - `All <provider> accounts have exhausted their quota` with zero call_logs rows (diegosouzapw#14359). Changes: - quotaCache: new EXHAUSTED_MAX_PARK_MS (30 min). Every park writer (setQuotaCache, snapshot hydration, Codex hydration) caps its deadline at observation + park window; setQuotaCache additionally preserves the prior deadline while the same exhausted streak continues, so the quota monitor's periodic rewrites cannot re-anchor (and thereby extend) the park. - quotaCache: healthyUntil override on the shared (diegosouzapw#8065) state, armed per successful dispatch (chat.ts success hook, beside clearModelLock) and cleared by markAccountExhaustedFrom429; both predicates stand down while armed. - quotaPreflight: evaluateQuotaCutoff honours the same override via QuotaCutoffScope.connectionId, next to the isClaudeExtraUsageAllowed escape. - auth: the all-accounts-skipped refusal now appends cached-state diagnostics (earliest reset) after the existing message; the literal `have exhausted their quota` substring is preserved for classify429. Retry-After for far-reset providers now advertises at most the park window; genuinely exhausted accounts re-probe at most once per park window and a real 429 re-parks them on the existing 5-minute TTL. Scoped to standard providers; codex/antigravity per-window branches are intentionally untouched beyond the shared writer caps. Tests: new quota-park-cap-14359 / quota-healthy-override-14359 / quota-preflight-healthy-override-14359 (RED on ea3c122, GREEN here), weekly-100% precondition added to the qwen fetcher suite; existing 5015/8065/13601/quota-preflight/account-fallback/classify429 suites and combo-matrix quota-aware pass; typecheck:core clean.
texastoland
force-pushed
the
fix/14359-quota-park-cap
branch
from
September 22, 2026 21:16
3d054e5 to
e5a6ed6
Compare
Owner
|
Nicely instrumented root-cause writeup with real production timestamps, and the fix design |
SCys
pushed a commit
to SCys/OmniRoute
that referenced
this pull request
Sep 25, 2026
Root cause: diegosouzapw#14555 added `import { isQuotaHealthy } from "@/domain/quotaCache"` to open-sse/services/quotaPreflight.ts. quotaCache already imports open-sse/services/usage.ts -> usage/openrouter.ts -> openrouterQuotaFetcher.ts -> quotaPreflight.ts, so the new edge closed an ESM cycle. In the esbuild MCP bundle every module is an async __esm initializer; quotaPreflight's init awaited quotaCache's still-pending init promise, the chain never settled and `import(server.js)` exited 13 with "Detected unsettled top-level await" (tests/unit/build/mcp-bundle-startup.test.ts). Fix: move the shared globalThis state (diegosouzapw#8065), the entry types, EXHAUSTED_MAX_PARK_MS and the healthy-override helpers into a runtime import-free leaf, src/domain/quotaCacheState.ts. quotaCache imports and re-exports them (chat.ts and tests unchanged); quotaPreflight imports the leaf directly. Same `__omnirouteQuotaCacheState.healthyUntil` map, so the diegosouzapw#14555 park cap and healthy override behave exactly as before. Refs diegosouzapw#14547
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14359.
Problem
qwen-cloud-token-planstopped receiving traffic withAll qwen-cloud-token-plan accounts have exhausted their quotawhile the same upstream keykept serving: the plan was topped up with a credit pack, which the console window percentage
does not reflect. Root cause chain (all read at
release/v3.8.51):src/domain/quotaCache.ts— an exhausted entry with anextResetAtis never TTL-expired;the park holds until that timestamp, however far out.
open-sse/services/qwenTokenPlanQuotaFetcher.ts— the 5-hour window is "TemporarilyRemoved" from the console, so the fetcher reports the weekly window and its reset
(~4 days) as the worst case ->
setQuotaCacheparks the connection on the weekly reset.src/sse/services/auth.ts— with zero quota candidates, the router refuses pre-dispatchwith the synthetic message; no
call_logsrow is written (log-visibility half: fix(sse): requests skipped by the quota-parking path leave no call_logs row #14360).Measured on the reporter's install: last real dispatch 2026-09-21T06:26Z, then ~12.5 h of
zero real dispatches while hourly
connection-testprobes returned 200, breaker CLOSED,failureCount: 0, dashboard "Weekly window 0% left, 40,000 / 40,000".Fix
"Upstream facts outrank console scrapes — but only within a freshness window."
EXHAUSTED_MAX_PARK_MS, 30 min): every park writer (setQuotaCache,snapshot hydration, Codex hydration) caps the deadline at observation + park window, so
nextResetAtbecomes "when we stop trusting the park", not "when quota resets". Displaypaths read per-window
resetAt, so dashboards stay truthful.with a future deadline),
setQuotaCachekeeps the existing capped deadline instead ofre-anchoring at
Date.now(). Without this the quota monitor's periodic rewrites re-anchorthe park every tick and it never expires (this is exactly why a cap-only first attempt
failed its live test).
healthyUntilmap on the#8065shared globalThis state): asuccessful upstream dispatch (
chat.tssuccess hook, besideclearModelLock) armsnow + 30 min, sliding; both predicates stand down while armed; a genuine 429(
markAccountExhaustedFrom429) clears it and re-parks on the existing 5-minute TTL.The map lives on the shared state, not module scope, because standalone builds duplicate
this module across webpack chunks (fix(backend): quota cache keeps renewed Codex accounts exhausted after restart #8065) — per-entry fields silently no-op on the
apikey + hydration path, which is precisely the qwen path.
evaluateQuotaCutoffreads fetcher data independently of the cache,so it honoured neither the cap nor the override; it now accepts
QuotaCutoffScope.connectionIdand has the same healthy escape, beside the existingisClaudeExtraUsageAllowedescape.(cached quota state, no upstream attempt; earliest reset ...)after the existing text — the literalhave exhausted their quotasubstring is preserved becauseclassify429matches on it(fix(backend): auto/coding zero-config routes do not fall back when LKGP-sticky provider is quota-exhausted (429) #9269); only the reset hint travels in
Retry-After.Behaviour change:
Retry-AfterFar-reset providers now advertise at most the park window; clients retry sooner. Extra 429
traffic is bounded: genuinely exhausted accounts are re-probed at most once per park window,
and the first real 429 re-parks them on the existing 5-minute TTL.
Scope
Standard providers only. Codex/antigravity per-window branches are intentionally untouched
beyond the shared writer caps (their parks are released via the capped
nextResetAtthroughadvancedWindowResetAt, butgetQuotaWindowStatus().reachedThresholdstill reads raw windowdata once dispatched).
Prior art
#8065 (chunk-duplicated quota cache state), #12860 (Codex preflight cooldown never lifted),
#12451 / #12452 (healthy accounts persisted as exhausted), #14072 (Antigravity stuck 11+ h).
Tests
New (fail on
ea3c12264b= currentrelease/v3.8.51, pass here):tests/unit/quota-park-cap-14359.test.ts— writer/hydration caps, streak preservation,stale-snapshot hydration unblocks, fresh snapshot still parks, expired park unblocks both
predicates.
tests/unit/quota-healthy-override-14359.test.ts— override arms/stands down bothpredicates, 429 clears it, expiry after the park window.
tests/unit/quota-preflight-healthy-override-14359.test.ts—evaluateQuotaCutoffhonours the override; still cuts off without it.
qwen-token-plan-quota-fetcher— weekly-only, fully-used case pins the parked-stateprecondition (
limitReached, weeklyresetAt).Existing suites re-run green:
quota-cache-hydrate-5015,repro-8065-quota-cache-cross-instance,13601-quota-far-reset-skip,quota-preflight,account-fallback-anthropic-quota,classify429,tests/integration/combo-matrix/quota-aware.npm run typecheck:coreclean.Live validation (reporter's install, 3.8.51-dev dist)
The same design, hot-patched into the running gateway's compiled chunks, was validated
end-to-end: a routed combo test dispatched the previously-parked
qwen-cloud-token-planmember (HTTP 200 "pong", 1.5 s) while the other members reached their real upstreams
(command-code 400 insufficient-credits, opencode-go 429 weekly) — no synthetic pre-dispatch
refusals anywhere.
Out of scope
#14360 (the skip writes no
call_logsrow) — separate log-visibility issue.