Repository navigation
fix(resilience): derive the chat_admission_busy Retry-After from observed lease occupancy - #12395
Merged
diegosouzapw merged 2 commits intoSep 2, 2026
Conversation
…pancy Root cause: the retryable `chat_admission_busy` 503 advertised a fixed `Retry-After` — "1" on the structural path (src/shared/middleware/chatAdmissionResponses.ts, structuralRejectionResponse) and "2" on the byte-stage path (chatAdmissionRejectionResponse) — regardless of how long heavyweight capacity had actually been busy or how long the waiter had already spent in the bounded queue. A heavyweight lease is held for the entire SSE lifetime (tens of seconds to minutes), so clients that honor the header (Codex CLI, agent fan-out) re-sent the same ~1 MiB `/v1/responses` body every second into a gate that could not possibly have cleared, producing the `queue_timeout` retry storm reported in cheaper-inference#12135 even after `OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT` was raised. Fix: `ChatAdmissionController` now records the acquisition time of every live heavy lease and exposes `retryAfterSeconds(queueMs)` — the larger of the queue window the waiter already exhausted and the age of the youngest live lease (time since capacity last turned over), rounded up to whole seconds and capped at `CHAT_ADMISSION_RETRY_AFTER_MAX_SECONDS` (60). Both 503 builders take the hint and floor it at their historical value (1 s / 2 s), so an idle gate answers exactly as before and no existing client behavior changes on a quiet host. The troubleshooting guide entry that documented the fixed values is updated. Boundary: the maintainer split cheaper-inference#12135 into the Retry-After defect ("stands regardless") and the `OMNIROUTE_CHAT_ADMISSION_QUEUE_MS` default posture, which "deserves an explicit decision + documentation". This change touches only the former: no queue default, threshold, status, error code, reason or routing behavior changes. Verification: - tests/unit/heavy-admission-retry-after-12135.test.ts: 7 tests; on the base 2 pass / 5 fail, with the fix 7 pass; reverting the source change restores 5 fail (mutation check). - 15 neighbouring admission test files: 132 pass / 0 fail. - eslint on the touched files, prettier, `npm run typecheck:core`, `check-complexity-ratchets --base-ref origin/release/v3.8.51`, `check:changelog-integrity`, `git diff --check`: all clean. Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
4 of 5 tasks
diegosouzapw
pushed a commit
that referenced
this pull request
Sep 2, 2026
Four documentation claims contradicted the code: .env.example called OMNIROUTE_USE_TURBOPACK dev-only and said the production build still uses webpack (it reads the same flag and defaults to Turbopack); the README's Bun section said `bun run build` auto-detects Bun and switches to Webpack (only `bun run dev` does — the production bundler is decided by the flag alone); TROUBLESHOOTING.md gave OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT a default of 1 when unset means no request-count cap; and it quoted the pre-#12223 wording of the structural 503 chat_admission_busy message. The Retry-After bullet in the same section is deliberately untouched because #12395 rewrites it. Validated in a combined worktree with all 25 PRs of this batch boarded together: typecheck:core clean, 443/443 node-runner tests plus 14/14 vitest across every test file the batch touches, and check-changelog-integrity, check:cycles (418 files), check:provider-consistency (272 REGISTRY entries, 355 canonical providers), check:docs-counts, check:docs-sync (42 locales) and check-file-size all green. Thanks @pacocartones.
diegosouzapw
merged commit Sep 2, 2026
62e2481
into
cheaper-inference:release/v3.8.51
16 checks passed
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…ence#12404) Four documentation claims contradicted the code: .env.example called OMNIROUTE_USE_TURBOPACK dev-only and said the production build still uses webpack (it reads the same flag and defaults to Turbopack); the README's Bun section said `bun run build` auto-detects Bun and switches to Webpack (only `bun run dev` does — the production bundler is decided by the flag alone); TROUBLESHOOTING.md gave OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT a default of 1 when unset means no request-count cap; and it quoted the pre-cheaper-inference#12223 wording of the structural 503 chat_admission_busy message. The Retry-After bullet in the same section is deliberately untouched because cheaper-inference#12395 rewrites it. Validated in a combined worktree with all 25 PRs of this batch boarded together: typecheck:core clean, 443/443 node-runner tests plus 14/14 vitest across every test file the batch touches, and check-changelog-integrity, check:cycles (418 files), check:provider-consistency (272 REGISTRY entries, 355 canonical providers), check:docs-counts, check:docs-sync (42 locales) and check-file-size all green. Thanks @pacocartones.
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…rved lease occupancy (cheaper-inference#12395) The retryable chat_admission_busy 503 advertised a fixed Retry-After of 1s or 2s while the heavyweight lease it waits on is held for the entire SSE lifetime. Clients that honour the header — Codex CLI, agent fan-out — re-sent the same ~1 MiB /v1/responses body every second into a gate that could not have cleared, producing the queue_timeout retry storm that persisted even after OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT was raised. ChatAdmissionController now tracks each live heavy lease's acquisition time and derives the hint from observed occupancy: the larger of the queue window the waiter already exhausted and the age of the youngest live lease, rounded up and capped at 60s. Both builders floor it at the historical 1s / 2s, so an idle gate answers exactly as before. Using the youngest rather than the oldest lease avoids a pessimistic hint when several slots are in flight. Validated in a combined worktree with all 25 PRs of this batch boarded together: typecheck:core clean, 443/443 node-runner tests plus 14/14 vitest across every test file the batch touches, and check-changelog-integrity, check:cycles (418 files), check:provider-consistency (272 REGISTRY entries, 355 canonical providers), check:docs-counts, check:docs-sync (42 locales) and check-file-size all green. Thanks @pacocartones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
chat_admission_busy503 advertised a fixedRetry-Afterof 1 s (structural path) or 2 s (byte-stage path) while the heavyweight lease it waits on is held for the entire SSE lifetime. Clients that honor the header (Codex CLI, agent fan-out) re-sent the same ~1 MiB/v1/responsesbody every second into a gate that could not have cleared, producing thequeue_timeoutretry storm even afterOMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHTwas raised.ChatAdmissionControllernow tracks the acquisition time of every live heavy lease and derives the hint from observed occupancy: the larger of the queue window the waiter already exhausted and the age of the youngest live lease (time since capacity last turned over), rounded up and capped at 60 s. Both 503 builders floor it at the historical 1 s / 2 s, so an idle gate answers exactly as before. The youngest lease is used rather than the oldest to avoid a pessimistic hint when several slots are in flight; the 60 s cap is a constant to keep the surface small and can be made tunable if preferred.OMNIROUTE_CHAT_ADMISSION_QUEUE_MSdefault posture (deliberately left for the maintainer's decision), thresholds, status, error code, reason or routing. Troubleshooting guide entry updated.Related Issues
Validation
node scripts/check/check-complexity-ratchets.mjs --base-ref origin/release/v3.8.51(no regression);npm run check:changelog-integrityOK;npm run typecheck:core0npm run lint— repository-wide eslint exit 0 (run with--pass-on-unpruned-suppressions; the literal command reports only pre-existing unused global suppressions on this base)release/v3.8.51; focused checks rerun afterwardMutation: with the source change reverted, 5 of the 7 new tests fail (actual
Retry-After1/2against expected5,45,7,60,30).Tests Added Or Updated
tests/unit/heavy-admission-retry-after-12135.test.ts(new, 7 tests, mock timers): (a)queue_timeoutRetry-After is at least the exhausted queue window; (b) structural and byte-stage 503s reflect observed lease age, using the youngest live lease; (c) the hint is capped at 60 s; (d) an idle gate keeps the historical 1 s / 2 s floors; (e) with the count cap raised, a third heavy/v1/responsesrequest queues and is admitted when a lease frees insidequeueMs. On the base: 2 pass / 5 fail; with the fix: 7 pass.Coverage Notes
src/shared/middleware/chatBodyAdmission.ts:retryAfterSeconds, lease-time tracking intryAcquireHeavy, and all four 503 emission sites are exercised by the new test; the existing admission suites (chat-body-admission*.test.ts,probe-10268-structural-503.test.ts,chat-admission-visibility-11244.test.ts,with-chat-admission-10786.test.ts,per-connection-admission-9654.test.ts, and others) still pass unchanged, covering the untouched paths.src/shared/middleware/chatAdmissionResponses.ts: the newretryAfterHeaderfloor is covered by test (d) and bychat-admission-rejection-reason.test.ts(a call without a hint keepsRetry-After: 1).