Skip to content

fix(resilience): derive the chat_admission_busy Retry-After from observed lease occupancy - #12395

Merged
diegosouzapw merged 2 commits into
cheaper-inference:release/v3.8.51from
pacocartones:fix/12135-heavy-admission-retry-after
Sep 2, 2026
Merged

diegosouzapw merged 2 commits into
cheaper-inference:release/v3.8.51from
pacocartones:fix/12135-heavy-admission-retry-after

Conversation

@pacocartones

Copy link
Copy Markdown
Contributor

Summary

  • The retryable chat_admission_busy 503 advertised a fixed Retry-After of 1 s (structural path) or 2 s (byte-stage path) while the heavyweight lease it waits on is held for the entire SSE lifetime. Clients that honor the header (Codex CLI, agent fan-out) re-sent the same ~1 MiB /v1/responses body every second into a gate that could not have cleared, producing the queue_timeout retry storm even after OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT was raised.
  • ChatAdmissionController now tracks the acquisition time of every live heavy lease and derives the hint from observed occupancy: the larger of the queue window the waiter already exhausted and the age of the youngest live lease (time since capacity last turned over), rounded up and capped at 60 s. Both 503 builders floor it at the historical 1 s / 2 s, so an idle gate answers exactly as before. The youngest lease is used rather than the oldest to avoid a pessimistic hint when several slots are in flight; the 60 s cap is a constant to keep the surface small and can be made tunable if preferred.
  • Scope: only the Retry-After defect the maintainer marked as standing. No change to the OMNIROUTE_CHAT_ADMISSION_QUEUE_MS default posture (deliberately left for the maintainer's decision), thresholds, status, error code, reason or routing. Troubleshooting guide entry updated.

Related Issues

Validation

  • Change type: Backend (resilience / chat admission middleware)
  • Focused tests and category gates from the golden path: new test file 7/7; 15 neighbouring admission test files 132/132; node scripts/check/check-complexity-ratchets.mjs --base-ref origin/release/v3.8.51 (no regression); npm run check:changelog-integrity OK; npm run typecheck:core 0
  • npm run lint — repository-wide eslint exit 0 (run with --pass-on-unpruned-suppressions; the literal command reports only pre-existing unused global suppressions on this base)
  • Reconciled with the current active release base release/v3.8.51; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR
  • SonarQube is temporarily opt-in while the private project has no quota; it is not a PR gate.

Mutation: with the source change reverted, 5 of the 7 new tests fail (actual Retry-After 1/2 against expected 5, 45, 7, 60, 30).

Tests Added Or Updated

  • tests/unit/heavy-admission-retry-after-12135.test.ts (new, 7 tests, mock timers): (a) queue_timeout Retry-After is at least the exhausted queue window; (b) structural and byte-stage 503s reflect observed lease age, using the youngest live lease; (c) the hint is capped at 60 s; (d) an idle gate keeps the historical 1 s / 2 s floors; (e) with the count cap raised, a third heavy /v1/responses request queues and is admitted when a lease frees inside queueMs. On the base: 2 pass / 5 fail; with the fix: 7 pass.

Coverage Notes

  • src/shared/middleware/chatBodyAdmission.ts: retryAfterSeconds, lease-time tracking in tryAcquireHeavy, and all four 503 emission sites are exercised by the new test; the existing admission suites (chat-body-admission*.test.ts, probe-10268-structural-503.test.ts, chat-admission-visibility-11244.test.ts, with-chat-admission-10786.test.ts, per-connection-admission-9654.test.ts, and others) still pass unchanged, covering the untouched paths.
  • src/shared/middleware/chatAdmissionResponses.ts: the new retryAfterHeader floor is covered by test (d) and by chat-admission-rejection-reason.test.ts (a call without a hint keeps Retry-After: 1).

…pancy

Root cause: the retryable `chat_admission_busy` 503 advertised a fixed
`Retry-After` — "1" on the structural path
(src/shared/middleware/chatAdmissionResponses.ts, structuralRejectionResponse)
and "2" on the byte-stage path (chatAdmissionRejectionResponse) — regardless of
how long heavyweight capacity had actually been busy or how long the waiter had
already spent in the bounded queue. A heavyweight lease is held for the entire
SSE lifetime (tens of seconds to minutes), so clients that honor the header
(Codex CLI, agent fan-out) re-sent the same ~1 MiB `/v1/responses` body every
second into a gate that could not possibly have cleared, producing the
`queue_timeout` retry storm reported in cheaper-inference#12135 even after
`OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT` was raised.

Fix: `ChatAdmissionController` now records the acquisition time of every live
heavy lease and exposes `retryAfterSeconds(queueMs)` — the larger of the queue
window the waiter already exhausted and the age of the youngest live lease
(time since capacity last turned over), rounded up to whole seconds and capped
at `CHAT_ADMISSION_RETRY_AFTER_MAX_SECONDS` (60). Both 503 builders take the
hint and floor it at their historical value (1 s / 2 s), so an idle gate
answers exactly as before and no existing client behavior changes on a quiet
host. The troubleshooting guide entry that documented the fixed values is
updated.

Boundary: the maintainer split cheaper-inference#12135 into the Retry-After defect ("stands
regardless") and the `OMNIROUTE_CHAT_ADMISSION_QUEUE_MS` default posture, which
"deserves an explicit decision + documentation". This change touches only the
former: no queue default, threshold, status, error code, reason or routing
behavior changes.

Verification:
- tests/unit/heavy-admission-retry-after-12135.test.ts: 7 tests; on the base
  2 pass / 5 fail, with the fix 7 pass; reverting the source change restores
  5 fail (mutation check).
- 15 neighbouring admission test files: 132 pass / 0 fail.
- eslint on the touched files, prettier, `npm run typecheck:core`,
  `check-complexity-ratchets --base-ref origin/release/v3.8.51`,
  `check:changelog-integrity`, `git diff --check`: all clean.

Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
diegosouzapw pushed a commit that referenced this pull request Sep 2, 2026
Four documentation claims contradicted the code: .env.example called OMNIROUTE_USE_TURBOPACK dev-only and said the production build still uses webpack (it reads the same flag and defaults to Turbopack); the README's Bun section said `bun run build` auto-detects Bun and switches to Webpack (only `bun run dev` does — the production bundler is decided by the flag alone); TROUBLESHOOTING.md gave OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT a default of 1 when unset means no request-count cap; and it quoted the pre-#12223 wording of the structural 503 chat_admission_busy message. The Retry-After bullet in the same section is deliberately untouched because #12395 rewrites it.

Validated in a combined worktree with all 25 PRs of this batch boarded together: typecheck:core clean, 443/443 node-runner tests plus 14/14 vitest across every test file the batch touches, and check-changelog-integrity, check:cycles (418 files), check:provider-consistency (272 REGISTRY entries, 355 canonical providers), check:docs-counts, check:docs-sync (42 locales) and check-file-size all green.

Thanks @pacocartones.
@diegosouzapw
diegosouzapw merged commit 62e2481 into cheaper-inference:release/v3.8.51 Sep 2, 2026
16 checks passed
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ence#12404)

Four documentation claims contradicted the code: .env.example called OMNIROUTE_USE_TURBOPACK dev-only and said the production build still uses webpack (it reads the same flag and defaults to Turbopack); the README's Bun section said `bun run build` auto-detects Bun and switches to Webpack (only `bun run dev` does — the production bundler is decided by the flag alone); TROUBLESHOOTING.md gave OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT a default of 1 when unset means no request-count cap; and it quoted the pre-cheaper-inference#12223 wording of the structural 503 chat_admission_busy message. The Retry-After bullet in the same section is deliberately untouched because cheaper-inference#12395 rewrites it.

Validated in a combined worktree with all 25 PRs of this batch boarded together: typecheck:core clean, 443/443 node-runner tests plus 14/14 vitest across every test file the batch touches, and check-changelog-integrity, check:cycles (418 files), check:provider-consistency (272 REGISTRY entries, 355 canonical providers), check:docs-counts, check:docs-sync (42 locales) and check-file-size all green.

Thanks @pacocartones.
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…rved lease occupancy (cheaper-inference#12395)

The retryable chat_admission_busy 503 advertised a fixed Retry-After of 1s or 2s while the heavyweight lease it waits on is held for the entire SSE lifetime. Clients that honour the header — Codex CLI, agent fan-out — re-sent the same ~1 MiB /v1/responses body every second into a gate that could not have cleared, producing the queue_timeout retry storm that persisted even after OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT was raised.

ChatAdmissionController now tracks each live heavy lease's acquisition time and derives the hint from observed occupancy: the larger of the queue window the waiter already exhausted and the age of the youngest live lease, rounded up and capped at 60s. Both builders floor it at the historical 1s / 2s, so an idle gate answers exactly as before. Using the youngest rather than the oldest lease avoids a pessimistic hint when several slots are in flight.

Validated in a combined worktree with all 25 PRs of this batch boarded together: typecheck:core clean, 443/443 node-runner tests plus 14/14 vitest across every test file the batch touches, and check-changelog-integrity, check:cycles (418 files), check:provider-consistency (272 REGISTRY entries, 355 canonical providers), check:docs-counts, check:docs-sync (42 locales) and check-file-size all green.

Thanks @pacocartones.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants