fix(resilience): decouple rate-limit execution expiration from queue-wait budget; preserve errors in oversized call-log artifacts by alvinveroy · Pull Request #12027 · diegosouzapw/OmniRoute

alvinveroy · 2026-08-29T11:59:26Z

Fixes #12025
Fixes #12026

What

1. Execution expiration decoupled from the queue-wait budget (#12025)

withRateLimit() passed requestQueue.maxWaitMs to Bottleneck as the job expiration. Bottleneck's expiration timer starts only after a job leaves QUEUED, so the queue-wait budget actually bounded execution — killing legitimate long-running LLM calls mid-flight with a false 504 RATE_LIMIT_EXECUTION_TIMEOUT (mechanism documented in #9533, item 1; still unfixed on main).

Production evidence (3-day window, one instance): 2,197 × 504 on opencode-go/glm-5.3-flash with ~45s durations and 0 tokens — requests killed exactly at maxWaitMs=45000 while the upstream was still generating. Non-incremental gateways (Console Go / Command Code tiers) buffer entire generations before the first upstream byte, and the local limiter deadline undercut provider-aware upstream fetch-start timeouts.

Fix: new requestQueue.executionMaxWaitMs (env RATE_LIMIT_EXECUTION_MAX_WAIT_MS, default 600000 = 10 min) feeds Bottleneck's expiration. maxWaitMs keeps its documented queue-wait semantics (factory default unchanged at 15s per #6593). The surfaced 504 message names the new knob and keeps every #4165 guarantee: disclaims an upstream timeout, preserves the Bottleneck error as cause, branded code + trusted provenance, classified request-scoped so combo falls back to the next target.

A true queue-wait bound (the Promise.race around limiter.schedule() sketched in #9533) is deliberately out of scope — this PR stops the budget from killing executions, which is the production-facing bug.

2. Error field preserved in oversized call-log artifacts (#12026)

The size-limit fallback in src/lib/usage/callLogArtifacts.ts replaced the error with [omitted: call log artifact size limit exceeded] — 91 of 847 opencode-go 504 rows on one instance were undiagnosable from the dashboard. The error (typically ~90 bytes, e.g. [504]: Fetch timeout after 110000ms on https://…) is now preserved truncated to 4KB in every fallback stage; oversized request/response bodies are still omitted exactly as before.

Tests

tests/unit/rate-limit-execution-timeout-message-4165.test.ts — execution expiration driven by the new knob; new regression: execution outliving the queue-wait budget completes (the exact production bug).
tests/unit/call-log-cap.test.ts — sentinel-fallback shape; new regression: error preserved verbatim in size-limit fallback artifacts.
tests/unit/ratelimit-admission-control-6593.test.ts — new default assertion for executionMaxWaitMs (600000), maxWaitMs default assertions unchanged.
tests/unit/i18n-vi-completeness.test.ts, tests/unit/i18n-pt-br.test.ts — new resilienceMaxExecutionWait key added to all 43 locales (parity clean).

All 38 targeted tests pass. Full npm run test:coverage on a clean Linux runner with this change: Statements 84.81% / Lines 84.81% / Functions 88.26% / Branches 79.47% (gate ≥ 60%).

Notes

changelog.d/fixes/ fragments included for both issues.
UI: Settings → Resilience gains an "Execution timeout (rate-limit backstop)" field next to "Maximum queue wait time" (all 43 locales).
.env.example documents RATE_LIMIT_EXECUTION_MAX_WAIT_MS.

…ead to the npm leg (diegosouzapw#11973) v3.8.50 shipped with zero desktop assets. The tag push did trigger electron-release.yml (run 33005490476) but GitHub refused the run at startup: Error calling workflow 'npm-publish.yml@5458026'. The nested job 'publish' is requesting 'actions: read', but is only allowed 'actions: none'. npm-publish.yml's `publish` job gained `actions: read` (it downloads the next-build artefact) and the caller job here never widened its grant — a reusable workflow may not request more than its caller allows, and the refusal is a startup failure of the WHOLE run, so the `release` job that attaches the installers, the source archives and the SBOM never ran either. Nothing about it is visible through the API (no jobs, no check-runs); only the run page shows the annotation. - publish-npm: `actions: read` added, with the rule written down (keep the block a superset of every job in npm-publish.yml). - workflow_dispatch: new boolean input `publish_npm` (default true) and the npm leg is gated on it, so re-attaching assets to a release whose package already shipped does not try to publish the same version twice. - web-build / build / release checkouts pin `ref: needs.validate.outputs.version`: a dispatch builds the tag it names, not the dispatching branch (a tag push resolves to the same commit, so nothing changes on the normal path). actionlint clean; electron-release-desktop-channel-8949, electron-release-efficiency, build-next-isolated-windows-home-2402, electron-release-latest-yml.repro and check-workflows suites pass. Next step: dispatch on main with version=v3.8.50 and publish_npm=false to attach the missing assets.

…ob and the main run (main twin of diegosouzapw#11972) (diegosouzapw#11978) Same change as diegosouzapw#11972 on release/v3.8.51: the Coverage job had timeout-minutes: 20, the c8 merge across 8 shards takes ~10 min and the informational Codecov upload hung for the rest of the budget on two consecutive main runs (33207760653, 33215115341), ending the job cancelled and turning the run's conclusion cancelled with every blocking job green. Codecov step: 5-minute ceiling + continue-on-error; job: 30 min.

…wait budget requestQueue.maxWaitMs was passed to Bottleneck as the job expiration, but Bottleneck's expiration timer starts only after a job leaves QUEUED — so the queue-wait budget actually bounded execution and killed legitimate long-running LLM calls mid-flight with a false 504 (RATE_LIMIT_EXECUTION_TIMEOUT) on non-incremental gateways that buffer whole generations before the first upstream byte (e.g. Console Go / Command Code tiers serving GLM models). Introduce requestQueue.executionMaxWaitMs (env RATE_LIMIT_EXECUTION_MAX_WAIT_MS, default 600000 = 10 min) as the dedicated execution backstop feeding Bottleneck's expiration. maxWaitMs keeps its documented queue-wait semantics (gates, admission, rolling RPM gate). The surfaced 504 message names the new knob and keeps the diegosouzapw#4165 guarantees: disclaims an upstream timeout, preserves the Bottleneck error as cause, branded code + trusted provenance, classified request-scoped so combo falls back. Also preserve the error field (truncated to 4KB) in every call-log artifact size-limit fallback stage: the minimal fallback previously replaced the error with '[omitted: call log artifact size limit exceeded]', leaving oversized- artifact rows with no diagnosable cause. The maxWaitMs factory default stays 15s per diegosouzapw#6593; queue-wait bounding via a Promise.race around limiter.schedule() (diegosouzapw#9533) remains future work. Fixes diegosouzapw#12025 Fixes diegosouzapw#12026

…the bodies (#12026) (#12095) Mantém o erro do call-log quando o limite de tamanho corta os bodies, com `preserveErrorForSizeLimit` (UTF-8-safe, preserva o valor original quando cabe, trata erro circular/não-serializável) — implementação mais robusta que a alternativa que já estava na tip (via #12027, que resolvi combinando: mantive a camada extra "errorOnly" do #12027 usando o helper mais seguro deste). Testes próprios + os de #12027 todos verdes (30/30) no worktree combinado. Obrigado!

… to .env.example only)

…e/v3.8.51 (round 5: provider count 352, TS2554/TS2677) (#12144) * fix(ci): clear the base-reds the 2026-08-30 afternoon merge batch left on release/v3.8.51 (round 5) - docs-counts / check-docs-counts-sync test: #12103 (Perplexity Agent) made it 352 providers; README, AGENTS.md, llm.txt (+42 i18n mirrors), package.json description and the 4 README diagrams still said 351. - api-route-typecheck: #11971 passes a third `{ featureEnabled }` argument to appendNoThinkingVariants() that the helper never accepted (TS2554 — and the flag silently did nothing); the helper now honours it. src/lib/skills/interception.ts narrowed a mapped object with a `Record<string, string>` predicate (TS2677) — predicate typed with the actual element shape. Gates: check:docs-counts OK (test 28/28), check:docs-sync PASS, check:api-typecheck OK (289 frozen). Refs #12103, #11971 * docs(env): document RATE_LIMIT_EXECUTION_MAX_WAIT_MS (#12027 added it to .env.example only) * fix(ci): round 5b — freeze the react-hooks compiler-rule violations, align 7 tests to merged contracts No new ESLint warnings: the exact CI command (lint:json --max-warnings 0) reports 278 problems on the tip — 226 from eslint-plugin-react-hooks 7 compiler rules (set-state-in-effect 167, immutability 36, refs/static-components/purity/ preserve-manual-memoization) that were masked until the lockfile change of dfc84ba invalidated the ESLint cache, plus 46 no-explicit-any in tests/unit/call-log-cap.test.ts (#12026). Velocity phase: frozen with `eslint --suppress-all` (+668 suppressions); the 5 now-unused `eslint-disable react-hooks/immutability` directives and one unused import removed. Verified: lint:json --max-warnings 0 → 0 problems. Tests aligned to contracts merged this afternoon (all reproduced red on the pure tip): - providers-constants-split: 235 → 236 (Perplexity Agent, #12103) - sse-auth: a forced pin outside allowedConnections now yields no credential instead of silently falling back (#12080) - with-chat-admission-10786: withInjectionGuard(postHandler, { logger: null }) (#12117) - hard-session-lease-bypass-inventory: classify src/app/api/oauth/codex/import/route.ts (#12116) - usage-service-hardening: OpenCode Go official usage API shape (#12124) - i18n placeholder parity: apiManager.restrictedToConnections rewritten as a plain ICU plural (`{count, plural, one {# connection} other {# connections}}`) in en, vi, pt-BR and the 40 __MISSING__ mirrors — the parity extractor counts every `{word}` including the old literal `{s}` Refs #12103, #12080, #12117, #12116, #12124, #12026 * fix(ci): run the ESLint warnings job on the box with an 8 GB heap; reserved-prefix set 398 → 400 The cold full lint with the react-hooks 7 compiler rules is killed on the 7 GB hosted runner with no message (status null → exit 1, JSON never written) — it only looked green while the ESLint cache was warm. tests/unit/provider-node-reserved-prefix.test.ts aligned to the two prefixes the afternoon batch registered (#12103). * test(ci): document the lint-guard runner exception; #9147 event-loop gap 400 → 800 ms quality-rail-gate-membership pinned lint-guard to ubuntu-latest; the cold full lint is OOM-killed there, so the job now runs on omni-light with an 8 GB heap — the test keeps fast-gates pinned and asserts the documented exception. With the catalog at 352 providers the hosted shards measure 410–633 ms gaps on 9147-catalog-eventloop-yield (3 runs); 800 ms still fails a true pin. Re-tighten with the v4.0 catalog split. * chore(quality): summarize the ESLint report on failure — a red lint:json printed nothing --format json --output-file swallows every problem; a red 'No new ESLint warnings' job gave zero output (three blind debugging rounds in #12144), and a killed process (OOM, status null) was equally silent. On any non-zero exit the runner now prints the problem count and the first 60 'file:line rule — message' lines from the report. * chore(lint): freeze react-hooks/immutability for the 5 UI test harnesses in the suppressions file The rule fires for these files in CI but not locally (compiler analysis divergence), so the inline eslint-disable directives read as 'unused directive' warnings locally. A suppressions entry is symmetric: suppressed where the rule fires, tolerated as unpruned (--pass-on-unpruned-suppressions) where it does not. Found via the new lint:json failure summary.

…wait budget; preserve errors in oversized call-log artifacts (diegosouzapw#12027) Desacopla a expiração de execução do rate-limit do orçamento de espera na fila, e preserva erros em artefatos de call-log oversized. Testes próprios (`call-log-cap.test.ts` + atualizações em `rate-limit-execution-timeout-message-4165.test.ts`/`ratelimit-admission-control-6593.test.ts`). Validado no worktree combinado. Obrigado!

…the bodies (diegosouzapw#12026) (diegosouzapw#12095) Mantém o erro do call-log quando o limite de tamanho corta os bodies, com `preserveErrorForSizeLimit` (UTF-8-safe, preserva o valor original quando cabe, trata erro circular/não-serializável) — implementação mais robusta que a alternativa que já estava na tip (via diegosouzapw#12027, que resolvi combinando: mantive a camada extra "errorOnly" do diegosouzapw#12027 usando o helper mais seguro deste). Testes próprios + os de diegosouzapw#12027 todos verdes (30/30) no worktree combinado. Obrigado!

…e/v3.8.51 (round 5: provider count 352, TS2554/TS2677) (diegosouzapw#12144) * fix(ci): clear the base-reds the 2026-08-30 afternoon merge batch left on release/v3.8.51 (round 5) - docs-counts / check-docs-counts-sync test: diegosouzapw#12103 (Perplexity Agent) made it 352 providers; README, AGENTS.md, llm.txt (+42 i18n mirrors), package.json description and the 4 README diagrams still said 351. - api-route-typecheck: diegosouzapw#11971 passes a third `{ featureEnabled }` argument to appendNoThinkingVariants() that the helper never accepted (TS2554 — and the flag silently did nothing); the helper now honours it. src/lib/skills/interception.ts narrowed a mapped object with a `Record<string, string>` predicate (TS2677) — predicate typed with the actual element shape. Gates: check:docs-counts OK (test 28/28), check:docs-sync PASS, check:api-typecheck OK (289 frozen). Refs diegosouzapw#12103, diegosouzapw#11971 * docs(env): document RATE_LIMIT_EXECUTION_MAX_WAIT_MS (diegosouzapw#12027 added it to .env.example only) * fix(ci): round 5b — freeze the react-hooks compiler-rule violations, align 7 tests to merged contracts No new ESLint warnings: the exact CI command (lint:json --max-warnings 0) reports 278 problems on the tip — 226 from eslint-plugin-react-hooks 7 compiler rules (set-state-in-effect 167, immutability 36, refs/static-components/purity/ preserve-manual-memoization) that were masked until the lockfile change of 9d8554f invalidated the ESLint cache, plus 46 no-explicit-any in tests/unit/call-log-cap.test.ts (diegosouzapw#12026). Velocity phase: frozen with `eslint --suppress-all` (+668 suppressions); the 5 now-unused `eslint-disable react-hooks/immutability` directives and one unused import removed. Verified: lint:json --max-warnings 0 → 0 problems. Tests aligned to contracts merged this afternoon (all reproduced red on the pure tip): - providers-constants-split: 235 → 236 (Perplexity Agent, diegosouzapw#12103) - sse-auth: a forced pin outside allowedConnections now yields no credential instead of silently falling back (diegosouzapw#12080) - with-chat-admission-10786: withInjectionGuard(postHandler, { logger: null }) (diegosouzapw#12117) - hard-session-lease-bypass-inventory: classify src/app/api/oauth/codex/import/route.ts (diegosouzapw#12116) - usage-service-hardening: OpenCode Go official usage API shape (diegosouzapw#12124) - i18n placeholder parity: apiManager.restrictedToConnections rewritten as a plain ICU plural (`{count, plural, one {# connection} other {# connections}}`) in en, vi, pt-BR and the 40 __MISSING__ mirrors — the parity extractor counts every `{word}` including the old literal `{s}` Refs diegosouzapw#12103, diegosouzapw#12080, diegosouzapw#12117, diegosouzapw#12116, diegosouzapw#12124, diegosouzapw#12026 * fix(ci): run the ESLint warnings job on the box with an 8 GB heap; reserved-prefix set 398 → 400 The cold full lint with the react-hooks 7 compiler rules is killed on the 7 GB hosted runner with no message (status null → exit 1, JSON never written) — it only looked green while the ESLint cache was warm. tests/unit/provider-node-reserved-prefix.test.ts aligned to the two prefixes the afternoon batch registered (diegosouzapw#12103). * test(ci): document the lint-guard runner exception; diegosouzapw#9147 event-loop gap 400 → 800 ms quality-rail-gate-membership pinned lint-guard to ubuntu-latest; the cold full lint is OOM-killed there, so the job now runs on omni-light with an 8 GB heap — the test keeps fast-gates pinned and asserts the documented exception. With the catalog at 352 providers the hosted shards measure 410–633 ms gaps on 9147-catalog-eventloop-yield (3 runs); 800 ms still fails a true pin. Re-tighten with the v4.0 catalog split. * chore(quality): summarize the ESLint report on failure — a red lint:json printed nothing --format json --output-file swallows every problem; a red 'No new ESLint warnings' job gave zero output (three blind debugging rounds in diegosouzapw#12144), and a killed process (OOM, status null) was equally silent. On any non-zero exit the runner now prints the problem count and the first 60 'file:line rule — message' lines from the report. * chore(lint): freeze react-hooks/immutability for the 5 UI test harnesses in the suppressions file The rule fires for these files in CI but not locally (compiler analysis divergence), so the inline eslint-disable directives read as 'unused directive' warnings locally. A suppressions entry is symmetric: suppressed where the rule fires, tolerated as unpruned (--pass-on-unpruned-suppressions) where it does not. Found via the new lint:json failure summary.

diegosouzapw and others added 3 commits August 29, 2026 03:15

alvinveroy requested a review from diegosouzapw as a code owner August 29, 2026 11:59

diegosouzapw changed the base branch from main to release/v3.8.51 August 30, 2026 13:18

diegosouzapw merged commit 838fc00 into diegosouzapw:release/v3.8.51 Aug 30, 2026
3 checks passed

diegosouzapw added a commit that referenced this pull request Aug 30, 2026

docs(env): document RATE_LIMIT_EXECUTION_MAX_WAIT_MS (#12027 added it…

71f74e6

… to .env.example only)

diegosouzapw mentioned this pull request Aug 30, 2026

fix(ci): clear the base-reds the afternoon merge batch left on release/v3.8.51 (round 5: provider count 352, TS2554/TS2677) #12144

Merged

diegosouzapw mentioned this pull request Sep 1, 2026

bug(sse): rate-limit execution expiration (requestQueue.maxWaitMs=15s) aborts long-running LLM requests with 504 #12092

Closed

maxmad64bis mentioned this pull request Sep 23, 2026

fix(rate-limit): keep queued jobs in their own async context #14621

Merged

5 tasks done

Navigation Menu

Uh oh!

fix(resilience): decouple rate-limit execution expiration from queue-wait budget; preserve errors in oversized call-log artifacts - #12027

fix(resilience): decouple rate-limit execution expiration from queue-wait budget; preserve errors in oversized call-log artifacts#12027
diegosouzapw merged 3 commits into
diegosouzapw:release/v3.8.51from
alvinveroy:fix/12025-decouple-rate-limit-execution-from-queue-wait

alvinveroy commented Aug 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

alvinveroy commented Aug 29, 2026

What

Tests

Notes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants