Skip to content

fix(resilience): recover idle wedged limiters - #8616

Merged
diegosouzapw merged 4 commits into
diegosouzapw:release/v3.8.49from
excessivechaos:observability/rate-limit-state
Jul 27, 2026
Merged

diegosouzapw merged 4 commits into
diegosouzapw:release/v3.8.49from
excessivechaos:observability/rate-limit-state

Conversation

@excessivechaos

@excessivechaos excessivechaos commented Jul 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Bottleneck can leave a request queued with no running or executing jobs even though reservoir capacity is available. The request then waits until the local queue budget expires.

This change:

  • Detects that idle-capacity wedge at queue expiry.
  • Evicts the stranded limiter and drops its orphaned queue.
  • Retries the never-dispatched request once on a fresh limiter.
  • Retains the watchdog reset for wedges that persist beyond the queue budget.

Validation

  • 10 rate-limit-manager tests pass
  • npm run typecheck:core
  • Focused ESLint on changed files
  • Regression test covers idle-capacity expiry and fresh-limiter retry

Follow-up to the recurring queue wedge behavior documented in #4165.

@excessivechaos excessivechaos changed the title chore(resilience): instrument local queue expiry fix(resilience): recover idle wedged limiters Jul 25, 2026
@excessivechaos
excessivechaos marked this pull request as ready for review July 25, 2026 22:13
@diegosouzapw

Copy link
Copy Markdown
Owner

Hi @excessivechaos — wedged limiter recovery is the right escape valve. Two things to confirm in the merge:

  1. The retry only fires when the limiter itself is the failure (not masking upstream 429s etc.).
  2. Bounded retries (no infinite loop) and a debug log so we can see when it triggers.

Approved as merge-ready (★3). /merge-prs will run the limiter/rate-limit focused tests.

@diegosouzapw
diegosouzapw merged commit dd22d9c into diegosouzapw:release/v3.8.49 Jul 27, 2026
3 checks passed
diegosouzapw added a commit that referenced this pull request Jul 27, 2026
…stryker drift

file-size: nine frozen entries could not absorb the combined result of the
31-PR train. Two distinct causes, kept apart in the baseline note on purpose:

  (1) GENUINE irreducible growth at existing chokepoints —
      providerLimits/auth (#8632), rateLimitManager (#8616),
      models-catalog-route.test (#8610).
  (2) COLLISION with #8585, which banked shrinks measured on the pre-train
      release tip while 30 sibling PRs in the SAME train grew those files
      again — chat/accountFallback (#8628), chatCore (#8613),
      videoGeneration (#8581), imageGeneration.

Ceilings re-pinned to the post-merge tip. #8612 (also in this train) automates
shrink-banking so this self-inflicted drift stops recurring.

stryker: three covering unit tests were missing from tap.testFiles —
isLocalStreamLifecycleError-abort-shape (circuitBreaker.ts, a shared base-red
that was reddening Fast Quality Gates on every open PR),
noauth-autocombo-lockout-7623 (accountFallback.ts) and
kimi-quota-reset-recovery (auth.ts), the latter two landed with this train.
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
* chore(resilience): log queue state on expiry

* docs(changelog): document rate limiter instrumentation

* fix(resilience): recover idle wedged limiters

* chore(resilience): remove diagnostic queue logging
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
…stryker drift

file-size: nine frozen entries could not absorb the combined result of the
31-PR train. Two distinct causes, kept apart in the baseline note on purpose:

  (1) GENUINE irreducible growth at existing chokepoints —
      providerLimits/auth (diegosouzapw#8632), rateLimitManager (diegosouzapw#8616),
      models-catalog-route.test (diegosouzapw#8610).
  (2) COLLISION with diegosouzapw#8585, which banked shrinks measured on the pre-train
      release tip while 30 sibling PRs in the SAME train grew those files
      again — chat/accountFallback (diegosouzapw#8628), chatCore (diegosouzapw#8613),
      videoGeneration (diegosouzapw#8581), imageGeneration.

Ceilings re-pinned to the post-merge tip. diegosouzapw#8612 (also in this train) automates
shrink-banking so this self-inflicted drift stops recurring.

stryker: three covering unit tests were missing from tap.testFiles —
isLocalStreamLifecycleError-abort-shape (circuitBreaker.ts, a shared base-red
that was reddening Fast Quality Gates on every open PR),
noauth-autocombo-lockout-7623 (accountFallback.ts) and
kimi-quota-reset-recovery (auth.ts), the latter two landed with this train.
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
* chore(resilience): log queue state on expiry

* docs(changelog): document rate limiter instrumentation

* fix(resilience): recover idle wedged limiters

* chore(resilience): remove diagnostic queue logging
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…stryker drift

file-size: nine frozen entries could not absorb the combined result of the
31-PR train. Two distinct causes, kept apart in the baseline note on purpose:

  (1) GENUINE irreducible growth at existing chokepoints —
      providerLimits/auth (diegosouzapw#8632), rateLimitManager (diegosouzapw#8616),
      models-catalog-route.test (diegosouzapw#8610).
  (2) COLLISION with diegosouzapw#8585, which banked shrinks measured on the pre-train
      release tip while 30 sibling PRs in the SAME train grew those files
      again — chat/accountFallback (diegosouzapw#8628), chatCore (diegosouzapw#8613),
      videoGeneration (diegosouzapw#8581), imageGeneration.

Ceilings re-pinned to the post-merge tip. diegosouzapw#8612 (also in this train) automates
shrink-banking so this self-inflicted drift stops recurring.

stryker: three covering unit tests were missing from tap.testFiles —
isLocalStreamLifecycleError-abort-shape (circuitBreaker.ts, a shared base-red
that was reddening Fast Quality Gates on every open PR),
noauth-autocombo-lockout-7623 (accountFallback.ts) and
kimi-quota-reset-recovery (auth.ts), the latter two landed with this train.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants