Skip to content

fix(ratelimit): keep operator minTime floor when relaxing limiter on headroom (#9763) - #11086

Merged
diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.50from
pacocartones:fix/ratelimit-mintime-floor-9763
Aug 23, 2026
Merged

diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.50from
pacocartones:fix/ratelimit-mintime-floor-9763

Conversation

@pacocartones

Copy link
Copy Markdown
Contributor

Fixes #9763.

What

updateFromHeaders in open-sse/services/rateLimitManager.ts learns rate limits from upstream x-ratelimit-* headers. In the "plenty of headroom" branch (remaining > limit * 0.5) it relaxed the limiter by setting updates.minTime = 0 unconditionally:

} else if (remaining > limit * 0.5) {
  // Plenty of headroom — relax the limiter
  updates.minTime = 0;   // <-- overrides the operator's configured floor
  updates.reservoir = null;
  ...
}

That 0 overwrites resilienceSettings.requestQueue.minTimeBetweenRequestsMs — the minimum gap an operator explicitly configures (for example, to protect a sensitive self-hosted upstream). The override happens silently and persists for the life of the limiter, so a single well-provisioned response effectively disables the operator's throttle until the next restart. That is the behavior reported in #9763.

Notably the module already resolves that floor everywhere else through resolveMinTime(...) — buildLimiterDefaults() uses resolveMinTime(currentRequestQueueSettings.minTimeBetweenRequestsMs), and the RPM-derived assignment a few lines above already clamps with Math.max(0, ...). Only this headroom branch bypassed the floor.

Fix

Relax toward the configured floor instead of a hard 0:

updates.minTime = resolveMinTime(currentRequestQueueSettings.minTimeBetweenRequestsMs);

resolveMinTime returns 0 when no floor is configured, so the default (no operator throttle) behavior is unchanged. When a floor is set, headroom still lifts learned throttling down to — but never below — the operator's minimum spacing. Reservoir relaxation is untouched.

Test

New tests/unit/rateLimitManager-mintime-floor-9763.test.ts sets an operator floor of 200ms via applyRequestQueueSettings, injects a spy limiter through the existing __setLimiterFactoryForTests hook, drives a headroom response (limit=100, remaining=80) through updateFromHeaders, and asserts the limiter's minTime stays at 200. It fails on the current base (observed got 0) and passes with this change. The three existing tests in rateLimitManager-update-sequencing.test.ts only exercise the retry-after path, so this branch was previously uncovered.

Verified on the release/v3.8.50 base:

  • new + sequencing + headers-split: 11/11 pass
  • broader ratelimit sweep (idle-eviction, queue-timeout, reservoir-refresh, headers-split): 18/18 pass
  • RED without the fix, GREEN with it.

Changelog

Adds changelog.d/fixes/9763-ratelimit-mintime-floor.md.

@diegosouzapw
diegosouzapw merged commit 376b49d into cheaper-inference:release/v3.8.50 Aug 23, 2026
9 of 16 checks passed
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…cheaper-inference#9763) (cheaper-inference#11086)

Validated on the combined batch board over tip 429b730: static gates clean (changelog, file-size 158 frozen, complexity 2626<=2774, cognitive 1183<=1223, dead-code 409<=416), typecheck:core clean, 70 focused tests green (PR suites 49/49 + auth/combo neighbors 21/21).

Operator-configured positive minTime floor now survives the plenty-of-headroom relaxation (resolveMinTime instead of a hard 0). Fixes cheaper-inference#9763. Thank you @pacocartones!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Request Queue "Minimum time between requests" is overridden to 0 by adaptive rate-limit learning

2 participants