Skip to content

fix: release gate reservation on retry to prevent queue wedge (#311) - #334

Merged
jtechapps merged 2 commits into
llm-d:mainfrom
shimib:fix/gate-reservation-leak-311
Jul 22, 2026
Merged

fix: release gate reservation on retry to prevent queue wedge (#311)#334
jtechapps merged 2 commits into
llm-d:mainfrom
shimib:fix/gate-reservation-leak-311

Conversation

@shimib

@shimib shimib commented Jul 21, 2026

Copy link
Copy Markdown
Member

What / why

Fixes #311. The Redis sorted-set flow's per-queue gate reservation (a LocalConcurrencyGate slot or a redis-quota INCR) was released only by resultWorker (terminal results). The retry path (retryWorker) re-enqueued the request without releasing, so every retry leaked a reservation:

  • with LocalConcurrencyGate (no TTL), inFlight ratcheted up until Budget() hit 0 and the queue stopped dispatching permanently;
  • with redis-quota, it was masked by the key's TTL reset — which itself over-admits while requests are in flight.

Compounded by activeReleases.Store overwriting a prior closure on re-dispatch, orphaning it.

Changes (pkg/redis/sortedset_impl.go)

  • retryWorker now releases each retried request's reservation before re-enqueue (a re-dispatched request re-reserves via gate.Apply). Releasing before flushRetryBatch's ZAdd means a re-dispatch cannot race with / overwrite the reservation.
  • Dispatch-time store uses sync.Map.Swap and releases any prior closure defensively — never orphans a lingering reservation.
  • Regression test TestSortedSetFlow_RetryReleasesReservation for the retry→release path (previously untested — why the leak stayed latent). It fails without this fix and passes with it.

Not affected

Pub/Sub holds the reservation in a function-scoped defer that runs on every return path including Nack/retry — no analogous leak, so the fix is scoped to the sorted-set flow.

Testing

  • make test green across all modules.
  • New regression test verified to fail on main and pass with the fix.

Closes #311

)

The queue-gate reservation stored in RedisSortedSetFlow.activeReleases was
released only by resultWorker (terminal results). The retry path (retryWorker)
re-enqueued the request without releasing, so every retry leaked a reservation:
inFlight ratcheted up until Budget() reached 0 and the queue stopped dispatching
entirely (LocalConcurrencyGate, no TTL), or the redis-quota counter over-admitted
when its key TTL reset while requests were still in flight.

- retryWorker now releases each retried request's reservation before re-enqueue
  (a re-dispatched request re-reserves via gate.Apply), ordered before
  flushRetryBatch's ZAdd so the reservation cannot be overwritten/orphaned.
- Dispatch-time store uses sync.Map.Swap and releases any prior closure
  defensively, so a lingering reservation is never orphaned.
- Add a regression test for the retry->release path (previously untested, which
  is why the leak stayed latent).

Pub/Sub is unaffected: it holds the reservation in a function-scoped defer that
runs on every return path including Nack/retry.

Closes llm-d#311

Signed-off-by: Shimi Bandiel <shimib@google.com>
…k fix)

Signed-off-by: Shimi Bandiel <shimib@google.com>
@jtechapps
jtechapps merged commit e54ce53 into llm-d:main Jul 22, 2026
8 checks passed
jtechapps pushed a commit that referenced this pull request Jul 23, 2026
) (#336)

* fix: refresh redis-quota concurrency TTL to prevent over-admission (#335)

Concurrency-mode redis-quota tracks in-flight requests as an INCR/DECR
counter, but set the key's EXPIRE only on the first acquire (new_val==1)
and never refreshed it. The TTL equals window (factory default 1m; the
multitenant overlays don't set it). Under sustained per-tenant load the
counter stays positive, so ~window seconds after it first went positive
the key expires mid-flight, the counter resets to 0, and further requests
are admitted beyond the limit -- over-admission up to 'limit' extra
concurrent per tenant, recurring every window (and for any request in
flight longer than window).

This is the correctness residual after #311/#334: #334 made the counter
accurate (retries no longer leak INCRs), but an accurate-but-positive
counter was still wiped by the TTL.

- Refresh EXPIRE on every acquire so the counter cannot expire while there
  is activity; the key now expires only after window of total inactivity
  (crash-orphan cleanup).
- Refresh EXPIRE on release while reservations remain in flight.
- Add a regression test (TTL refresh across interleaved acquires) that
  fails before this change and passes after.

Rate-limit mode is unchanged (its sorted set expires per-member correctly).

Closes #335

Signed-off-by: Shimi Bandiel <shimib@google.com>

* docs(release-notes): add fragment for #336 (redis-quota TTL over-admission fix)

Signed-off-by: Shimi Bandiel <shimib@google.com>

---------

Signed-off-by: Shimi Bandiel <shimib@google.com>
This was referenced Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gate capacity reservation leaks on retry → inFlight ratchets up → queue wedges permanently

2 participants