Repository navigation
Conversation
Adds a modality-agnostic primitive to free a block-aligned token range of a request that stays alive, for long-running streaming-input sessions (vllm-project#28973) that must bound per-session KV: - SingleTypeKVCacheManager.evict_blocks_for_request(request_id, block_ids): frees the given blocks from a live request's block table, removes them from the prefix-cache index so their content can never be served to another request, and keeps num_cached_block consistent. - KVCacheCoordinator.evict_blocks_for_request: fan-out per group. - KVCacheManager.evict_token_range_for_request(request_id, start, end) -> (aligned_start, aligned_end, num_freed): rounds the range INWARD to whole blocks and returns the range actually freed, so the caller compacts its own per-token arrays by exactly the same span and the block table and token bookkeeping cannot go out of sync. Guards the hybrid (multi-group) case with an assert rather than fanning one token range to groups of different block sizes. No behavior change for requests that do not call it. Consumer follows in the bounded-session retention PR (RFC vllm-project#51948); the primitive is also usable for StreamingLLM-style sink+window text sessions or pruning stale tool-call context from a live session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
Four independent fixes for resumable (streaming-input, vllm-project#28973) sessions, found running multi-hour video-streaming sessions: 1. RequestOutputCollector merges DELTA outputs when the producer gets ahead of the consumer, and RequestOutput.add() overwrites finish_reason. Per-chunk boundaries are conveyed as transient finish_reason markers on non-finished outputs, so a merge across a boundary silently erases the marker and the consumer loses a chunk boundary. Replace the single-slot collector with a deque that never merges into an entry that is finished or carries a finish_reason. 2. process_outputs() treats ANY finish_reason on a streaming-input request as a chunk boundary. An engine-initiated ABORT or ERROR is terminal for the session; masking it wedges the consumer and desyncs the frontend and scheduler chunk queues. Gate boundary handling on the finish reason and route an abort back to the engine core. 3. Abort races: cancel AND await the input-stream task before aborting (close() only cancels, leaving a window where an in-flight chunk ADD is sent after the abort), and mark first chunks on the wire (EngineCoreRequest.first_chunk) so the scheduler rejects non-first chunks for unknown ids instead of resurrecting the finished session as an unabortable request that pins its KV blocks. 4. Zero-chunk streams put STREAM_FINISHED instead of running the placeholder final request as a real generation; sticky DP routing keeps all chunks of a session on the engine that holds it; sessions salt their prefix-cache hash chain (chunk folds mutate block content under unchanged hashes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
…KV eviction, consolidation Engine core for RFC vllm-project#51948: give a long-running streaming-input session (vllm-project#28973) a constant memory footprint, for continuous video understanding and StreamingLLM-style sink+window text sessions. Mechanism (shared between text and multimodal sessions): - vllm/v1/streaming/: retention policy (per-session HistorySegment bookkeeping + sliding-window budgets), eviction (drop the oldest segments of a LIVE session in place via the KV range-free primitive, releasing encoder-cache entries and compacting all per-token arrays in lockstep by the block-aligned range actually freed), consolidation (self-preempting re-prefill of the surviving window before the model's trained position range is exceeded), chunk-relative position helpers. - Survivors keep their original positions; the position space develops gaps. Cached K stays rotationally consistent with no kernel changes. mRoPE models carry a per-token position tensor, extended per chunk from max_cached_position + 1 and sliced on eviction. 1D-RoPE (text) models need only a per-request scalar: RoPE position = index + cumulative evicted width, reset at re-prefill (the degenerate case of the same design, so LLM and VLM sessions share the mechanics end to end). - Scheduler: segment recording at chunk folds, over-budget eviction, re-prefill trigger + self-preemption (phantom-sample discard on idle triggers), fold-time cumulative context-length guard, grammar soft-stop for sessions, admission checks (single KV-cache group, no async scheduling / PP / DP for retention sessions). - Worker: position slice/extend/reset mirroring the scheduler, max_cached_position watermark reporting, persistent encoder cache for re-prefill (immediate physical eviction of dead entries), orphan mm-placeholder neutralization, streaming-aware encoder cache floor. Sessions that do not set streaming_retention (in SamplingParams.extra_args) take none of these paths. Draft caveats: 1D-RoPE position shift is applied to the shared positions buffer post-slot-mapping; backends consuming CommonAttentionMetadata.positions for masking are audited for the default FA path only. GPU E2E evidence to follow. 86 unit tests (tests/v1/streaming/). Existing suite failure sets are byte-identical to base on the same host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
mergify
Bot
added
multi-modality
nvbfalk
changed the title
This was referenced Aug 26, 2026
[RFC]: Bounded-memory video sessions — KV retention for long-running streaming-input requests
#51948
Open
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Draft implementation of RFC #51948: give a long-running streaming-input session
(#28973) a constant memory footprint, no matter how long it runs. Nothing here activates unless a request sets
streaming_retentioninSamplingParams.extra_args- requests that don't take none of the new paths.Size, before anything else. The diff looks large, so here is where the lines
actually are. The first two commits are #53861 and #53871. The feature commit itself is
5,573 lines, of which 4,029 are tests.
What it does. Each session's context is tracked as segments (a pinned anchor =
system prompt + first turn; content segments; reply segments). When a session goes
over its budget, the oldest unpinned segments are evicted in place while the session
keeps running: KV blocks freed through the #53871 primitive, encoder-cache entries
released, and every per-token array compacted by exactly the block-aligned range the
KV manager reports back.
Positions: survivors keep theirs. After an eviction the position space has gaps.
Cached K stays rotationally consistent with no kernel changes and no re-rotation. We
also built the alternative (compact survivors to new positions and re-rotate cached K,
which is what #43374's design delegates to an out-of-tree runner) and benchmarked both
on the same model and harness: keeping positions was clearly ahead on accuracy at
identical latency.
Shared mechanics for text and multimodal sessions:
mRoPE models carry a per-token position tensor, extended each chunk from
max_cached_position + 1and sliced on eviction. For 1D-RoPE text models the samedesign collapses to a per-request scalar — RoPE position = token index + cumulative
evicted width, reset at re-prefill.
Consolidation. Position numbers still creep upward, so before the model's trained
range is exceeded the session self-preempts and re-prefills its surviving window from
position 0. Cheap: surviving frames' embeddings come from the (persistent) encoder
cache, so the vision tower does not re-run.
Guards. Admission requires a single KV-cache group and rejects async scheduling,
pipeline parallelism, and DP for retention sessions. A fold-time guard finishes
sessions that outgrow
max_model_len(applied to retention sessions only).Test Plan
pytest tests/v1/streaming/
86 unit tests: eviction and its edge cases, prefix-cache interaction, re-prefill
(848-line suite incl. phantom-sample discard), worker-side position slicing, mRoPE
chunk continuation, text-session position offsets, and a regression suite for the
session lifecycle around consolidation.
E2E: served
Qwen/Qwen3-VL-8B-Instructon an H100 with the session API PR stacked ontop, pushed synthetic frames through one session.
Test Result
identical to main on the same machines.
frame's content correctly including all frames past the window (evictions active).
TTFT 322 ms for the first frame, then flat ~40 ms for every following frame.
41 ms (without eviction it would exceed
max_model_lenaround frame 180).Consolidation fired 7 times.
Portions of this work were developed with AI assistance; all of it has been reviewed
and tested by me.