Repository navigation
Conversation
Adds a modality-agnostic primitive to free a block-aligned token range of a request that stays alive, for long-running streaming-input sessions (vllm-project#28973) that must bound per-session KV: - SingleTypeKVCacheManager.evict_blocks_for_request(request_id, block_ids): frees the given blocks from a live request's block table, removes them from the prefix-cache index so their content can never be served to another request, and keeps num_cached_block consistent. - KVCacheCoordinator.evict_blocks_for_request: fan-out per group. - KVCacheManager.evict_token_range_for_request(request_id, start, end) -> (aligned_start, aligned_end, num_freed): rounds the range INWARD to whole blocks and returns the range actually freed, so the caller compacts its own per-token arrays by exactly the same span and the block table and token bookkeeping cannot go out of sync. Guards the hybrid (multi-group) case with an assert rather than fanning one token range to groups of different block sizes. No behavior change for requests that do not call it. Consumer follows in the bounded-session retention PR (RFC vllm-project#51948); the primitive is also usable for StreamingLLM-style sink+window text sessions or pruning stale tool-call context from a live session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
Four independent fixes for resumable (streaming-input, vllm-project#28973) sessions, found running multi-hour video-streaming sessions: 1. RequestOutputCollector merges DELTA outputs when the producer gets ahead of the consumer, and RequestOutput.add() overwrites finish_reason. Per-chunk boundaries are conveyed as transient finish_reason markers on non-finished outputs, so a merge across a boundary silently erases the marker and the consumer loses a chunk boundary. Replace the single-slot collector with a deque that never merges into an entry that is finished or carries a finish_reason. 2. process_outputs() treats ANY finish_reason on a streaming-input request as a chunk boundary. An engine-initiated ABORT or ERROR is terminal for the session; masking it wedges the consumer and desyncs the frontend and scheduler chunk queues. Gate boundary handling on the finish reason and route an abort back to the engine core. 3. Abort races: cancel AND await the input-stream task before aborting (close() only cancels, leaving a window where an in-flight chunk ADD is sent after the abort), and mark first chunks on the wire (EngineCoreRequest.first_chunk) so the scheduler rejects non-first chunks for unknown ids instead of resurrecting the finished session as an unabortable request that pins its KV blocks. 4. Zero-chunk streams put STREAM_FINISHED instead of running the placeholder final request as a real generation; sticky DP routing keeps all chunks of a session on the engine that holds it; sessions salt their prefix-cache hash chain (chunk folds mutate block content under unchanged hashes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
…KV eviction, consolidation Engine core for RFC vllm-project#51948: give a long-running streaming-input session (vllm-project#28973) a constant memory footprint, for continuous video understanding and StreamingLLM-style sink+window text sessions. Mechanism (shared between text and multimodal sessions): - vllm/v1/streaming/: retention policy (per-session HistorySegment bookkeeping + sliding-window budgets), eviction (drop the oldest segments of a LIVE session in place via the KV range-free primitive, releasing encoder-cache entries and compacting all per-token arrays in lockstep by the block-aligned range actually freed), consolidation (self-preempting re-prefill of the surviving window before the model's trained position range is exceeded), chunk-relative position helpers. - Survivors keep their original positions; the position space develops gaps. Cached K stays rotationally consistent with no kernel changes. mRoPE models carry a per-token position tensor, extended per chunk from max_cached_position + 1 and sliced on eviction. 1D-RoPE (text) models need only a per-request scalar: RoPE position = index + cumulative evicted width, reset at re-prefill (the degenerate case of the same design, so LLM and VLM sessions share the mechanics end to end). - Scheduler: segment recording at chunk folds, over-budget eviction, re-prefill trigger + self-preemption (phantom-sample discard on idle triggers), fold-time cumulative context-length guard, grammar soft-stop for sessions, admission checks (single KV-cache group, no async scheduling / PP / DP for retention sessions). - Worker: position slice/extend/reset mirroring the scheduler, max_cached_position watermark reporting, persistent encoder cache for re-prefill (immediate physical eviction of dead entries), orphan mm-placeholder neutralization, streaming-aware encoder cache floor. Sessions that do not set streaming_retention (in SamplingParams.extra_args) take none of these paths. Draft caveats: 1D-RoPE position shift is applied to the shared positions buffer post-slot-mapping; backends consuming CommonAttentionMetadata.positions for masking are audited for the default FA path only. GPU E2E evidence to follow. 86 unit tests (tests/v1/streaming/). Existing suite failure sets are byte-identical to base on the same host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
Session frontend for RFC vllm-project#51948, on top of the bounded-session core: - POST /v1/streaming/sessions: open a session (model, prompt template, retention budgets, sampling); returns a session id. - POST /v1/streaming/sessions/{id}/frames: push one chunk (video frame or text turn), block until that chunk's reply is complete, return it. One-chunk-in-flight per session by design: the producer paces the engine so replies stay aligned with the chunks they describe. - DELETE /v1/streaming/sessions/{id}: close (also closed on client disconnect and engine error; fail-closed). Gated behind --enable-streaming (off by default). Startup guards fail fast on incompatible configs instead of dying at first use: multimodal processor cache is forced off (evicted-frame content must never be served from a data-less cache hit), async scheduling defaults off, pipeline parallelism and multi-API-server configs are rejected (the session registry is per-process). Transport shape is a proposal: happy to converge with /v1/realtime (vllm-project#33187) or the ingest path in vllm-project#50570 instead; the session layer is a thin adapter over AsyncLLM.generate(stream) either way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
mergify
Bot
added
frontend
multi-modality
nvbfalk
changed the title
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Draft session frontend for RFC #51948, on top of the bounded-session core PR
(#53876 — this branch is stacked on it, so its diff shows up here until it merges;
this PR's own commit is +1,326, of which 291 are tests). Gated behind a new
--enable-streamingflag, off by default; with the flag off, no route isregistered and nothing changes.
The API is the smallest lifecycle that makes the core usable:
POST /v1/streaming/sessions— open a session (task prompt, retention budgets,sampling); returns a session id.
POST /v1/streaming/sessions/{id}/frame— push one chunk (a JPEG frame or a textturn, raw request body); blocks until that chunk's reply is complete and returns
it, with per-frame timing. One chunk in flight per session by design: the client
paces the engine, so replies always stay aligned with the chunks they answer.
DELETE /v1/streaming/sessions/{id}— close. Sessions also close on clientdisconnect and on engine error (fail-closed).
Startup guards fail fast instead of dying at first use: the multimodal processor
cache is forced off (a later identical frame must not hit a data-less cache entry
for content whose KV was evicted), async scheduling defaults off, and pipeline
parallelism / multiple API servers are rejected (the session registry is
per-process).
The transport shape is a proposal, not a position.: it could equally be a video
modality on
/v1/realtime(#33187) or converge with the RTSP ingest path in #50570.The session layer here is a thin adapter over
AsyncLLM.generate(stream), so whichever shape maintainers prefer,the others become adapters.
Test Plan
pytest tests/v1/streaming/test_caption_server.py
12 tests: session lifecycle, unknown-session 404s, disconnect cleanup,
concurrent-session rejection, retention-parameter plumbing, oversized-frame
rejection.
E2E:
vllm serve Qwen/Qwen3-VL-8B-Instruct --enable-streaming --limit-mm-per-prompt '{"image": 16}'on an H100, driving synthetic framesthrough one session.
Test Result
main.
(the model reads each frame's content, including all frames past the window),
TTFT 322 ms for the first frame and flat ~40 ms for every frame after.
flat at 41 ms at frame 300. The startup guards were observed doing their job in
the server log.
Portions of this work were developed with AI assistance; all of it has been
reviewed and tested by me.