Repository navigation
Conversation
|
Documentation preview: https://vllm--60526.org.readthedocs.build/en/60526/ |
mergify
Bot
added
documentation
e895851 to
c1b6226
Compare
|
Thanks for putting this together — the One design concern before this goes further: the descriptor/API layer looks like a third parallel schema next to the existing metadata-only paths, and I think it should be aligned with what vLLM already has instead:
The lifecycle contract itself (bypassing the receiver cache for intentionally-absent media, and failing the request whenever KV coverage is missing — including after preemption) has no counterpart in the EC path and does need new logic; my suggestion is only that the descriptor and admission layers not fork from the existing protocol. |
c1b6226 to
ca3214b
Compare
HanFa
changed the title
|
This pull request has merge conflicts that must be resolved before it can be |
ca3214b to
f15e70b
Compare
|
Hmmm, I think passing |
|
This pull request has merge conflicts that must be resolved before it can be |
977423e to
020b5e7
Compare
|
Thanks @Isotr0py. Two separate points, and I agree with you on the first one.
If the concern is the amount of surface this adds, I am happy to cut it down to the engine contract only: |
|
Reviewed and signed off by @HanFa Benchmark update: InternVL3-8B and LLaVA-1.5-7B, same-host H100 P/D, three decode-side forms Per @DarkLight1337's suggestion to use a model with heavier image processing, and @Isotr0py's question about pixel-less decode, I reran on 
Arms. Same prefill engine, decode engine, NIXL transfer, prompt tokens and 128 output tokens; only what the decode side receives differs.
Setup. 2 x H100 80GB in one pod, one P and one D engine, NIXL over UCX Results. p50 over 72 requests per cell. "Decode-attr." is TTFT minus the prefill adapter's own time; at C8 the prefill side's p50 moves by +-100 ms between arms from CPU contention, so that column is the one to compare arms on.
Handoff vs full media: TTFT p50 -39% / -33% (InternVL3 C1 / C8) and -43% / -38% (LLaVA), p95 -23% to -42%, C8 throughput +15-17%, decode request body 5.9 MB -> 5-48 KB. Decode submit-to-first-token is equal across arms, so the whole difference is the decode-side input preparation. Parity. LLaVA: 8/8 requests bit-identical across arms for all 128 tokens. InternVL3: of five serial passes over the same images, passes 2-5 are bit-identical to each other; pass 1, the only one where prefill computed the prompt from scratch instead of hitting its prefix cache, differs from every later pass at the same positions. Given the same KV, handoff equals the full-media output bit for bit. Takeaways.
Bug found. Limitations. In-process Update (additional configurations, InternVL3-8B, same setup). Four images per request (4 x 4032x3024, 13348 prompt tokens,
At C1 the decode side re-processes four 12 MP images for 1.66 s per request; handoff removes it (TTFT -46%). At C8 the single prefill engine is saturated by eight 13k-token prompts (P total 4.7-6.7 s), so TTFT is prefill-bound and the gain shrinks to -4% with throughput +6%; the closed loop also pushes more load onto prefill in the arms whose decode side is faster, which is why their P total is higher. One observation that is not specific to this PR: decode submit-to-first-token is ~85 ms higher for requests that carry multimodal features at this prompt length (full 153 ms, handoff 142 ms) than for the feature-less text-only request (57 ms); with one image it was equal across arms. Worth profiling separately, since it is the per-request cost of keeping multimodal-aware block hashes on decode. Decode engine pinned to 8 CPU cores (
With eight cores the baseline's decode-side preparation grows from 330 ms (C1) to 405 ms (C8) and 667 ms (C16) while handoff stays at 0.7 ms; TTFT p50 -41% at C8 and -12% at C16, throughput +17% / +6%. At C16 the prefill engine is again the bottleneck (P total 1.5-1.8 s), which caps the relative gain in this single-P/single-D topology. Image-size control, LLaVA-1.5-7B with 640x480 inputs (~170 KB as base64, otherwise the same protocol as the main LLaVA matrix):
This reproduces the earlier LLaVA result: with small images the decode-side preparation is 7-10 ms and TTFT improves by 7% (C1) / 2% (C8). The same model with 12 MP inputs pays 230 ms and improves by 43% / 38%. The benefit scales with media size and preprocessing cost, not with the model alone. AI assistance (Claude Code) was used for the harness, the runs and this write-up; all numbers are from the runs described. |
b837be7 to
d37246d
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
d37246d to
880acdd
Compare
|
@claude review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
880acdd to
51cb98c
Compare
|
Reviewed and signed off by @HanFa @DarkLight1337 here is the data behind the "
Arms. Setups (one per x-axis group in the chart):
C1 / C8 = conversations in flight. Every setup: 32 conversations per round, 2 rounds, arms paired on identical images and conversations, prefix caches on both engines reset before each round so the first request is cold in both arms, 640x480 single-color images with a one-word color question, 16 output tokens, temperature 0, every answer checked against the image's color. What is measured (decode engine's own counters, deltas over a round): local hit rate =
Reading it. The hit rates are the block arithmetic: On one host over NVLink the saved transfer is tens of milliseconds (decode submit-to-first-token 53 vs 71 ms at C1, 129 vs 201 ms at C8 for the 2100-token prefix; TTFT -15% / -29%). Where the bytes cost more it is seconds (bottom-right panel): with UCX forced to Correctness was identical across arms: all system-prompt, control and pressure/TCP answers correct, and the LLaVA-1.5 multi-image misses (75% on the reference-image setup, 79-92% on the 4-turn chat) agree across arms on 637 of 640 paired requests. Mechanism check without a GPU: building the same requests and hashing them with Limitations: in-process |
|
This pull request has merge conflicts that must be resolved before it can be |
|
IIUC, to fix the block corruption issue, we just need to append So I significantly feel most of engine side changes and new implementation are unnecessary here. |
…ith features.requires_kv Let a decode instance continue a multimodal request from decoder KV that already exists, locally or through a KV connector, without the media and without encoder capacity. The Generate API input carries the prefill-side token ids plus item hashes and placeholder positions, and `features.requires_kv` states that every item must be covered by KV. Engine contract: - `MultiModalFeatures.requires_kv` (omitted from the wire format when false) rejects serialized payloads and empty hashes; `engine_input_from_features` builds the engine input and `serve_tokens` uses it. `MultiModalInput.mm_requires_kv` and `MultiModalFeatureSpec.requires_kv` carry the flag through IPC. - Admission exempts KV-only inputs from the encoder-cache capacity check and validates them instead: nonempty hash and no payload per item, spans within the prompt and ending before the final prompt token (which is always recomputed); prompt logprobs, pooling, M-RoPE and encoder-decoder models are rejected. - The scheduler caches `Request.mm_required_kv_tokens` and fails the request instead of recomputing a span from placeholder ids whenever `num_computed_tokens` is below it, including after preemption and failed KV loads; streaming sessions extend the boundary. - The receiver cache skips `requires_kv` features and the renderers refuse to render such inputs again. Because the decode input carries prefill's hashes and positions, decode computes the same block hashes as prefill, so prefix caching and hash-keyed KV stores see the same blocks on both sides. Compared with the previous revision, the Python export/restore helpers, the kv_handoff descriptor and model fingerprint, the vllm/inputs/mm_features package move and the is_embed round-trip are dropped; the latter will be a separate PR. Closes vllm-project#60521 Co-authored-by: Codex <noreply@openai.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Fang Han <hanfa@umich.edu>
51cb98c to
e8ea00b
Compare
Purpose
Let a decode instance continue a multimodal request from existing decoder KV, without the media and without encoder capacity. Closes #60521.
This revision is the engine contract only, cut down to the scope discussed in the review (@Isotr0py's
mm-kv-minimaland the reduced scope offered on Oct 26):features.requires_kvonMultiModalFeatures. All items must already be covered by decoder KV; they carry hashes and placeholder positions only, and the validator rejects serialized payloads inkwargs_data/mm_metadataand empty hashes. The field is omitted from the wire format when false.engine_input_from_featuresturns a Generate request into the engine input (mm_requires_kv=True), andserve_tokensuses it.MultiModalInput.mm_requires_kvandMultiModalFeatureSpec.requires_kvcarry the same flag per request and per item through IPC.InputProcessor): KV-only inputs are exempt from the encoder-cache capacity check, so a decode worker with--limit-mm-per-prompt '{"image": 0}'admits them. They must carry a nonempty hash and no payload per item, stay within the prompt, and end before the final prompt token, which is always recomputed. Prompt logprobs, pooling, M-RoPE and encoder-decoder models are rejected: M-RoPE derives its positions from the media the input does not carry, so it needs a metadata channel first.Request.mm_required_kv_tokenscaches the furthest required position._try_schedule_encoder_inputsfails the request (FINISHED_ERROR) whenevernum_computed_tokensis below it, instead of running the decoder over placeholder ids. The check re-applies after preemption and after failed KV loads, and_update_request_as_sessionraises the boundary when a streaming session appends arequires_kvfeature.requires_kvfeatures (data=Noneis a deliberate absence, not a cache reference), and the renderers refuse to render such inputs again, soLLM.generate()cannot silently drop the requirement.With the prefill-side hashes and placeholder positions, the decode engine computes the same block hashes as prefill, so the local prefix cache and hash-keyed KV consumers see the same blocks on both sides; the alternative of folding image hashes into
cache_saltrewrites every decode block hash (data).Removed from the previous revision: the Python
prepare_multimodal_kv_handoff/restore_multimodal_kv_handoff/export_multimodal_kv_handoffhelpers, thekv_handoffversion/position-state descriptor and the model fingerprint, thevllm/inputs/mm_featurespackage (types and serde stay intoken_in_token_out/), theis_embedround-trip onPlaceholderRangeInfo(separate PR), and the long docs section. Callers that embedAsyncLLMbuild the input withmm_input(...)plusmm_requires_kv=True, or throughengine_input_from_features.Test Plan
CPU-only unit tests on the rebased tree (Linux, Python 3.12, torch 2.14.0, editable install with the precompiled extensions of
866bccf083):tests/multimodal/test_mm_requires_kv.py(new): block hashes equal prefill's and differ across images; the receiver cache skips onlyrequires_kvfeatures; the flag survives engine IPC; admission with a zero encoder budget, and rejection of payloads, empty hashes, out-of-range spans, spans reaching the last token, prompt logprobs, pooling and M-RoPE models; ordinary multimodal inputs still need encoder capacity;LLM.generate()preprocessing and the async renderer reject KV-only inputs.tests/v1/core/test_scheduler.py::test_media_free_*: fail-closed below the boundary, re-check after preemption, async load lifecycle (success / failure / cancel, with and withoutrecompute), sync load failure, local prefix-cache coverage without a connector (block-granularity limit), streaming-session boundary update.tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.pyandtest_generate_stream.py: the wire format omitsrequires_kvwhen false, restores a media-free engine input, rejects payloads and empty hashes;serve_tokenspasses the flag through without rendering or media.pre-commit: ruff, ruff-format and the repo's mypy wrapper on the changed files.Test Result
The branch is a single commit,
e8ea00b1cf, onmainat39dbf51839. Results on this tree (Linux, Python 3.12, torch 2.14.0, one RTX 5090, editable install with the precompiled extensions of866bccf083):tests/multimodal/test_mm_requires_kv.py: 18 passed.tests/v1/core/test_scheduler.py: 299 passed, 3 failed (test_async_scheduling_pp_allows_rescheduling_with_output_placeholders,test_max_loras_counts_running_request_awaiting_output,test_diffusion_read_deferral_keeps_a_longer_pp_wait); the same three fail identically on a cleanupstream/mainworktree at39dbf51839in this environment, so they are unrelated to this change. Alltest_media_free_*tests and the FSM-abort test pass.tests/entrypoints/scale_out/token_in_token_out/: 141 passed in the unit files, and the 29 server-backed tests intest_serving_tokens.pyandtest_serving_multimodal_tokens.py(a realvllm serve, including the render → generate round trip) pass on the GPU.tests/entrypoints/scale_out/renderandtests/renderers: identical results on this branch and on clean main (24 passed, 38 fixture errors from models unavailable in this environment), so the renderer guard changes nothing there.tests/multimodal/test_cache.py: 34 passed;test_sleep_wake_preserves_mm_cache_consistencyfails identically on clean main here.multimodal/inputs.py,scheduler.pyandinput_processor.pyare present on clean main with the same toolchain), typos, markdownlint-cli2, SPDX headers, forbidden and lazy imports,torch.cudacalls, test tethering: all passed.GPU evidence from earlier revisions of the same contract (LLaVA-1.5 / InternVL3, NIXL P/D): latency and throughput and decode-side prefix caching. The engine behaviour exercised there (admission exemption, scheduler coverage check, block-hash parity) is unchanged in this revision; the removed pieces were orchestrator-side helpers.
Dynamo is one motivating caller: HanFa/dynamo#2, ai-dynamo/dynamo#15343. AI assistance was used (Codex and Claude Code) for the implementation, tests and this description; the commit carries DCO sign-off and attribution trailers.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.