Bump base to nightly-0f173945 for the DSV4 KV-capacity fixes (vllm#48993, vllm#48317); drop vendored vllm#48748 - #36
Merged
Conversation
…ndored vllm#48748
The DeepSeek-V4 KV work all landed after the v0.26.0 branch cut:
#48993 packed KV group overlays — per-block cost sum(groups) -> max(groups)
#48317 get_max_concurrency_for_kv_cache_config counted only ONE group page
size, so every concurrency figure recorded so far was overstated
#50312 448 MiB GPU memory saved (dsv4 pp buffer)
#50298 ~1.88x dsv4 kernel perf; #48957 ~2x on c128; #49486 3.4% decode TTFT
v0.26.1rc0 carries #48993/#48317 but publishes no image, so a nightly it is.
Nightly tags are pruned after ~2 weeks — noted in the Dockerfile and README.
#48748 is in this base, so the vendored patch is deleted; its tripwire is kept
as a regression check that the base still carries the fix.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
fank
marked this pull request as ready for review
July 31, 2026 15:28
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why bump
The DeepSeek-V4 KV-capacity work all landed after the
v0.26.0branch cut, so none of it is in the current base:sum(groups)tomax(groups), because a block id is owned by one group at a timeget_max_concurrency_for_kv_cache_configcounted only ONE group's page size#48317 is the headline, and it is not a win. DSV4 has three KV groups whose page sizes differ ~86× (149,760 / 33,984 / 1,728 B). The old formula collapsed them into one group's page size; the new one sums the real per-group requirement. Blocks-per-request therefore goes up and reported concurrency goes down — meaning every concurrency figure this repo has ever recorded (4.60×, 4.38×, 3.19×) was overstated, and the margin over the
4 × 262144guarantee is thinner than believed. #48993's capacity gain may be needed to hold 4×, not to exceed it. Both land in this one bump.Why a nightly again
v0.26.1rc0carries #48993 and #48317 — but it is a git tag only, no image is published (vllm/vllm-openai:v0.26.1rc0→ 404). So a pinned nightly is the only way to get these today.Nightly tags are pruned after ~2 weeks. Recorded in both the Dockerfile and the README: if a rebuild ever fails on an unresolvable
FROM, that is why — move to the first release tag that is a superset rather than silently picking a newer nightly.Branch-cut check
compare/v0.26.0...nightly-0f173945reportsdiverged, ahead_by=481, behind_by=12 — not theaheadthe README asks for. Investigated rather than trusted:FlashInferFp8DeepGEMMDynamicBlockScaledKernel— the exact kernel our boot log selects — and #49294 (MLA context chunks).mainfirst. Thebehind_byis a SHA-comparison artifact, not missing functionality.I've added a note to the Dockerfile so the next person hitting
divergedchecks for backports before treating it as a blocker.Patch changes
#48748dropped. Merged upstream 2026-07-22 and present in this base. Verified the mechanism the repo relies on:git applyof the vendored patch now fails against the nightly, which is exactly the "will fail loudly" contract in its Dockerfile comment. Its tripwire is kept, repurposed as a regression check that the base still carries the fix.#46257kept — still open upstream, verified still applies.#48023kept — still open upstream, verified still applies; file renamed to drop the now-wrong-on-v0.26.0suffix.Verification done
nightly-0f173945git apply --checkclean against the nightly_dflash_needs_multi_kv_group,SpeculativeConfig.kv_cache_dtype,dspark/DSparkModelTypes,DSML_PARAM_CLOSE,DeepSeekV4ParserReasoningAdapter,DeepSeekV4EngineToolParser,encode_messages, and thedeepseek_v4/nvidia/dspark,spec_decode/dspark/speculator,qwen3_dflash,parser/deepseek_v32modulesbuild_appstill a plain module-level def, so the/collect_envwrapper still appliesNot verified locally: the image was not built here (no
vllminstalled; ~15 GB free vs a ~20 GB base). CI is the gate. The behavioural tripwires — DSpark, DFlash,add_generation_prompt, spec-draftmodel_weights— all run at build time, so a base regression fails on the runner rather than on a GPU pod.What to measure on the first boot
num_gpu_blocksand the reported max concurrency. That figure will be the first correct one this deployment has produced. If it lands below 4× atmax_model_len 262144, the4 × 262144guarantee was never actually met and--max-num-seqs 4needs revisiting — better learned from a boot log than from a production stall.Also stale after this bump: any
--kv-cache-memoryvalue derived onv0.26.0. The weight/activation footprint changes here, so re-derive it (drop the flag, boot, read the suggestion) rather than carrying the old number over.