Skip to content

Bump base to nightly-0f173945 for the DSV4 KV-capacity fixes (vllm#48993, vllm#48317); drop vendored vllm#48748 - #36

Merged
fank merged 1 commit into
mainfrom
claude/bump-nightly-0f173945-dsv4-kv
Jul 31, 2026
Merged

Bump base to nightly-0f173945 for the DSV4 KV-capacity fixes (vllm#48993, vllm#48317); drop vendored vllm#48748#36
fank merged 1 commit into
mainfrom
claude/bump-nightly-0f173945-dsv4-kv

Conversation

@fank

@fank fank commented Jul 31, 2026

Copy link
Copy Markdown
Member

Why bump

The DeepSeek-V4 KV-capacity work all landed after the v0.26.0 branch cut, so none of it is in the current base:

PR what it does
#48993 packed KV group overlays — per-block cost drops from sum(groups) to max(groups), because a block id is owned by one group at a time
#48317 correctness fix: get_max_concurrency_for_kv_cache_config counted only ONE group's page size
#50312 448 MiB GPU memory saved (dsv4 pp buffer)
#50298 ~1.88× dsv4 kernel perf
#48957 ~2× on c128 layers (20 of 44)
#49486 3.4% decode TTFT
#50004 adaptive topk width, 1.0% E2E

#48317 is the headline, and it is not a win. DSV4 has three KV groups whose page sizes differ ~86× (149,760 / 33,984 / 1,728 B). The old formula collapsed them into one group's page size; the new one sums the real per-group requirement. Blocks-per-request therefore goes up and reported concurrency goes down — meaning every concurrency figure this repo has ever recorded (4.60×, 4.38×, 3.19×) was overstated, and the margin over the 4 × 262144 guarantee is thinner than believed. #48993's capacity gain may be needed to hold 4×, not to exceed it. Both land in this one bump.

Why a nightly again

v0.26.1rc0 carries #48993 and #48317 — but it is a git tag only, no image is published (vllm/vllm-openai:v0.26.1rc0 → 404). So a pinned nightly is the only way to get these today.

Nightly tags are pruned after ~2 weeks. Recorded in both the Dockerfile and the README: if a rebuild ever fails on an unresolvable FROM, that is why — move to the first release tag that is a superset rather than silently picking a newer nightly.

Branch-cut check

compare/v0.26.0...nightly-0f173945 reports diverged, ahead_by=481, behind_by=12 — not the ahead the README asks for. Investigated rather than trusted:

  • 6 of the 12 are CI/build only (macOS wheel, ROCm release pipeline, CI timeouts) — irrelevant.
  • The other 6 include #49467, which fixes DeepGEMM warmup for FlashInferFp8DeepGEMMDynamicBlockScaledKernelthe exact kernel our boot log selects — and #49294 (MLA context chunks).
  • All 6 were verified present in the nightly under different SHAs: they are release-branch backports of fixes that landed on main first. The behind_by is a SHA-comparison artifact, not missing functionality.

I've added a note to the Dockerfile so the next person hitting diverged checks for backports before treating it as a blocker.

Patch changes

  • #48748 dropped. Merged upstream 2026-07-22 and present in this base. Verified the mechanism the repo relies on: git apply of the vendored patch now fails against the nightly, which is exactly the "will fail loudly" contract in its Dockerfile comment. Its tripwire is kept, repurposed as a regression check that the base still carries the fix.
  • #46257 kept — still open upstream, verified still applies.
  • #48023 kept — still open upstream, verified still applies; file renamed to drop the now-wrong -on-v0.26.0 suffix.

Verification done

  • all 7 target PRs confirmed present in nightly-0f173945
  • both remaining patches git apply --check clean against the nightly
  • every tripwire symbol confirmed to exist at that SHA: _dflash_needs_multi_kv_group, SpeculativeConfig.kv_cache_dtype, dspark/DSparkModelTypes, DSML_PARAM_CLOSE, DeepSeekV4ParserReasoningAdapter, DeepSeekV4EngineToolParser, encode_messages, and the deepseek_v4/nvidia/dspark, spec_decode/dspark/speculator, qwen3_dflash, parser/deepseek_v32 modules
  • build_app still a plain module-level def, so the /collect_env wrapper still applies

Not verified locally: the image was not built here (no vllm installed; ~15 GB free vs a ~20 GB base). CI is the gate. The behavioural tripwires — DSpark, DFlash, add_generation_prompt, spec-draft model_weights — all run at build time, so a base regression fails on the runner rather than on a GPU pod.

What to measure on the first boot

num_gpu_blocks and the reported max concurrency. That figure will be the first correct one this deployment has produced. If it lands below 4× at max_model_len 262144, the 4 × 262144 guarantee was never actually met and --max-num-seqs 4 needs revisiting — better learned from a boot log than from a production stall.

Also stale after this bump: any --kv-cache-memory value derived on v0.26.0. The weight/activation footprint changes here, so re-derive it (drop the flag, boot, read the suggestion) rather than carrying the old number over.

…ndored vllm#48748

The DeepSeek-V4 KV work all landed after the v0.26.0 branch cut:

  #48993  packed KV group overlays — per-block cost sum(groups) -> max(groups)
  #48317  get_max_concurrency_for_kv_cache_config counted only ONE group page
          size, so every concurrency figure recorded so far was overstated
  #50312  448 MiB GPU memory saved (dsv4 pp buffer)
  #50298  ~1.88x dsv4 kernel perf;  #48957 ~2x on c128;  #49486 3.4% decode TTFT

v0.26.1rc0 carries #48993/#48317 but publishes no image, so a nightly it is.
Nightly tags are pruned after ~2 weeks — noted in the Dockerfile and README.

#48748 is in this base, so the vendored patch is deleted; its tripwire is kept
as a regression check that the base still carries the fix.
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@fank
fank marked this pull request as ready for review July 31, 2026 15:28
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@fank
fank merged commit 07dccc5 into main Jul 31, 2026
5 checks passed
@fank
fank deleted the claude/bump-nightly-0f173945-dsv4-kv branch July 31, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant