Repository navigation
[Chore] Use upstream Transformers DeepseekV4Config - #61392
Merged
Merged
Conversation
Split from vllm-project#60823. Derives the legacy compress_ratios, num_hash_layers and mm-prefix flags from the upstream layer_types / mlp_layer_types fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…ng, evals for vllm/models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Keep get_compress_ratios only where V4.1 configs also flow, inline the mm-prefix clamp check and drop PR numbers from the skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
cjackal
marked this pull request as ready for review
October 14, 2026 04:28
cjackal
requested review from
AndreasKaratzas,
DarkLight1337,
LucasWilkinson,
MatthewBonanni,
benchislett,
dllehr-amd,
hongxiayang,
njhill,
pavanimajety,
tjtanaa,
ywang96 and
zyongye
as code owners
October 14, 2026 04:28
hmellor
added a commit
to hmellor/vllm
that referenced
this pull request
Oct 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
5 of 6 tasks
Isotr0py
approved these changes
Oct 14, 2026
hmellor
enabled auto-merge (squash)
October 14, 2026 13:05
Member
Author
|
/ci run --allow-stale |
|
✅ Triggered Buildkite CI #94801 for commit
|
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #94802 for commit |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #94855 for commit ℹ️ This PR is 11 commits behind upstream |
Member
Author
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #94855. |
This was referenced Oct 15, 2026
michael-gpu
pushed a commit
to practical-inference/vllm-ampere-extended
that referenced
this pull request
Oct 20, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
5 tasks
3 of 5 tasks
zzzzwwjj
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Oct 23, 2026
### What this PR does / why we need it?
Port `vllm-ascend` to vLLM main `c0b6f74` (bumped from `a98247a`) while
keeping the
`v0.31.0` release lane green. The upstream window moved the KV-cache
planning
helpers into a new module, made EAGLE draft groups explicit, typed
DeepSeek-V4
configs through upstream Transformers, removed Medusa/MLP speculator
support, and
changed several spec-decode/KV-cache/EPLB signatures. Every gate below
is either a
`vllm_version_is("0.31.0")` split or an attribute/argument probe where
two main dev
builds share one version string.
| Files | Upstream vLLM change (why it breaks vllm-ascend) | vllm-ascend
adaptation |
|-------|--------------------------------------------------|------------------------|
| `.github/vllm-main-verified.commit` | — (main-lane pin only) |
Verified main commit bumped `a98247a` -> `c0b6f74` |
| `tests/e2e/pull_request/four_card/test_minimax_m3.py` | The main-lane
pin re-JITs the static kernels, so the first generate on a cold cache
shifts the near-uniform dummy logprobs (~3e-3) enough to flip a greedy
token | On main run a throwaway `generate_greedy_logprobs` before the
baseline/replay pair so both measure the warmed steady state and share
the cached prefix; v0.31.0 keeps the single-shot comparison |
| `tests/ut/core/test_circular_buffer.py` |
[vllm#61126](vllm-project/vllm#61126)
(cascade-attention removal) moved the prefix-block query up to
`KVCacheManager` and dropped it from single-type managers | On main
assert only the manager's own `get_num_skipped_tokens(65536)`; keep the
combined `get_num_common_prefix_blocks(...)` assert on v0.31.0 |
| `tests/ut/core/test_kv_cache_interface.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`is_kv_cache_spec_uniform` into `vllm.v1.core.kv_cache_planning` |
Version-aware import (`kv_cache_utils` on v0.31.0 / `kv_cache_planning`
on main) |
| `tests/ut/distributed/ascend_store/_mock_deps.py` |
[vllm#61183](vllm-project/vllm#61183) added the
`vllm.v1.core.kv_cache_planning` module | Add it to the mocked-vLLM
module list |
| `tests/ut/distributed/ascend_store/test_request_lifecycle.py` |
[vllm#58668](vllm-project/vllm#58668) trims
`num_computed_tokens` to the tokens actually kept when a request stops
mid token-list | On main subtract `(accepted + 1) - <newly appended
output tokens>` from the expected computed count; keep the old
expectation on v0.31.0 |
| `tests/ut/distributed/kv_transfer/kv_p2p/test_mooncake_connector.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`get_kv_cache_config_from_groups` / `is_kv_cache_spec_uniform` |
Version-aware import |
| `tests/ut/models/test_deepseek_v41_cache.py` |
[vllm#61183](vllm-project/vllm#61183) relocated
the planning helpers;
[vllm#53558](vllm-project/vllm#53558) gave
`_max_memory_usage_bytes_from_groups` a leading `builder` argument |
Import from the live planning module and probe the upstream signature to
call the 2-arg (v0.31.0) vs 3-arg (main) form, feeding an Ascend
`_get_pool_bytes_per_block` builder |
| `tests/ut/ops/test_gdn_attn_builder.py` |
[vllm#58737](vllm-project/vllm#58737) derives
`num_reqs` from the block table and asserts its row count equals
`seq_lens`; Ascend's `unpadded()` keeps the graph-padded rows | New UT:
a padded 4-row block table with 2 real requests must still yield a 2-row
`cache_indices` |
| `tests/ut/patch/platform/test_packed_kv_cache_compat.py` |
[vllm#61183](vllm-project/vllm#61183) relocated
the patched module;
[vllm#61455](vllm-project/vllm#61455) removed
`_annotate_eagle_groups` from main | Patch
`cache_patch._kv_cache_planning`; new UT asserts the packed hook
tolerates an upstream module without the EAGLE-annotation API |
| `tests/ut/patch/platform/test_patch_eplb.py` |
[vllm#57549](vllm-project/vllm#57549) emits the
async EPLB cycle-committed log from the async worker wrapper on main |
Expect the workspace-move log only on v0.31.0 |
| `tests/ut/patch/platform/test_patch_kv_cache_coordinator.py` |
[vllm#61183](vllm-project/vllm#61183) relocated
the patch entry point | Assert against `kcu_patch._kv_cache_planning`
instead of `vllm.v1.core.kv_cache_utils` |
| `tests/ut/patch/platform/test_patch_pp_mtp.py` |
[vllm#56322](vllm-project/vllm#56322) makes
`ModelConfig.verify_with_parallel_config` read
`parallel_config.enable_hc_sp` | Add `enable_hc_sp=False` to the
constructed parallel config (ignored on v0.31.0) |
| `tests/ut/patch/platform/test_prefix_cache_cp_patches.py` |
[vllm#60043](vllm-project/vllm#60043) removed
the `mamba_cache_mode="all"` leftover and requires `"align"` with prefix
caching; [vllm#53558](vllm-project/vllm#53558)
added the `builder` argument | Version-select the mamba cache mode;
probe the upstream admission signature for the `builder` argument |
| `tests/ut/spec_decode/test_mla_dspark.py` |
[vllm#60692](vllm-project/vllm#60692) renamed
the capture hook `precompute_context_kv(num_reqs)` ->
`prepare_forward(num_reqs, attn_metadata)` | Version-split the expected
capture args/kwargs |
| `tests/ut/test_compressed_prefix_cache.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`is_kv_cache_spec_uniform` | Version-aware import |
| `tests/ut/worker/test_attn_utils_v2.py` |
[vllm#61455](vllm-project/vllm#61455) annotates
EAGLE groups from `spec.is_draft`;
[vllm#60766](vllm-project/vllm#60766) resolves
pin memory lazily via `is_pin_memory_available()` | New UT that
draft-layer specs get tagged; on main monkeypatch
`is_pin_memory_available` to `False` |
| `tests/ut/worker/test_model_runner_v2.py` |
[vllm#56716](vllm-project/vllm#56716) suppresses
EPLB even for `skip_eplb` dummy runs;
[vllm#60946](vllm-project/vllm#60946) renamed
the PCP staging hook | Add a `runner.eplb` stub; call the
version-appropriate staging method |
| `tests/ut/worker/test_mtp_pcp_speculator_v2.py` |
[vllm#60885](vllm-project/vllm#60885) rejects a
mismatched explicit float16/bfloat16 KV-cache dtype | Build the model
config with `bfloat16` instead of `float32` |
| `tests/ut/worker/test_pcp_manager_v2.py` |
[vllm#60946](vllm-project/vllm#60946) renamed
`prepare_inputs_to_capture` -> `prepare_inputs_for_capture_or_profile` |
Patch and call the live method name per lane |
| `tests/ut/worker/test_v2_dummy_padding.py` |
[vllm#61767](vllm-project/vllm#61767) collects
DP votes through `get_dp_group().cpu_all_reduce_()` instead of
`dist.all_reduce(..., group=cpu_group)` | Version-split the mocked
collective |
| `tests/ut/worker/v2/test_eplb_controller.py` |
[vllm#57549](vllm-project/vllm#57549) gives
`add_model` an optional `model_name` and names the draft entry | Expect
the 3-arg call (`model_name="draft (draft)"`) on main, 2-arg on v0.31.0
|
| `vllm_ascend/_310p/worker_310p.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`get_kv_cache_groups` | Version-aware import |
| `vllm_ascend/distributed/ec_transfer/ec_memcache_connector.py` |
[vllm#57904](vllm-project/vllm#57904) dropped
`_get_encoder_cache_hidden_dim` from `cpu.common` (encoder widths are
measured at startup) | On main size the receive buffer as
`out_hidden_size * (1 + len(deepstack_visual_indexes))` (Qwen3-VL
DeepStack) else `get_inputs_embeds_size()`; keep the helper on v0.31.0 |
| `vllm_ascend/models/deepseek_v4/compressor.py` |
[vllm#61392](vllm-project/vllm#61392) switched
`DeepseekV4Config` to the upstream Transformers class | Import from
`transformers` on main, from
`vllm.transformers_utils.configs.deepseek_v4` on v0.31.0 |
| `vllm_ascend/models/deepseek_v4/indexer.py` | Same
[vllm#61392](vllm-project/vllm#61392) | Same
version-aware import |
| `vllm_ascend/models/deepseek_v4/mm_preprocess.py` | Same
[vllm#61392](vllm-project/vllm#61392) | Same
version-aware import |
| `vllm_ascend/models/deepseek_v4/model.py` |
[vllm#61392](vllm-project/vllm#61392) moves the
config to Transformers, folds `compress_ratios`/`num_hash_layers` into
`layer_types`/`mlp_layer_types`, and keys YaRN under the compressor
(`compress`) group | Restore the legacy fields on main from the new
lists, read the `compress`-scoped rope params with safe defaults, and
keep the legacy path on v0.31.0 |
| `vllm_ascend/models/deepseek_v41/cache_config.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`may_override_num_blocks` | Version-aware import |
| `vllm_ascend/models/glm5next/cache_config.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`create_kv_cache_group_specs` / `may_override_num_blocks` |
Version-aware import |
| `vllm_ascend/ops/gdn_attn_builder.py` |
[vllm#58737](vllm-project/vllm#58737) derives
`num_reqs` from the block table and asserts row parity with `seq_lens` |
On main trim the padded block table to `seq_lens.shape[0]` before
`mamba_get_block_table_tensor` when the sizes differ |
| `vllm_ascend/patch/__init__.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`get_kv_cache_config_from_groups` to `kv_cache_planning` | Update the
patch-registry doc |
| `vllm_ascend/patch/platform/__init__.py` |
[vllm#61183](vllm-project/vllm#61183) moved the
KV-cache planning/patcher module | Update the patch-registry doc |
| `vllm_ascend/patch/platform/patch_eplb.py` |
[vllm#57549](vllm-project/vllm#57549) moves the
async EPLB cycle-committed log onto the async worker path on main |
Report completion from the wrapped async worker (rank 0) on main; keep
the workspace-move log on v0.31.0 |
| `vllm_ascend/patch/platform/patch_kv_cache_utils.py` |
[vllm#61183](vllm-project/vllm#61183) moved the
planning helpers;
[vllm#53558](vllm-project/vllm#53558) added the
`builder` argument;
[vllm#61455](vllm-project/vllm#61455) dropped
`_annotate_eagle_groups` | Resolve the live planning module by version,
patch its symbols, gate the EAGLE annotation on attribute presence, and
dispatch the admission helper by argument count |
| `vllm_ascend/quantization/methods/kv_cache/turboquant/cache.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`may_override_num_blocks` | Version-aware import |
| `vllm_ascend/spec_decode/__init__.py` |
[vllm#61701](vllm-project/vllm#61701) removed
Medusa/MLP speculator support | Import `AscendMedusaProposer` only on
v0.31.0; raise a clear error for `method="medusa"` on main |
| `vllm_ascend/spec_decode/medusa_proposer.py` |
[vllm#61701](vllm-project/vllm#61701) removed
`vllm.v1.spec_decode.medusa` | Add `# type: ignore[import-not-found]`;
the module only loads on the lane that still ships it |
| `vllm_ascend/worker/v1/model_runner.py` | Same
[vllm#61701](vllm-project/vllm#61701) | Gate the
Medusa import and the `isinstance(self.drafter, AscendMedusaProposer)`
branch on v0.31.0 |
| `vllm_ascend/worker/v2/attn_utils.py` |
[vllm#61455](vllm-project/vllm#61455) annotates
EAGLE groups from `spec.is_draft` | On main tag the spec with `is_draft`
from `attn_module.is_draft_layer` |
| `vllm_ascend/worker/v2/pcp_manager.py` |
[vllm#60946](vllm-project/vllm#60946) added the
`moe_gathers_tokens` argument and renamed `prepare_inputs_to_capture` ->
`prepare_inputs_for_capture_or_profile` | Forward the new argument;
version-split the staging hook |
| `vllm_ascend/worker/v2/sample/apply_top_k_top_p.py` |
[vllm#61996](vllm-project/vllm#61996) added
`has_topp_only_rows` to the split top-p pipeline | Accept and ignore the
extra argument on main (the PyTorch fallback handles top-p-only rows) |
| `vllm_ascend/worker/v2/spec_decode/dflash/aclgraph.py` |
[vllm#60692](vllm-project/vllm#60692) renamed
`precompute_context_kv` -> `prepare_forward` | Pass the capture hook the
installed vLLM expects |
| `vllm_ascend/worker/v2/spec_decode/lmhead_tp_utils.py` |
[vllm#59554](vllm-project/vllm#59554) added
`num_steps` to `Speculator.sample_draft` | Version-split the override
signature and forward `num_steps` on main |
| `vllm_ascend/worker/v2/spec_decode/rejection_sampler_utils.py` |
[vllm#61451](vllm-project/vllm#61451) renamed
the logits-stats kernel argument `num_speculative_steps` -> `num_logits`
and moved bonus-token detection into `_is_bonus_logit` | Pass
`num_logits` on main; keep `num_speculative_steps` on v0.31.0 |
| `vllm_ascend/worker/worker.py` |
[vllm#61183](vllm-project/vllm#61183) moved
`get_kv_cache_groups`/`unify_hybrid_kv_cache_specs`;
[vllm#57516](vllm-project/vllm#57516) reclaims
cached memory via `torch.accelerator.empty_cache()` inside `load_model`
| Version-aware imports; on main neutralize `torch.npu.empty_cache`
while loading weights under the sleep-mode pluggable memory pool |
### Does this PR introduce _any_ user-facing change?
No. This is an internal main2main compatibility update; on the `v0.31.0`
release lane
behavior is unchanged, and on main the only behavioral note is that
Medusa
speculative decoding now raises a clear error (upstream removed it in
vllm#61701).
### How was this patch tested?
- Unit tests: the changed UT files above (KV-cache planning/coordinator,
EPLB,
PCP/spec-decode, DeepSeek-V4/V4.1 cache, mamba align,
`worker/worker.py`).
- E2E: `tests/e2e/pull_request/four_card/test_minimax_m3.py`.
- Run the main lane against the pinned `c0b6f74` and the `v0.31.0` lane
via
`vllm_version_is`.
---
### Checklist
#### Testing
- [ ] **Boundary / edge-case coverage**: I reviewed whether this change
involves
boundary conditions (e.g. empty input, single/zero element, min/max
values,
off-by-one, sequence-length limits, integer overflow). Where applicable,
I
added unit tests covering these edge cases in `tests/ut/` or
`tests/e2e/`.
- [ ] **Regression test**: If this PR fixes a bug, I added a regression
test
that would fail without the fix.
- [ ] **Error / failure path coverage**: I covered abnormal paths
(exceptions,
timeouts, out-of-range, division by zero) in addition to the happy path.
- [ ] **Meaningful & reliable tests**: New tests assert real behavior
and are
not written merely to satisfy CI.
- [ ] **Performance impact**: If this PR touches a performance-sensitive
path,
I attached benchmark results and added performance regression tests.
- vLLM main:
vllm-project/vllm@a98247a
Signed-off-by: zhangxinyuehfad <starmoon_zhang@163.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Split from #60823: deletes the vendored
DeepseekV4Configand ports the DeepSeek-V4 code to the upstream Transformers 5.16.1 schema. Also updates thetransformers-upgradeskill.Related PRs:
compress_ratios(attention.py). This PR covers every reader, so it supersedes [Bugfix][DeepseekV4] Harden compress_ratio fallback for transformers >=4.57 #43443.Claims
vllm/transformers_utils/configs/deepseek_v4.py. Upstream foldscompress_ratiosandnum_hash_layersintolayer_types/compress_rates/mlp_layer_types:mla_attention, the arch convertor, the spec-decode proposer and the warmup kernels) usesget_compress_ratios. That helper returns V4.1's legacy field, or derives it from the upstream fields with the MTP layers appended. V4.1 can't move to the upstream schema: it uses compress ratios 1 and 2, which have no upstream layer type.mm_prefix_span_leading_pad_modulusmoves from the config ontoDeepseekV4ForConditionalGeneration, where the model runner already looks for it. The sparse-SWA image-span widening is now keyed on an upstreamDeepseekV4Configwithvision_n_layers > 0, matching what the vendored config set.is_rope_parameters_nestedaccepts V4's{"main", "compress"}rope layout.main, and GSM8k, GPQA Diamond and MTP acceptance match.transformers-upgradeskill:vllm/models/must run GSM8k and GPQA Diamond againstmain, and these evals should disable FlashInfer autotuning.text_config.Validation
deepseek-ai/DeepSeek-V4-Flashon 8x B200,main(bc3ddb198a) on GPUs 0-3 vs this branch on GPUs 4-7, TP4 each,transformers5.19.0.mainFlashInfer re-tunes its FP4 MoE tactics on every server start, even when given a cache. With autotuning on, the two servers picked different tactics and GSM8k drifted by about 1 point (0.947-0.951 vs 0.939-0.941). With
--kernel-config '{"enable_flashinfer_autotune": false}'greedy outputs match exactly, so the drift is tactic selection, not this change.Config dumps for Flash, Flash-Vision-Exp and Pro, using the skill's scripts:
get_compress_ratios, the per-layer upstream reads, the hash-layer count and the C4/C128 kernel flags all reproduce each checkpoint's legacy values exactly; the mm-prefix clamp is on only for Vision-Exp, as before; and the DeepSeek-V4 RoPEcos_sin_cacheis bit-identical for sliding, CSA and HCA layers. Vision-Exp and Pro were not served (not enough disk on the host); Vision-Exp'sconfig.jsonsets everyvision_*field the processor reads.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
🤖 Generated with Claude Code