[Perf] fuse more rmsnorm and all-reduce in qwen3.5 - #46998
Merged
Conversation
ZJY0516
requested review from
sighingnow,
tdoublep and
vadiklyutiy
as code owners
June 29, 2026 05:38
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
ZJY0516
changed the title
Member
|
@ZJY0516 could you pls provide the description? |
Member
|
Is there some E2E improvement? |
Member
Author
yes |
RhizoNymph
added a commit
to RhizoNymph/vllm
that referenced
this pull request
Jul 12, 2026
* fix(capture): post_block captures the true block output (residual + mlp), not bare residual * fix(capture): apply post-FFN norm before post_block hook in gemma2/gemma3 (capture true block output) * fix(capture): global filesystem hook validator rejected post_block (stale post_mlp) * docs(capture): fsync is a no-op on Linux NFS exports (sync+async A/B); file count is the lever, COMMIT-honoring NAS is the only fsync-cost regime * docs(capture): fsync cost is storage-dependent (3 regimes incl. COMMIT-honoring NAS); credit fd-cache mechanism; file count is the universal lever * fix(steering): route post_block steering through the shared 12-arg op helper * fix(steering): size gemma3 steering table for the dynamic-override pool * test(steering): update stale post_mlp hook references to post_block * update readme * feat(capture): wire mlp_in/mlp_out hooks for transcoder training data * feat(steering): wire steering/capture hooks into gpt_oss and dbrx * fix(qwen3_next/qwen3_5): repair gdn import after mamba.gdn package refactor * fix(capture): keep post_block capture op live so cudagraph writes the buffer * feat(capture): port sync-execution capture consumers to the v2 runner * fix(capture): v2 sync view uses real scheduled-token count and skips dummy runs * feat: activation patching — data plane, injection plane, source store * feat: activation patching config, request spec, admission, and resolution * feat: PatchStudy client library for activation-patching sweeps * fix(capture): skip v2 sync consumers during kernel warmup forwards * test: offline GPU validation harness for activation patching * fix(capture): keep post_block capture op live so cudagraph writes the buffer * fix: PatchStudy uses capture_wait so clean sources are durable before patching * feat: scheduler per-site patch backpressure + admission source-existence check * test: add TP/PP args to patch validation harness * feat: server-side /v1/patch_sweep endpoint (one-call grid sweeps) * test: live validation of /v1/patch_sweep vs per-cell path * feat(steering): per-conversation latched dynamic steering * test(capture): set total_num_scheduled_tokens in v2 sync-view test stubs * feat(api): forward conversation_id from OpenAI chat/completion requests to SamplingParams * fix(steering): add declared_graphsafe_keys to sync example consumers (registry config build) * feat(steering): sync-consumer contract ABC + latching steering controller base * feat: move conversation_id off SamplingParams into RequestMetadata channel * feat(steering): per-row monitor probe table in apply_steering kernel + op * feat(steering): per-row monitor control plane (manager, action, runner, config) * test(steering): per-row monitor manager + op tests * docs(steering): document per-row (per-request) monitor * feat(steering): wire enable_row_monitor through EngineArgs * fix(steering): e2e_stub consumers inherit SyncCaptureConsumer; add per-row monitor mode * test(steering): per-row monitor serve e2e (gate on/off isolates per-request probe) * feat(steering): declarative gate schema + request-metadata plumbing * feat(steering): surface declarative gates on StepRequestView (v1 + v2) * feat(steering): substrate for declarative gates (compose-on-top, precedence, row cleanup) * feat(steering): built-in declarative per-request steering consumer + auto-register * feat(steering): named probe/steer vector registry + admin endpoints * test(steering): declarative gates schema/consumer/registry/cleanup + EngineArgs wiring * docs(steering): declarative per-request gates (design §8.2 + overview) * fix(steering): bridge latched conversations for later gateless turns * feat(patch): wire activation patching into the v1 model runner Patch was only wired into the v2 runner, so any model not on the v2 allowlist (e.g. gemma3) silently accepted patch specs without applying them. Wire the same control plane into the v1 GPUModelRunner: PatchModelRunnerMixin, _init_patch_state, per-step _update_patch_buffers, and add/finish hooks. Move the runner-agnostic _patch_add_request into the base mixin (shared by both runners). Root fix: set the process-global patch slot count before the v1 model build so register_steering_buffers attaches patch buffers (the v2 runner already did this; v1 did not, so no patchable layers were discovered). GPU-validated on gemma3-4b (v1 runner) and Qwen3-0.6B (both runners), eager + cudagraph: no-op/self-identity bit-exact, cross-run replace reproduces clean, denoising surfaces the clean answer. * docs: activation patching feature doc + interp-infra OVERVIEW index * feat(steering): log sync-capture VRAM footprint + robust declarative probe-site parsing * docs(steering): declarative capture footprint + probe-site notes (§8.2) * fix(patch): patched KV must not poison the prefix cache A patched request re-forwards from its patch floor and registers its computed blocks under vanilla token hashes, so a later unpatched request with the same prompt could be served the patched KV (GPU repro: 0.47 max logprob corruption; only unnoticed because short validation prompts never filled a full block). Fold a deterministic patch-spec hash into the block hashes of all blocks at or after the lowest patched position (attention propagates the patch forward), the same mechanism steering uses. Blocks below the floor stay shareable, preserving the corrupt-prefix sharing that makes sweeps cheap; distinct specs get distinct KV chains. GPU-validated both ways: with the fix an unpatched rerun after a patched run is bit-identical to a fresh-engine ground truth; with the fix disabled it differs by 0.47. * fix(steering): gate vector registry on dev-mode only, not the steering API key * feat(patch): exact answer grading via logprob_token_ids Sweep cells graded the answer/foil by looking them up in the generated top-k logprobs — an answer outside top-k graded as None (top-k boundary flicker), silently dropping cells from the grid. Use the engine's logprob_token_ids to score the answer/foil ids exactly on every request: the sweep endpoint resolves answer_token/foil_token to single token ids via the tokenizer (400 if multi-token), and PatchStudy resolves them via /tokenize, both passing the ids through (logprob_token_ids is now exposed on the completions API). The engine requires logprobs == len(ids) when ids are given. Live-validated: a token far outside top-1 is reported exactly; the full sweep grid grades every cell (0 top-k None-mismatches, 63/63 cells). * fix(patch): lease source runs against eviction + void silently-unpatched cells A source run evicted between admission (manifest check, positively cached) and worker resolution made the patch entry log-and-skip: the request ran UNPATCHED and its sweep cell silently reported the corrupt baseline as a patched result. Two layers of defense: - Leases: the admission path leases referenced runs on the workers (throttled to ~one RPC per run per half-TTL); store eviction skips unexpired-leased runs, soft-exceeding the byte budget with a warning instead of un-patching in-flight requests. Live-validated: a leased run survives capture pressure that would previously have evicted it, and re-sweeps grade 4/4 cells. - Backstop: any residual resolution miss is recorded per-request in a worker registry; the sweep endpoint drains it after each sweep (collective_rpc) and voids the affected cells (grid=None + skipped[] entries) instead of returning unpatched values. * refactor(patch): resolve buffer slots from the VllmConfig context, not a runner-set global The process-global slot count had to be set by each runner before its model build — the v1 runner didn't, which shipped patching as a silent no-op there. Resolve the slot count inside maybe_register_patch_buffers from get_current_vllm_config_or_none() (models are always built under set_current_vllm_config, on every runner), removing the runner-side setup from both runners; the global remains only as a test-context fallback. GPU-checked: buffers register and patching validates on both runners with no runner code. * fix(steering): content-keyed bounded probe tensor cache * fix(steering): bridged overrides preserve compose_admitted * fix(steering): fail-safe declarative gate resolution at admission * feat(patch): clean/corrupt token-position alignment source_position == dest_position silently patches shifted positions when the clean and corrupt prompts tokenize to different lengths — a plausible-looking but wrong heatmap. Add alignment: equal lengths map identity (corresponding positions are the causal-tracing pairing); unequal lengths map the common token prefix by identity and the common suffix by the length delta, and skip the differing middle loudly (skipped[] + alignment summary in the response). The sweep endpoint takes clean_prompt and refuses a length mismatch without it (the source run's captured prompt length is exposed via the admission cache); PatchStudy records the clean prompt on CleanRun and aligns automatically on both the per-cell and server-side paths. Live-validated: mismatch 400s without clean_prompt; an 11-vs-9-token pair aligns (prefix 4, suffix 4, middle skipped) and grades 16/16 aligned cells. * feat(patch): report empirical batch-nondeterminism noise floor per sweep vLLM is not batch-invariant by default, so identical requests in different batch compositions return slightly different logprobs. Rather than forcing batch-invariant mode (a server-wide throughput tax far below causal-tracing signal), each sweep re-runs the corrupt baseline inside the cell batch and reports |delta| vs the solo baseline as noise_floor — grid differences at or below it are not meaningful. Docs point at batch_invariance for exact reproducibility. * fix(steering): declarative probe gates fail closed * fix(steering): port declarative override parity (compose+precedence) to v2 runner * test(steering): update stale fixtures for post-#217/#219 runner state * test(patch): alpha-interpolation GPU check + reject multimodal prompts - gpu_patch_validate gains check F: at the best denoising site, alpha in {0, 0.5, 1} must move the answer logprob monotonically corrupt -> clean (exact grading via logprob_token_ids). Validates the lerp path between its endpoints, which was only CPU-tested. - Chat admission rejects patch specs on multimodal prompts: prompt positions include image placeholder tokens, so patch positions would target placeholder activations — semantically undefined and unvalidated. Documented text-only scope. * docs(patch): sync feature doc with config-context registration + gemma3 TP2/PP2 * feat(patch): --enable-patching implies patch_source capture consumer * feat(patch): one-call sweeps via server-side auto-capture * refactor(patch): promote PatchStudy to vllm package, share alignment, add span positions * chore(steering): latch byte bounds + documented trust model * fix(steering): warmup matches runtime row-monitor specialization; single-source op args * fix(capture): port client_request_id sidecar + streaming metadata refresh to v1 * test(steering): cross-runner conformance harness for the control plane * feat(steering): cross-rank applied-action checksum in dynamic status * chore(steering): typed RowOwner state + refcount-0 purge + dirty-state grouping * feat(patch): compose server-side spans + one-call auto-capture in sweeps * refactor(patch): drop superseded resolve_positions helper * docs: add performance benchmarks section to readme * feat(patch): opt-in SSE streaming for /v1/patch_sweep grids * feat(patch): multi-hook sweeps + source-run lifecycle (auto-drop, DELETE) * fix(patch): stream cells and noise floor in the summary's metric units * feat(steering): worker-registered named vectors + latch-by-reference * test(steering): drop stale second arg from scheduler override hook call * test(patch): streamed multi-hook + auto-drop both-path coverage * test(steering): conformance harness tracks typed RowOwner keys * fix(patch): scheduler backpressure must reserve against usable slots (slot 0 sentinel) * chore(patch): ruff lint + format pass over the feature surface * docs(patch): read-through fixes — usable-slots wording, section levels, streaming units * docs: update dynamic steering benchmarks with latest sweep results * docs: patch-sweep vs TransformerLens benchmark in README performance section * refactor(steering): canonical per-request state + release-at-preemption on both runners * refactor(capture): shared capture/sync-consumer runner mixin * refactor(steering): shared _apply_request_override via req-position accessor * refactor(steering): unified per-step hot path over SteeringBatchView * test(capture): update stale v2 glue fixtures for post-#219 state * refactor(steering): delete gpu steering mixin; one control plane, two accessors * docs(patch): rework example into a coarse-to-fine walkthrough; client all_prompt + noise_floor * fix(patch): reject patch+multimodal before rendering to avoid mm-cache desync * fix(steering): eager/kernel dtype parity, params-buffer guard, kernel-helper dedup, gating invariant docs * chore(patch): mypy fixes for CI parity * docs(patch): multi-rank configs now validated under cudagraph too * fix(steering): JIT cache probe handles Triton >= 3.6 device_caches * test(steering): declarative-gates + preemption-pressure GPU e2e * feat(patch): engine-side offline patch admission in input processor * feat(rust): forward per-request patch spec through OpenAI routes * fix(rust): re-sync EngineCoreOutputs wire format and skip undecodable output frames * feat(patch): spawn Python patch sidecar alongside the Rust frontend * feat(rust): reverse-proxy patch sweep routes to the patch sidecar * docs: document the Rust patch sidecar sweep surface * feat: per-dim patch alpha buffer (alpha*mask folded into one table) * feat: worker resolution of module/zeros/inline patch sources with per-dim masks * feat: client-provided patch vector sources on the wire + admission validation * feat: vector-sourced patch sweeps (source_module/inline/mask) + ablation client * feat(rust): forward patch_vectors table verbatim like patch * docs: patch value sources (module/zeros/inline, masks, ablation sweeps) * test: live validation harness for client-provided patch vector sources * test: pacing-robust streaming parity + mask tolerance in patch vector harness * docs: standalone per-request patching walkthrough * fix: reject wrong-width module/inline patch rows at resolve instead of crashing staging * fix: width-validate steering vectors at registration and admission seams * feat: mount steering routes without dev mode, key-gate module mutations * feat(rust): gate steering module mutations behind the steering API key * fix(capture): register patch_source as a built-in consumer * fix(steering): base-tier vectors apply under Ray executor TP * fix(ray): RayExecutorV2 actor handles survive multi-node session init * fix(ray): RayExecutorV2 actor handles survive multi-node session init (backport to feat/integration) * fix(capture): register patch_source as a built-in consumer (backport to feat/integration) * fix(steering): base-tier vectors apply under Ray executor TP (backport to feat/integration) * DCP supports hybrid attention (vllm-project#40996) Signed-off-by: YanXu <yancey.yx@alibaba-inc.com> Signed-off-by: Jingyi Yang <girasoleyang@gmail.com> Co-authored-by: Jingyi Yang <girasoleyang@gmail.com> * [Core][KV events] Report prefix-cache-reused blocks in full report mode (vllm-project#45261) Signed-off-by: Lei Gong <gonglei25@huawei.com> Co-authored-by: Lei Gong <gonglei25@huawei.com> Co-authored-by: Claude <noreply@anthropic.com> * [Feature][Parser] Support include_reasoning param for non-Harmony models (vllm-project#44301) Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com> * [Perf] fuse more rmsnorm and all-reduce in qwen3.5 (vllm-project#46998) Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> * [XPU] Enable v1/sample tests on XPU CI (vllm-project#44472) Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> * [CI] Right-size test-area timeouts from nightly durations (vllm-project#48186) Signed-off-by: khluu <khluu000@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Add XPU nightly and release image publishing to DockerHub (vllm-project#48126) Signed-off-by: wenjun.liu <wenjun.liu@intel.com> Signed-off-by: jun,du <jun.du@intel.com> Co-authored-by: jun,du <jun.du@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> * [Rust Frontend] Integrate MM video support (vllm-project#47959) Signed-off-by: Bugen Zhao <i@bugenzhao.com> * [Model][CI/Build] Cosmos3: enable registry tests and register Cosmos3-Super (vllm-project#48211) Signed-off-by: Mingfei Guo <1800012773@pku.edu.cn> * [CI] Add TORCH_NIGHTLY=1 build mode (run full suite on torch nightly) (vllm-project#47180) Co-authored-by: Andrey Talman <atalman@users.noreply.github.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com> * Deepstream video backend (vllm-project#42424) Signed-off-by: Viranjan Pagar <vpagar@nvidia.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Roger Wang <hey@rogerw.io> * [Rust Frontend] Add roundtrip fixtures for more chat parsers (vllm-project#47883) Co-authored-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: Bugen Zhao <i@bugenzhao.com> * [Misc] Remove dead code in ViT functionality test (vllm-project#48220) Signed-off-by: Isotr0py <Isotr0py@outlook.com> * [Bugfix][Spec Decode] Fix DFlash draft/target layer-count mismatch (vllm-project#48113) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Model] Migrate MistralLarge3ForCausalLM to AutoWeightsLoader (vllm-project#48153) Signed-off-by: Yuchen Fan <functionhx@gmail.com> * [Refactor] Remove unused rocm kernel `combine_topk_swa_indices_ragged` (vllm-project#48158) Signed-off-by: yewentao256 <zhyanwentao@126.com> * [Bugfix] Fix turboquant FP8 cast failure for BF16 models on Ampere GPUs (vllm-project#39988) Signed-off-by: Xu Zhou <xuzhou9417@163.com> Co-authored-by: Xu Zhou <xuzhou9417@163.com> Co-authored-by: Hoseung Kim <ghyutjik123@gmail.com> * fix: correct load_weights track logic and enable weight integrity for… (vllm-project#41811) Signed-off-by: Yipeng Hu <i26268@metax-tech.com> Signed-off-by: HuYiPeng <144002351+MynameFelix@users.noreply.github.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Yipeng Hu <i26268@metax-tech.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com> * [Build/CI] Build arm64 PR and postmerge image builds for Blackwell SM10x and SM110 (vllm-project#48041) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> * [Model] Add LongCat-Flash-Lite (n-gram embedding) (vllm-project#47857) Signed-off-by: mgoin <mgoin64@gmail.com> * [ROCm] Enable DeepSeek-V4 DSpark speculative decoding on AMD (MI350X / MI355X, gfx950) (vllm-project#47419) Signed-off-by: larryli2-amd <larryli2@amd.com> Signed-off-by: larryli2-amd <Larry.Li@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * patch: auto-size the clean-run source store by default so --enable-patching just works * [Bugfix] Fix FlashMLA dense fp8 metadata crash (num_sm_parts clamp) (vllm-project#48045) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> * handle topk_ids padding in align sum kernel (vllm-project#47785) Signed-off-by: gnovack <novackgm@gmail.com> * [Bugfix][Test] Register Qwen/Qwen3.5-4B example model (vllm-project#48276) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Bugfix] Fix thinking_token_budget not enforced after natural </think> re-entry (vllm-project#45984) Signed-off-by: Ashwin Giridharan <girida@amazon.com> * Add VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS and skip CuTeDSL fp4_gemm autotuning by default (vllm-project#48268) Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * [Quantization] Bound peak memory when repacking FP4 MoE weights for Marlin (vllm-project#47851) Signed-off-by: Joe Rowell <joerowell4@gmail.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: mgoin <mgoin64@gmail.com> * [BugFix] weights processing peak memory reduction for nvfp4 MoE layers (vllm-project#46276) Signed-off-by: Jimmy Lee <hirejimmylee@gmail.com> * [BugFix] Fix packed HND KV cache reshape for FlashAttention (vllm-project#47314) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Misc] Use meta tensor for KV cache stride calculation (vllm-project#47316) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Logs] DP Supervisor Log Improvement (vllm-project#48278) Signed-off-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> Co-authored-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> * perf(capture): metadata-only consumer fast path * perf(capture): metadata fast path skips gather and materialize * perf(capture): cache/batch per-step gather index tensors * [Revert] [Build] Update vllm ...builds FA3 with torch stable API (vllm-project#48269) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> * perf(capture): defer metadata delivery to finalize (accumulate row counts per step) * Bump Transformers version to 5.13.0 (vllm-project#47867) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [XPU]remove is_xxx from moe class and bump up kernels (vllm-project#48079) Signed-off-by: mayuyuace <qiming1.zhang@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> * [CI] Point CI at Transformers release rather than release branch (vllm-project#48328) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * FP32 router GEMV optimization (vllm-project#48335) Signed-off-by: peiyuanz <peiyuanz@inferact.ai> Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: peiyuanz <peiyuanz@inferact.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: zhouzhou <zhouzhou@zhouzhoudeMacBook-Pro.local> * [XPU][UT]Fix InternS1ProForConditionalGeneration AssertionError (vllm-project#48232) Signed-off-by: Lai, Yejing <yejing.lai@intel.com> * [2/N][KV-Cache Layout Refactor] Pack K/V into the content dim across attention backends (vllm-project#44455) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> * fix(entrypoints): stop resolve_items leaking in-flight media fetch tasks on partial failure (vllm-project#48333) Signed-off-by: ErenAta16 <erena6466@gmail.com> * fix(processor): route MiMo-V2-Omni media fetch through MediaConnector (vllm-project#43117) Signed-off-by: Ievgen Bondarenko <ibondarenko@student.sierracollege.edu> Signed-off-by: Ievgen (Jack) Bondarenko <ibondarenko@student.sierracollege.edu> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com> * document graph safe keys for per request capture * fix(capture): keep kernel-warmup requests out of capture tracking * test: adapt fork suites to upstream interfaces, drop removed-arch steering params --------- Signed-off-by: YanXu <yancey.yx@alibaba-inc.com> Signed-off-by: Jingyi Yang <girasoleyang@gmail.com> Signed-off-by: Lei Gong <gonglei25@huawei.com> Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> Signed-off-by: khluu <khluu000@gmail.com> Signed-off-by: wenjun.liu <wenjun.liu@intel.com> Signed-off-by: jun,du <jun.du@intel.com> Signed-off-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: Mingfei Guo <1800012773@pku.edu.cn> Signed-off-by: Viranjan Pagar <vpagar@nvidia.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: Yuchen Fan <functionhx@gmail.com> Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: Xu Zhou <xuzhou9417@163.com> Signed-off-by: Yipeng Hu <i26268@metax-tech.com> Signed-off-by: HuYiPeng <144002351+MynameFelix@users.noreply.github.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: larryli2-amd <larryli2@amd.com> Signed-off-by: larryli2-amd <Larry.Li@amd.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Signed-off-by: gnovack <novackgm@gmail.com> Signed-off-by: Ashwin Giridharan <girida@amazon.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: Joe Rowell <joerowell4@gmail.com> Signed-off-by: Jimmy Lee <hirejimmylee@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: mayuyuace <qiming1.zhang@intel.com> Signed-off-by: peiyuanz <peiyuanz@inferact.ai> Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Signed-off-by: Lai, Yejing <yejing.lai@intel.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Signed-off-by: ErenAta16 <erena6466@gmail.com> Signed-off-by: Ievgen Bondarenko <ibondarenko@student.sierracollege.edu> Signed-off-by: Ievgen (Jack) Bondarenko <ibondarenko@student.sierracollege.edu> Co-authored-by: Yan Xu <yancey.yx@alibaba-inc.com> Co-authored-by: Jingyi Yang <girasoleyang@gmail.com> Co-authored-by: GongLei-HW <1327185943@qq.com> Co-authored-by: Lei Gong <gonglei25@huawei.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: alberto <aperdomo@redhat.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com> Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Co-authored-by: Chaojun Zhang <chaojun.zhang@intel.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com> Co-authored-by: wenjun liu <wenjun.liu@intel.com> Co-authored-by: jun,du <jun.du@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> Co-authored-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: Mingfei Guo <52491257+guoriyue@users.noreply.github.com> Co-authored-by: Andrey Talman <atalman@fb.com> Co-authored-by: Andrey Talman <atalman@users.noreply.github.com> Co-authored-by: ViranjanPagar <vpagar@nvidia.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: FAN YUCHEN <2994114386@qq.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: XuZhou <17717803682@163.com> Co-authored-by: Xu Zhou <xuzhou9417@163.com> Co-authored-by: Hoseung Kim <ghyutjik123@gmail.com> Co-authored-by: HuYiPeng <144002351+MynameFelix@users.noreply.github.com> Co-authored-by: Yipeng Hu <i26268@metax-tech.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: larryli2-amd <Larry.Li@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: gnovack <novackgm@gmail.com> Co-authored-by: Ashwin Giridharan <ashwing@users.noreply.github.com> Co-authored-by: Joe Rowell <joerowell4@gmail.com> Co-authored-by: Jimmy Lee <58957694+thisisjimmyfb@users.noreply.github.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Co-authored-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Qiming Zhang <qiming1.zhang@intel.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: peiyuanz <peiyuanz@inferact.ai> Co-authored-by: zhouzhou <zhouzhou@zhouzhoudeMacBook-Pro.local> Co-authored-by: Yejing Lai <yejing.lai@intel.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: ErenAta16 <erena6466@gmail.com> Co-authored-by: Ievgen Bondarenko <ibondarenko@student.sierracollege.edu>
microslaw
pushed a commit
to microslaw/vllm
that referenced
this pull request
Jul 13, 2026
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
wangxiyuan
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 15, 2026
### What this PR does / why we need it? vllm upstream `e5588e49...85c09e98` (1/1 steps). #### Due to vllm-project/vllm#46998, modified `vllm_ascend/ops/gdn.py`, `vllm_ascend/patch/worker/patch_qwen3_5.py` - Upstream changed QwenGatedDeltaNetAttention.forward, Qwen3NextAttention.forward, and Qwen3NextDecoderLayer.forward to return tensors instead of writing to output buffers. Callers no longer pass output= kwarg. - vLLM Ascend Changes: Added output=None default parameter to AscendGatedDeltaNetAttention.forward and AscendQwen3NextAttention.forward with vllm_version_is("0.23.0") guards. Version-guarded AscendQwen3_5DecoderLayer.forward attention section. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@e5588e4 Signed-off-by: main2main-bot <main2main-bot@users.noreply.github.com> Co-authored-by: main2main-bot <main2main-bot@users.noreply.github.com>
NickLucche
pushed a commit
to NickLucche/vllm
that referenced
this pull request
Jul 15, 2026
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
realliujiaxu
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 15, 2026
### What this PR does / why we need it? This PR is a follow-up to #11875. PR #11875 introduced the vLLM v0.24.0 support. Those adaptation changes are treated as the baseline and are intentionally not repeated in this description. This PR removes the remaining vLLM v0.23.0 compatibility paths from `main`. Most changes only remove `v0.23.0` branches and keep the existing v0.24/main implementation. Those mechanical removals are not listed individually below. #### Release defaults and documentation Update the default `VLLM_TAG` in all maintained Dockerfiles from `v0.23.0` to `v0.24.0`. The main-branch support matrix, slash-command examples, and balance-scheduler design documents are updated accordingly. This prevents source-built images and contributor documentation from continuing to select the unsupported v0.23 release. #### Fused MoE weight layout boundary Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard and 310P unquantized Fused MoE implementations. Upstream vLLM PR vllm-project/vllm#44589 was merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is present in the verified-main revision. Therefore v0.24.0 and verified main do not share the same post-load weight-layout behavior: - v0.24.0 explicitly materializes the transposed weights as contiguous tensors before the NPU layout conversion; - verified main follows the post-PR #44589 path without forcing the same intermediate contiguous layout. The standard and 310P unit tests cover both version-specific layouts, the current MoE runner contract, shared-expert handling, and the 310P-specific communication method. #### Qwen3.5/Qwen3Next output contract Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches from `v0.23.0` to `v0.24.0`. Upstream vLLM PR vllm-project/vllm#46998 was merged as `300e3379`, after the v0.24.0 cut point and before the current verified-main revision. It changed the attention contract from writing into a caller-provided output buffer to returning the output tensor. #### Balance scheduler alignment `BalanceScheduler.schedule()` is a downstream copy of the upstream scheduler body because the balance admission logic cannot be implemented through a small wrapper. The copied body is therefore updated from the v0.23.0 implementation to the v0.24.0 implementation while preserving only the existing balance-scheduling deltas. Both supported upstream references now expose: ```python schedule(self, throttle_prefills: bool = False) ``` The old signature-introspection compatibility code is removed and the disabled path delegates directly to `super().schedule(throttle_prefills)`. The v0.24 scheduler alignment also preserves the corresponding upstream behavior for: - DP prefill throttling; - speculative-token and maximum-length accounting; - hybrid Mamba KV-cache hit handling; - resumed-request bookkeeping; - dynamic speculative decoding; - deferred KV-block freeing; - MRV1-only previous-step request tracking. The balance scheduler unit tests and English/Chinese design documents are updated to use v0.24.0 as the release reference. The drift test continues to verify that the copied scheduler body differs from the pinned upstream release only by the intended balance deltas. #### Deferred removal of owner-maintained patches The following compatibility patches and their unit tests are intentionally retained in this PR: - GLM47 zero-argument tool-call streaming parser; - MiniMax-M2 incremental tool-call parser; - MiniMax usage accounting; - `tool_choice=none` empty-`tool_calls` response cleanup. The first three remain behind the existing `vllm_version_is("0.23.0")` condition, with a TODO explaining that their owners will remove them in a follow-up. The `patch_tool_choice_none_content` registration is also left unchanged for the same ownership reason. These files are not required by the newly supported v0.24/main lanes, but deleting owner-maintained patches is intentionally outside the scope of this compatibility cleanup. #### Test boundaries The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were skipped on main. Since v0.23.0 is removed and the cases remain broken on both v0.24.0 and verified main, they are now explicitly skipped on both supported lanes instead of being unintentionally re-enabled. The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather than v0.23.0. MRV2 remains supported only by the verified-main lane, as established by PR #11875. The HunyuanVL release helper names are updated from `_v023_*` to `_v024_*` because the bundled-processor compatibility path now targets v0.24.0. This is a naming correction only; the HunyuanVL adaptation itself belongs to PR #11875. ### Does this PR introduce _any_ user-facing change? Yes. The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default. There is no additional API change for the supported v0.24.0 and verified-main lanes. ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@85c09e9 --------- Signed-off-by: zhao-stack <2020265299@qq.com> Signed-off-by: shenzhao <shenzhao9@huawei.com> Signed-off-by: MrZ20 <2609716663@qq.com> Co-authored-by: zhao-stack <2020265299@qq.com> Co-authored-by: shenzhao <shenzhao9@huawei.com>
lcskrishna
added a commit
to lcskrishna/vllm
that referenced
this pull request
Jul 15, 2026
* AGENTS MD: Add suggestion on how to incorporate tests (#47784) Signed-off-by: Simon Mo <simon.mo@hey.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com> * [BugFix][LoRA] Refresh punica metadata when LoRA slots are reassigned under an unchanged mapping (#47725) Signed-off-by: AmeenP <ameenp360@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bug] Fix tmp directory for `lm_eval` (#47755) Signed-off-by: yewentao256 <zhyanwentao@126.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [XPU] Fix topk_sigmoid arg mismatch on XPU (#47858) Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com> * [Doc] Fix grammatically incorrect error message in gpu_worker and xpu_worker (#36715) Signed-off-by: Hongbin10 <jdmjdm1998@163.com> * [Perf][3/N] Expand Triton kernel warmup coverage, Qwen (#47546) Signed-off-by: LopezCastroRoberto <rocastro@redhat.com> * Upgrade tpu-inference to v0.24.0 (#47835) Signed-off-by: StingLin <sting.lin@cienet.com> * [Bugfix] Exclude kv_cache_memory_bytes from CacheConfig.compute_hash (#47356) Signed-off-by: Nils Matteson <nils@thaw.sh> Signed-off-by: Nils Matteson <nilsmatteson@icloud.com> Co-authored-by: Nils Matteson <nils@thaw.sh> * fix: include topic frame in KV events replay response (#45177) Signed-off-by: RishabhSaini <rishabhsaini01@gmail.com> * [Bugfix] Fix mamba+dflash for MRV2 (#47698) * [Bugfix] Allow non-contiguous query in FlashInfer FP8 query quantization (#47908) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Doc] Fix manylinux tag in installation guide (#47913) Signed-off-by: Nick Cao <ncao@redhat.com> * [CI Bug Fix] Temp fix for v3.2 accuracy (#47902) * [Bugfix] Pad Mamba page size instead of scaling block_size in unify_kv_cache_spec_page_size (#45207) Signed-off-by: Sahil170595 <147995121+Sahil170595@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * [Doc] Surface the --kv-cache-memory suggestion at INFO and document fast-startup knobs (#47374) Signed-off-by: Nils Matteson <nils@thaw.sh> Signed-off-by: Nils Matteson <nilsmatteson@icloud.com> Co-authored-by: Nils Matteson <nils@thaw.sh> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add tuned selective_state_update config for AMD Instinct MI355 (#47767) Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> * [BugFix] Fix ModelOpt quantization inference for fused siblings (#47445) Signed-off-by: jasonlizhengjian <jasonlizhengjian@gmail.com> * fix(security): bound completion prompt list to prevent unbounded engine fan-out (#47845) Signed-off-by: jperezde <jperezde@redhat.com> * [Bugfix] Patch Hopper MXFP4 OOB scales reads leading to NaN (#47910) Signed-off-by: Yifan Zong <yzong@redhat.com> * [CI/Build] Accept ready-run-all-tests label in pre-commit gate (#47897) Signed-off-by: AmeenP <ameen@primeintellect.ai> Co-authored-by: AmeenP <ameen@primeintellect.ai> Co-authored-by: Claude <noreply@anthropic.com> * [ROCm][CI] Minimize comment in RocmAttention q_scale check (#47731) Signed-off-by: stefankoncarevic <stefan.koncarevic@amd.com> * [ROCm][Bugfix] Key sparse-MLA persistent metadata on per-request context lengths (#47766) Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Core] Persist and reuse the memory-profiling result across boots (opt-in) (#47388) Signed-off-by: Nils Matteson <nils@thaw.sh> Signed-off-by: Nils Matteson <nilsmatteson@icloud.com> Co-authored-by: Nils Matteson <nils@thaw.sh> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [CPU][Bugfix] Fix flaky ShortConv prefill test on ARM (uninitialized weights) (#47848) Signed-off-by: Rahul Vishwakarma <Rahul.Vishwakarma2@ibm.com> * [Frontend] Add endpoint plugins framework (#47454) Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> * [Bugfix] Avoid blocking model launching when no system ffmpeg available for TorchCodec (#47888) Signed-off-by: Isotr0py <Isotr0py@outlook.com> * [Core] Pass request context to CPU offload cache policy touch (#47744) Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> * [Core][Engine] only materialize tokens when thinking budget is in req (#47053) Signed-off-by: walterbm <walter.beller.morales@gmail.com> * [Bugfix][Pooling] Align CrossEncoder token type ids after truncation (#47772) Signed-off-by: Ting Sun <suntcrick@gmail.com> * [Minimax-M3] Using tok_sparse_select from MSA instead of triton kernels (#47502) Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [ROCm][Perf] MXFP8 dense-linear + grouped-MoE GEMM optimizations for MiniMax-M3 (#46117) Signed-off-by: amd-ethany <amd-ethany@users.noreply.github.com> Co-authored-by: amd-ethany <amd-ethany@users.noreply.github.com> * [Bugfix] DSV4 TP16 garbage output (#47493) Signed-off-by: Jeff Ma <jeffjma@umich.edu> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> * [Perf] Minimax M3 - Support cross-layer allreduce-norm fusion (#47631) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> * [Bugfix][Frontend] Fix http_requests_total metric recording some 4xx errors as 5xx (#44303) Signed-off-by: Zach Zhu <zzqshu@126.com> * [Bugfix] Register VLLM_BUILD_* and VLLM_IMAGE_TAG provenance env vars (#45313) Signed-off-by: Nicklas Frahm <nicklas.frahm@gmail.com> * [Bugfix] Avoid leaking Pydantic repr in tool_choice error message (#47028) Signed-off-by: muhammadfawaz1 <135441198+muhammadfawaz1@users.noreply.github.com> Co-authored-by: Mahad Durrani <114791389+mahadrehmann@users.noreply.github.com> * [ROCm] fixed aiter master flag and expert parallelism compatibility on minimax-m3-mxfp8 (#47158) Signed-off-by: Hongxia Yang <hongxia.yang@amd.com> * fix: hash speculative draft model config (#47911) Signed-off-by: Ace Eldeib <aeldeib@coreweave.com> Signed-off-by: Ace Eldeib <alexeldeib@gmail.com> * [Model][HunyuanVL] Use native transformers processor and adapt to transformers 5.13 (#47872) Co-authored-by: manayang <manayang@tencent.com> * [Docs] `kv_sharing_fast_prefill` correction (#47044) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> * Remove unused _get_kv_cache_config_deepseek_v4 alias (#47969) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> * [Test] Skip DeepEP MoE layer tests without P2P access (#47946) Signed-off-by: Tyler Michael Smith <tyler@tylermsmith.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> * [XPU] [Fusion passes] Disable fuse_rope_kvcache_cat_mla & qk_norm_rope_ fusion on XPU (#47962) Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> * [Misc][Docs] Add human-readable integer support for more cli-args (#47608) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> * [Frontend] Support bad_words in the /v1/completions endpoint (#46793) Signed-off-by: sungbin1015 <sbin@solbox.com> Co-authored-by: Claude <noreply@anthropic.com> * [ROCm] Add tuned selective_state_update float16 config for AMD Instinct MI300X (#47945) Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> * [Model] Support MOSS-Transcribe-Diarize (#47729) Signed-off-by: gcanlin <canlinguosdu@gmail.com> * [ROCm] Add tuned selective_state_update float32 config for AMD Instinct MI300X (#47947) Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> * Remove router weight upcast for DSv2-related models (#47970) Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg> * [Bugfix] Allocate HY V3 expert_bias in float32 to prevent silent downcasting (#47797) Signed-off-by: 辰言 <oncwnuIWp30GguOyJ615Fqj8H-yc@git.weixin.qq.com> Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: 辰言 <oncwnuIWp30GguOyJ615Fqj8H-yc@git.weixin.qq.com> Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> * Add tuned selective_state_update float32 config for AMD Instinct MI355 (#47943) Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> * [Rust Frontend] Handle `continue_final_message` with renderer sentinel (#47844) Signed-off-by: Bugen Zhao <i@bugenzhao.com> * [CI/Build][BugFix][The Rock] Fix get_ssm_device_name to return sanitized, usable filename (#47781) Signed-off-by: Randall Smith <Randall.Smith@amd.com> * [Bugfix][V1] Free out-of-window blocks on the processed-token basis under async scheduling (#47728) Signed-off-by: Saddss <28726669061@qq.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Saddss <28726669061@qq.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> * Improvement of Docker image build for IBM Power using prebuilt wheels from IBM published devpi index (#46017) Signed-off-by: vivek sharma <vivsharm@redhat.com> Signed-off-by: puneetsharma21 <puneet.sharma21@ibm.com> Signed-off-by: Puneet Sharma <puneet.sharma21@ibm.com> Co-authored-by: vivek sharma <vivsharm@redhat.com> Co-authored-by: Puneet Sharma <puneet.sharma21@ibm.com> Co-authored-by: depthfirst-app[bot] <184448029+depthfirst-app[bot]@users.noreply.github.com> * [Bug] Fix Batched DeepGEMM (#47884) Signed-off-by: Robert Shaw <robertgshaw2-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com> Co-authored-by: Robert Shaw <robertgshaw2-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com> Co-authored-by: Claude <noreply@anthropic.com> * Fix NVML capability lookup for visible devices (#47892) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> * [Spec Decode] Support hybrid (SWA + full attention) DFlash drafters (#47914) Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix FlashAttention MLA prefill V unpadding (#42642) Signed-off-by: Martin Vit <martin@voipmonitor.org> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> * [Bugfix][DCP] Cast LSE to fp32 in a2a combine to fix bf16 bitcast crash (#47801) Signed-off-by: Shawn Tsai <shawnyht@gmail.com> * [Bugfix] Re-enable benchmarking of librispeech dataset. (#47033) Signed-off-by: Anna Mayne <anna.mayne@arm.com> * [CI] BugFix Eval Small Models Distributed test for DiffusionGemma (#47980) Signed-off-by: Markov Ilya <markovilya19@gmail.com> Co-authored-by: Markov Ilya <markovilya19@gmail.com> * updated flash_attn GIT_TAG to point to torch Stable ABI FA3 commit (#47995) Signed-off-by: Chris Leonard <chleonar@redhat.com> * [Core] Move MRV1 `late_interaction_runner.py` out of MRV2 subtree (#48014) Signed-off-by: Nick Hill <nickhill123@gmail.com> * [Perf] Integrate TRTLLM BF16 MoE Modular Kernel (#45182) Signed-off-by: Kaihang Jiang <kaihangj@login-lyris02.lyris.clusters.nvidia.com> * [ROCm][Bugfix] Fix empty-tensor .max() crash in AITER FA (#47894) Signed-off-by: Djordje Ramic <djoramic@amd.com> * Allow FlashInfer A2A backends for TRTLLM FP8 MoE Modular (#46661) Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg> * [CI] GSM8K eval integration test for KV offloading (#46893) Signed-off-by: Tyler Michael Smith <tyler@tylermsmith.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: Itay Etelis <itay.etelis@ibm.com> Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Signed-off-by: Or Ozeri <oro@il.ibm.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Co-authored-by: Itay Etelis <itay.etelis@ibm.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Feat] Add runtime monitor for post-warmup TileLang compilation (#46718) Signed-off-by: LopezCastroRoberto <rocastro@redhat.com> Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com> * [docs] Fix the docs build (#48008) Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [ROCm] Fix pooling startup workspace lock (#47912) Signed-off-by: Andreas Karatzas <akaratza@amd.com> * Fix embed scaling + CUDA graphs in Transformers modelling backend (#48010) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Bugfix] Use int8 workspace for FlashInfer MLA decode (#48046) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [KVConnector][NIXL] Support pipeline-parallel prefill in push mode (#45880) Signed-off-by: zixi-qi <zixi@inferact.ai> Co-authored-by: Nick Hill <nickhill123@gmail.com> * [ROCm][CI][MoE] Fix double-transpose of fused w3 expert weights (#47874) Signed-off-by: stefankoncarevic <stefan.koncarevic@amd.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix][ROCm] Change AttentionCGSuppoort in TritonMLA to UNIFORM_SINGLE_TOKEN_DECODE (#47144) Signed-off-by: Dino Music <Dino.Music@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [XPU][LoRA] Fix torch.compile DEVICE_LOST by avoiding view-mutation in LoRA shrink (#47944) Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> * [KV Offloading] Add free block iterator for CPU offload scheduling (#47849) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> * [CPU] Fix Qwen-Next SSM type for AMX GDN (#48073) Signed-off-by: jiang1.li <jiang1.li@intel.com> * [Bugfix] Fix Qwen3-ASR transcription streaming postprocessing (#42478) Signed-off-by: JooHo Lee <BWAAEEEK@users.noreply.github.com> Signed-off-by: JooHo Lee <jooho414@gmail.com> Co-authored-by: JooHo Lee <BWAAEEEK@users.noreply.github.com> * Remove TeleChatForCausalLM (#47989) Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Add Intel XPU Docker release pipeline (#47880) Signed-off-by: wenjun.liu <wenjun.liu@intel.com> Signed-off-by: jun,du <jun.du@intel.com> Co-authored-by: jun,du <jun.du@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> * [Bugfix] Fix race condition in KVBlockZeroer (#48085) Signed-off-by: Benjamin Chislett <bchislett@nvidia.com> Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> * Sanitize server file paths from validation error responses (#46415) Signed-off-by: muhammadfawaz1 <135441198+muhammadfawaz1@users.noreply.github.com> Co-authored-by: Mahad Durrani <114791389+mahadrehmann@users.noreply.github.com> * [P/D][Bugfix] Fix PD async KV load lookahead handling for MTP spec decode (#46694) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> * Remove PersimmonForCausalLM and FuyuForCausalLM model architectures (#48096) Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> * Migrate Olmo and Olmo2 to the Transformers modeling backend (#48100) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Bugfix][Model Runner V2] Order uniform decodes first so spec decodes aren't misclassified as prefills (#47381) Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> * [CI] Annotate built Docker image tags on the Buildkite build page (#48101) Signed-off-by: khluu <khluu000@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Pin PyNvVideoCodec to tested 2.0.4 wheel (#48056) * [Bugfix] Guard CUDA-only rms_norm_per_block_quant in FUSED_OPS for non-CUDA builds (#47296) Signed-off-by: Tsvika Shapira <tsvika@moonmath.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [ROCm][CI] Set all timeout_in_minutes to 180 (#48146) Signed-off-by: charlifu <charlifu@amd.com> * [Bugfix] Preserve tensor causal metadata for grouped attention (#48135) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Codex <codex@openai.com> * [KVConnector] MultiConnector: give every sub-connector the request's real blocks in `update_state_after_alloc` (#46865) Signed-off-by: deng451e <838677410@qq.com> * [Bugfix][MRV2] Reset num_accepted_tokens on add_request in all modes (#48132) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [ROCM][DSV32][Perf][MTP] Enable UNIFORM_BATCH CG mode in rocm_aiter_mla_sparse (#45149) Signed-off-by: Teemu Virolainen <teemu.virolainen@amd.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [KV Connector][Mooncake] Apply SWA lookup mask before hashing/key build (#47317) Signed-off-by: Zhewen Li <zhewenli@inferact.ai> Co-authored-by: Zhewen Li <zhewenli@inferact.ai> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude <noreply@anthropic.com> * [ROCm] Synchronize sparse MLA metadata before graph replay (#47404) Signed-off-by: zihaomu <zmu@amd.com> * [CI] Increase extract hidden states TP2 timeout (#48161) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> * [ROCm] Revert Part of `[ROCm] Fix pooling startup workspace lock` #47912 (#48154) Signed-off-by: Micah Williamson <micah.williamson@amd.com> * [CI] Fix cargo-deny config flag ordering (#48170) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> * Correct model layer aliasing for Bert style models (#43896) * update marlin M size for EP (#48144) Signed-off-by: gnovack <novackgm@gmail.com> * [ROCm][CI] Move remaining engine/samplers AMD steps to mi325_1 (#48169) Signed-off-by: pei.zhang <pei.zhang@amd.com> Co-authored-by: Claude <noreply@anthropic.com> * [bugfix] bge-m3-sparse-plugin mismatch requests (#48112) Signed-off-by: augusto.yjh <augusto.yjh@antgroup.com> * [CI/Build][AMD] Fix ROCm OOM in eagle_correctness_heavy by reserving CUDA graph memory (#47366) Signed-off-by: pei.zhang <pei.zhang@amd.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [kv_offload] Emit tier-owned BlockStored events from FS/OBJ secondary tiers (#47923) Signed-off-by: Change72 <changg@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> * DCP supports hybrid attention (#40996) Signed-off-by: YanXu <yancey.yx@alibaba-inc.com> Signed-off-by: Jingyi Yang <girasoleyang@gmail.com> Co-authored-by: Jingyi Yang <girasoleyang@gmail.com> * [Core][KV events] Report prefix-cache-reused blocks in full report mode (#45261) Signed-off-by: Lei Gong <gonglei25@huawei.com> Co-authored-by: Lei Gong <gonglei25@huawei.com> Co-authored-by: Claude <noreply@anthropic.com> * [Feature][Parser] Support include_reasoning param for non-Harmony models (#44301) Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com> * [Perf] fuse more rmsnorm and all-reduce in qwen3.5 (#46998) Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> * [XPU] Enable v1/sample tests on XPU CI (#44472) Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> * [CI] Right-size test-area timeouts from nightly durations (#48186) Signed-off-by: khluu <khluu000@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Add XPU nightly and release image publishing to DockerHub (#48126) Signed-off-by: wenjun.liu <wenjun.liu@intel.com> Signed-off-by: jun,du <jun.du@intel.com> Co-authored-by: jun,du <jun.du@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> * [Rust Frontend] Integrate MM video support (#47959) Signed-off-by: Bugen Zhao <i@bugenzhao.com> * [Model][CI/Build] Cosmos3: enable registry tests and register Cosmos3-Super (#48211) Signed-off-by: Mingfei Guo <1800012773@pku.edu.cn> * [CI] Add TORCH_NIGHTLY=1 build mode (run full suite on torch nightly) (#47180) Co-authored-by: Andrey Talman <atalman@users.noreply.github.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com> * Deepstream video backend (#42424) Signed-off-by: Viranjan Pagar <vpagar@nvidia.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Roger Wang <hey@rogerw.io> * [Rust Frontend] Add roundtrip fixtures for more chat parsers (#47883) Co-authored-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: Bugen Zhao <i@bugenzhao.com> * [Misc] Remove dead code in ViT functionality test (#48220) Signed-off-by: Isotr0py <Isotr0py@outlook.com> * [Bugfix][Spec Decode] Fix DFlash draft/target layer-count mismatch (#48113) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Model] Migrate MistralLarge3ForCausalLM to AutoWeightsLoader (#48153) Signed-off-by: Yuchen Fan <functionhx@gmail.com> * [Refactor] Remove unused rocm kernel `combine_topk_swa_indices_ragged` (#48158) Signed-off-by: yewentao256 <zhyanwentao@126.com> * [Bugfix] Fix turboquant FP8 cast failure for BF16 models on Ampere GPUs (#39988) Signed-off-by: Xu Zhou <xuzhou9417@163.com> Co-authored-by: Xu Zhou <xuzhou9417@163.com> Co-authored-by: Hoseung Kim <ghyutjik123@gmail.com> * fix: correct load_weights track logic and enable weight integrity for… (#41811) Signed-off-by: Yipeng Hu <i26268@metax-tech.com> Signed-off-by: HuYiPeng <144002351+MynameFelix@users.noreply.github.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Yipeng Hu <i26268@metax-tech.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com> * [Build/CI] Build arm64 PR and postmerge image builds for Blackwell SM10x and SM110 (#48041) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> * [Model] Add LongCat-Flash-Lite (n-gram embedding) (#47857) Signed-off-by: mgoin <mgoin64@gmail.com> * [ROCm] Enable DeepSeek-V4 DSpark speculative decoding on AMD (MI350X / MI355X, gfx950) (#47419) Signed-off-by: larryli2-amd <larryli2@amd.com> Signed-off-by: larryli2-amd <Larry.Li@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [Bugfix] Fix FlashMLA dense fp8 metadata crash (num_sm_parts clamp) (#48045) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> * handle topk_ids padding in align sum kernel (#47785) Signed-off-by: gnovack <novackgm@gmail.com> * [Bugfix][Test] Register Qwen/Qwen3.5-4B example model (#48276) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Bugfix] Fix thinking_token_budget not enforced after natural </think> re-entry (#45984) Signed-off-by: Ashwin Giridharan <girida@amazon.com> * Add VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS and skip CuTeDSL fp4_gemm autotuning by default (#48268) Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * [Quantization] Bound peak memory when repacking FP4 MoE weights for Marlin (#47851) Signed-off-by: Joe Rowell <joerowell4@gmail.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: mgoin <mgoin64@gmail.com> * [BugFix] weights processing peak memory reduction for nvfp4 MoE layers (#46276) Signed-off-by: Jimmy Lee <hirejimmylee@gmail.com> * [BugFix] Fix packed HND KV cache reshape for FlashAttention (#47314) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Misc] Use meta tensor for KV cache stride calculation (#47316) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Logs] DP Supervisor Log Improvement (#48278) Signed-off-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> Co-authored-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> * [Revert] [Build] Update vllm ...builds FA3 with torch stable API (#48269) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> * Bump Transformers version to 5.13.0 (#47867) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [XPU]remove is_xxx from moe class and bump up kernels (#48079) Signed-off-by: mayuyuace <qiming1.zhang@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> * [CI] Point CI at Transformers release rather than release branch (#48328) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * FP32 router GEMV optimization (#48335) Signed-off-by: peiyuanz <peiyuanz@inferact.ai> Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: peiyuanz <peiyuanz@inferact.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: zhouzhou <zhouzhou@zhouzhoudeMacBook-Pro.local> * [XPU][UT]Fix InternS1ProForConditionalGeneration AssertionError (#48232) Signed-off-by: Lai, Yejing <yejing.lai@intel.com> * [2/N][KV-Cache Layout Refactor] Pack K/V into the content dim across attention backends (#44455) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> * fix(entrypoints): stop resolve_items leaking in-flight media fetch tasks on partial failure (#48333) Signed-off-by: ErenAta16 <erena6466@gmail.com> * fix(processor): route MiMo-V2-Omni media fetch through MediaConnector (#43117) Signed-off-by: Ievgen Bondarenko <ibondarenko@student.sierracollege.edu> Signed-off-by: Ievgen (Jack) Bondarenko <ibondarenko@student.sierracollege.edu> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com> * [CI] split tests to reduce CI time (#48219) Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> * [Bugfix][LoRA] Support ark_linear base layer in _get_lora_device (#47690) Signed-off-by: AlejandroParedesLT <alejandroparedeslatorre@gmail.com> * [CI][CPU] Add Qwen2-VL multimodal tests for CPU backend and fix incompatibilities (#48072) Signed-off-by: Zhenhui Zhao <zhenhui.zhao@intel.com> * [2/N][Core] support partial prefix cache hit for hybrid model (#46384) Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> * Runtime Draft Weight Update for Speculative Decoding (#46725) Signed-off-by: vx120 <893600387@qq.com> Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: crp0128 <191679376@qq.com> Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * [Perf][Qwen] Replace MOE all-reduce with reduce-scatter (#47006) Signed-off-by: gcanlin <canlinguosdu@gmail.com> Signed-off-by: yewentao256 <zhyanwentao@126.com> Co-authored-by: yewentao256 <zhyanwentao@126.com> * [Frontend] Add /abort_requests to the RLHF dev API router (#47173) Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix] Guard mixed-dtype allreduce RMSNorm quant fusions (#48330) Signed-off-by: hcenteno <hugo.centeno@estudiantat.upc.edu> * Make tiering offload region DP-replica aware (#47987) Signed-off-by: Liran Schour <lirans@il.ibm.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> * [CI][2/N] reduce CI time (#48394) Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> * [EC Connector] Add EC Transfer Params (#42433) Signed-off-by: omerpaz95 <omerpaz95@gmail.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> * [CI][Rust Frontend] Pin cargo tool versions (#48222) Signed-off-by: Bugen Zhao <i@bugenzhao.com> * [Bugfix][KV Offloading] Fix stale transfer_jobs after reset_cache + harden job completion (#48102) Signed-off-by: Alex <alex.tech.lab@outlook.com> * [Kernel] Implement CUDA kernel for ReLUSquaredActivation (relu^2) (#39058) Signed-off-by: Tanish Malekar <tanishmalekar32@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * [ROCm][MiniMax-M3] Add AITER sparse paged attention (#47287) Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com> Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Hongxia Yang <hongxia.yang@amd.com> Co-authored-by: Jun Kang Chow <junkangchow@gmail.com> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com> * [Model] Optimize Qwen3.5 on H20 (#48350) Signed-off-by: zzt <zengzetang.zzt@antgroup.com> * [Bugfix][ROCm] Keep TP all_gather on base-class collective (#48446) Signed-off-by: fai <fangzhouai@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Re-disable CUDA graph memory profiling on ROCm (#48440) Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [CPU][Spec Decode] Support DFlash speculative decoding for GDN models on CPU (#46090) Signed-off-by: guybd <guy.boudoukh@intel.com> Signed-off-by: Guy Boudoukh <guy.boudoukh@intel.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com> * [Bugfix][UT]Fix EagleMiniCPMForCausalLM meet TypeError (#48452) Signed-off-by: Lai, Yejing <yejing.lai@intel.com> * [CPU] Create Proper Numa topology for s390x (#40714) Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com> * [BugFix] Restore full tokens for Qwen MTP When MoE SP (#48429) Signed-off-by: gcanlin <canlinguosdu@gmail.com> * [ROCm][CI] Cache Rust builds by source inputs (#46527) Signed-off-by: Andreas Karatzas <akaratza@amd.com> Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com> Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com> * [Distributed][Perf] Enable FlashInfer MNNVL allreduce RMS quant fusion (#48064) Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com> * [Attention] Make sliding-window support an explicit backend capability (#48011) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [Bugfix][KV Cache] Don't route uniform-page-size MLA+SWA models into DeepseekV4 packing (#48256) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: Claude <noreply@anthropic.com> * [Core] Support fp32 lm_head for generation models via head_dtype (RFC #48305 §3.6) (#48390) Signed-off-by: Karthik Kothuri <karthikkothuri2009@gmail.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io> * [CI] Add SPDX license header to Rust/Protobuf sources (#48472) Signed-off-by: Bugen Zhao <i@bugenzhao.com> * [Mypy Fix] Split mypy work (#48490) Signed-off-by: yewentao256 <zhyanwentao@126.com> * lower memory required for capturing cudagraphs for large cudagraph sizes (#48483) Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com> * remove force channels_last in Idefics3MultiModalProcessor (#48467) Signed-off-by: Yan Ma <yan.ma@intel.com> * [XPU][CI]Adjust timeout_in_minutes in Intel GPU CI (#48418) Signed-off-by: zengxian <xiangdong.zeng@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> * [BugFix][ModelRunner V2] Fix stale attn metadata in speculator prefill cudagraph capture (#48261) Signed-off-by: Nick Hill <nickhill123@gmail.com> * [Misc] Improve Matryoshka pooling dimensions validation (#48057) Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [EC Connector] CPU Offloading EC Connector (#47423) Signed-off-by: omerpaz95 <omerpaz95@gmail.com> Signed-off-by: Or Ozeri <oro@il.ibm.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> * [ROCm][CI] Transformers: pass only one of input_ids/inputs_embeds (#48258) Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Doc] Add DeepseekV32ForCausalLM to supported_models.md (#48293) Signed-off-by: Gavin Morris <gmorriscs@gmail.com> * [Bugfix][Frontend] Flush engine reasoning parser at engine-reasoning → tool streaming boundary (#47606) Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com> Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com> * [ROCm][CI] Unblock `AMD: Language Models Test (Extended Pooling)` (#48513) Signed-off-by: Micah Williamson <micah.williamson@amd.com> * Add DCP + Eagle support for Tokenspeed MLA backends (#48180) Signed-off-by: Pavani Majety <pmajety@nvidia.com> Signed-off-by: Jingyi Yang <girasoleyang@gmail.com> Co-authored-by: Jingyi Yang <girasoleyang@gmail.com> * Added sliding window attention support for qwen-eagle3 architecture (#47568) Signed-off-by: shanjiaz <zsjwpianpian@gmail.com> * [Core] Preserve Marconi caching with selective hybrid cache retention (#47782) Signed-off-by: Nick Hill <nickhill123@gmail.com> * [BugFix] Initialize model_config for Qwen3-VL MoE (#44863) Signed-off-by: wenpengw-nv <wenpengw@nvidia.com> Co-authored-by: Roger Wang <hey@rogerw.io> * [Feat] Add Support for BertForMaskedLM to vLLM (#48463) Signed-off-by: atalhens <sneh.lata@nutanix.com> * [Frontend] Expose logprob_token_ids on Python OpenAI endpoints (#43463) Signed-off-by: Lang Zhao <lang.zhao@galileo.ai> Co-authored-by: Claude <noreply@anthropic.com> * [BugFix] Correct OTEL span start time for Dynamo compilation (#40698) Signed-off-by: emricksini-h <emrick.birivoutin@hcompany.ai> Co-authored-by: Simon Mo <simon.mo@hey.com> * add pad-aware swiglu limit kernel (#48287) Signed-off-by: gnovack <novackgm@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Quantization][INC][ARK] Support INT2 XPU WOQ Linear (#47521) Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com> Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> * [1/N] Add dense MHA path for sparse MLA short sequences (#47327) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * up FI fp8 moe topk to 32 (#44462) * [ROCm][MiniMax-M2] Dispatch fused QK-norm + AllReduce via AITER (#44849) Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com> Co-authored-by: Pawel Kowalski <pawel.kowalski@amd.com> * [ROCm][Kernel] Add HybridW4A16LinearKernel: Triton prefill + HIP skinny decode (#40977) Signed-off-by: Matthias Gehre <matthias.gehre@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [Perf][ROCm] Fix GDN KKT warmup regression on RDNA by avoiding fp32 tl.dot (#45000) Signed-off-by: Saeid Rostami <srostami@amd.com> Signed-off-by: nemanjaudovic <nudovic@amd.com> * [ROCm][MiniMax-M3][Spec Decode] Support speculative decode with AITER sparse PA (#47984) Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Hongxia Yang <hongxia.yang@amd.com> Co-authored-by: Jun Kang Chow <junkangchow@gmail.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com> * [Bugfix] Skip minimax_m3 tool parser tests when Rust extension is absent (#48523) Signed-off-by: Matt Woodson <mwoodson@redhat.com> * [Misc] Clean up "swap_space" (#48549) Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> * [ROCm] Retune MI355 selective_state_update float16 config on the unified effective_batch grid (#48372) Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> * [Perf][Feat] Add generic cuteDSL LL BF16 router (GEMM) (#42562) Signed-off-by: LopezCastroRoberto <rocastro@redhat.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com> * fix(security): guard lm-format-enforcer regex compile with timeout (#47595) Signed-off-by: jperezde <jperezde@redhat.com> * Add Cosmos3 Edge Reasoner model (#48291) Signed-off-by: Bartosz Stefaniak <bstefaniak@nvidia.com> Co-authored-by: Bartosz Stefaniak <bstefaniak@nvidia.com> * [Bugfix] Return 400 instead of 500 when multimodal data is sent to a text-only model (#48473) Signed-off-by: Hoang Nguyen Tien <hoang.nguyentien.2601@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * [XPU][CI] Add tests/v1/e2e/general/test_correctness_sliding_window.py in Intel GPU CI (#47231) Signed-off-by: zengxian <xiangdong.zeng@intel.com> Signed-off-by: xiangdong <40376367+zxd1997066@users.noreply.github.com> * [Test] Enable KV cache events for HMA models in CPU offloading test (#47754) Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> * [Misc] Rename VLLM_TRITON_ATTN_USE_TD to VLLM_TRITON_USE_TD (#45781) Signed-off-by: Artur Fierka <artur.fierka@intel.com> Co-authored-by: Claude <noreply@anthropic.com> * [Bugfix][Security] Fix concurrent sparse invariant race bypassing CVE remediation (#48583) Signed-off-by: jperezde <jperezde@redhat.com> * [Model] Enable LoRA support for tower and connector in LlavaNextVideo (#48594) Signed-off-by: gangula-karthik <gkarthik923@gmail.com> * [NIXL] Avoid reading expired blocks in bidirectional turn-2 read (#47021) Signed-off-by: Tomer Gilad <tgilad@nvidia.com> Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai> * [Bugfix] Include inline per-token-head scales in offloaded page transfer width (#48411) Signed-off-by: Itay Etelis <itay.etelis@ibm.com> Signed-off-by: Itay Etelis <Itay.etelis@gmail.com> Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Co-authored-by: Itay Etelis <itay.etelis@ibm.com> Co-authored-by: Itay Etelis <Itay.etelis@gmail.com> * [Bugfix] Gemma4 parser: classify channel-less output consistently in streaming and non-streaming (#48262) Signed-off-by: Adhithya Balakrishnan <adhithya.b2004@gmail.com> Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com> * [CI Bug] Fully solve accuracy issue for DSv3.2 + MTP + Sequence Parallel (#48036) Signed-off-by: yewentao256 <zhyanwentao@126.com> * [Bugfix] Make MLA+SWA check the layer's backend, not the model config (#48520) Signed-off-by: mgoin <mgoin64@gmail.com> * [Doc] Sync four function docstrings with their signatures (#45437) Signed-off-by: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [KV Offload] Split cpu_cache_usage_perc into write/read usage gauges (#47666) Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com> Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> * [ROCm] Retune MI355 selective_state_update float32 config on the unified effective_batch grid (#48373) Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> * [Reasoning] Optimize TPOT for thinking budget when used with speculative decoding (#46662) Signed-off-by: rishitdholakia13 <rishit+github@cohere.com> Co-authored-by: Cursor <cursoragent@cursor.com> * [CI] Build macOS arm64 CPU wheel natively on the macmini queue (#48289) Signed-off-by: mgoin <mgoin64@gmail.com> * [ROCm][CI] fix flashinfer import check (#48647) Signed-off-by: Divakar Verma <divakar.verma@amd.com> * [CI][Bugfix] Fix FlashAttention reported MLA dimension support (#48631) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> * Log fully resolved pooling config at startup (#48030) Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix] MoRIIO toy P/D proxy: add /health (#45222) Signed-off-by: Chaemin Lim <chaemin.lim@mangoboost.io> Signed-off-by: Edwin Lim <edwinlim0919@gmail.com> Co-authored-by: Edwin Lim <edwin.lim@mangoboost.io> Co-authored-by: Jaeyoun Kim <jaeyoun.kim@mangoboost.io> Co-authored-by: Edwin Lim <edwinlim0919@gmail.com> * [Bugfix][CI] Fix test_head_dtype quant_method test on ROCm (#48654) Signed-off-by: Micah Williamson <micah.williamson@amd.com> * fix: size FlashInfer prefill workspace to batch head footprint (#48428) Signed-off-by: Joe Rowell <joerowell4@gmail.com> * [Bugfix][R3] Exclude draft routers from expert capture (#48622) Signed-off-by: aoshen02 <aoshen@inferact.ai> * [Bugfix] Preserve unloaded non-persistent buffers during layerwise reload (#44371) Signed-off-by: Joan Velja <joan.velja22@gmail.com> Co-authored-by: Dakai An <77474977+andakai@users.noreply.github.com> * [Perf] Remove redundant repeat and copy for dsv4, 1.8% E2E TPOT improvement. (#48137) Signed-off-by: yewentao256 <zhyanwentao@126.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * add pad-aware reduce path (#48385) Signed-off-by: gnovack <novackgm@gmail.com> * [ROCm][CI] Remove mxfp4 test skips after `amd-quark` 0.12 release (#47330) Signed-off-by: Micah Williamson <micah.williamson@amd.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com> Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com> Co-authored-by: fxmarty-amd <felmarty@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [Core][LoRA] Support fp32 lm_head (head_dtype) on the LoRA path (#48525) Signed-off-by: Karthik Kothuri <karthikkothuri2009@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> * [CI][AMD] Configure MI300 tests for native execution without DinD (#48387) Signed-off-by: Andreas Karatzas <akaratza@amd.com> * [Quant] Enable humming w[2-7]a[4,8] inference with compressed-tensors (#46390) Signed-off-by: HDCharles <charlesdavidhernandez@gmail.com> * [LoRA][1/N] Integrate flashinfer MoE LoRA for BF16 model (#48632) Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> * [Security] Replace diskcache to eliminate pickle deserialization (#44549) Signed-off-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> * Build with ABI stable FlashMLA (#48174) Signed-off-by: Jane Xu <janeyx@meta.com> Signed-off-by: Shengqi Chen <i@harrychen.xyz> Co-authored-by: Shengqi Chen <i@harrychen.xyz> * [Bugfix] Set kv_quant_mode on the generic MLA KV-cache spec (#48379) Signed-off-by: Mikhail Kostryukov <mike@triptrack.net> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> * [Bugfix][Spec Decode] Support heterogeneous QK fusion geometry (#48671) Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: OpenAI Codex <codex@openai.com> * [ROCm] Run init test engine in-process to avoid KV-cache OOM (#48527) Signed-off-by: Djordje Ramic <djoramic@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [ROCm][CI] fix test_common.py (#48676) Signed-off-by: charlifu <charlifu@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [Rust Frontend] Integrate MM audio support (#48554) Co-authored-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> * [CI/Build] Split release artifact annotations by type (#48600) Signed-off-by: khluu <khluu000@gmail.com> Co-authored-by: OpenAI Codex <noreply@openai.com> * [ROCm] Add tuned selective_state_update config for AMD MI350 (#48159) Signed-off-by: Giuseppe Grossi <ggrossi@amd.com> * fix flaky multi example connector consistency (#48206) Signed-off-by: aarushjain29 <aarushi.jain2@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> * [ROCm][Bugfix] Enable the fp32 head_dtype torch.mm fast path on ROCm (#48688) Signed-off-by: Turner Jabbour <doubleujabbour@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> --------- Signed-off-by: Simon Mo <simon.mo@hey.com> Signed-off-by: AmeenP <ameenp360@gmail.com> Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com> Signed-off-by: Hongbin10 <jdmjdm1998@163.com> Signed-off-by: LopezCastroRoberto <rocastro@redhat.com> Signed-off-by: StingLin <sting.lin@cienet.com> Signed-off-by: Nils Matteson <nils@thaw.sh> Signed-off-by: Nils Matteson <nilsmatteson@icloud.com> Signed-off-by: RishabhSaini <rishabhsaini01@gmail.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Signed-off-by: Nick Cao <ncao@redhat.com> Signed-off-by: Sahil170595 <147995121+Sahil170595@users.noreply.github.com> Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Signed-off-by: jasonlizhengjian <jasonlizhengjian@gmail.com> Signed-off-by: jperezde <jperezde@redhat.com> Signed-off-by: Yifan Zong <yzong@redhat.com> Signed-off-by: AmeenP <ameen@primeintellect.ai> Signed-off-by: stefankoncarevic <stefan.koncarevic@amd.com> Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Signed-off-by: Rahul Vishwakarma <Rahul.Vishwakarma2@ibm.com> Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com> Signed-off-by: walterbm <walter.beller.morales@gmail.com> Signed-off-by: Ting Sun <suntcrick@gmail.com> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Signed-off-by: amd-ethany <amd-ethany@users.noreply.github.com> Signed-off-by: Jeff Ma <jeffjma@umich.edu> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Signed-off-by: Zach Zhu <zzqshu@126.com> Signed-off-by: Nicklas Frahm <nicklas.frahm@gmail.com> Signed-off-by: muhammadfawaz1 <135441198+muhammadfawaz1@users.noreply.github.com> Signed-off-by: Hongxia Yang <hongxia.yang@amd.com> Signed-off-by: Ace Eldeib <aeldeib@coreweave.com> Signed-off-by: Ace Eldeib <alexeldeib@gmail.com> Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Signed-off-by: Tyler Michael Smith <tyler@tylermsmith.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> Signed-off-by: sungbin1015 <sbin@solbox.com> Signed-off-by: gcanlin <canlinguosdu@gmail.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg> Signed-off-by: 辰言 <oncwnuIWp30GguOyJ615Fqj8H-yc@git.weixin.qq.com> Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Signed-off-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: Saddss <28726669061@qq.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Signed-off-by: vivek sharma <vivsharm@redhat.com> Signed-off-by: puneetsharma21 <puneet.sharma21@ibm.com> Signed-off-by: Puneet Sharma <puneet.sharma21@ibm.com> Signed-off-by: Robert Shaw <robertgshaw2-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Martin Vit <martin@voipmonitor.org> Signed-off-by: Shawn Tsai <shawnyht@gmail.com> Signed-off-by: Anna Mayne <anna.mayne@arm.com> Signed-off-by: Markov Ilya <markovilya19@gmail.com> Signed-off-by: Chris Leonard <chleonar@redhat.com> Signed-off-by: Kaihang Jiang <kaihangj@login-lyris02.lyris.clusters.nvidia.com> Signed-off-by: Djordje Ramic <djoramic@amd.com> Signed-off-by: Itay Etelis <itay.etelis@ibm.com> Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Signed-off-by: Or Ozeri <oro@il.ibm.com> Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: zixi-qi <zixi@inferact.ai> Signed-off-by: Dino Music <Dino.Music@amd.com> Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> Signed-off-by: jiang1.li <jiang1.li@intel.com> Signed-off-by: JooHo Lee <BWAAEEEK@users.noreply.github.com> Signed-off-by: JooHo Lee <jooho414@gmail.com> Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com> Signed-off-by: wenjun.liu <wenjun.liu@intel.com> Signed-off-by: jun,du <jun.du@intel.com> Signed-off-by: Benjamin Chislett <bchislett@nvidia.com> Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com> Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Signed-off-by: khluu <khluu000@gmail.com> Signed-off-by: Tsvika Shapira <tsvika@moonmath.ai> Signed-off-by: charlifu <charlifu@amd.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: deng451e <838677410@qq.com> Signed-off-by: Teemu Virolainen <teemu.virolainen@amd.com> Signed-off-by: Zhewen Li <zhewenli@inferact.ai> Signed-off-by: zihaomu <zmu@amd.com> Signed-off-by: Micah Williamson <micah.williamson@amd.com> Signed-off-by: gnovack <novackgm@gmail.com> Signed-off-by: pei.zhang <pei.zhang@amd.com> Signed-off-by: augusto.yjh <augusto.yjh@antgroup.com> Signed-off-by: Change72 <changg@nvidia.com> Signed-off-by: YanXu <yancey.yx@alibaba-inc.com> Signed-off-by: Jingyi Yang <girasoleyang@gmail.com> Signed-off-by: Lei Gong <gonglei25@huawei.com> Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Mingfei Guo <1800012773@pku.edu.cn> Signed-off-by: Viranjan Pagar <vpagar@nvidia.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: Yuchen Fan <functionhx@gmail.com> Signed-off-by: Xu Zhou <xuzhou9417@163.com> Signed-off-by: Yipeng Hu <i26268@metax-tech.com> Signed-off-by: HuYiPeng <144002351+MynameFelix@users.noreply.github.com> Signed-off-by: larryli2-amd <larryli2@amd.com> Signed-off-by: larryli2-amd <Larry.Li@amd.com> Signed-off-by: Ashwin Giridharan <girida@amazon.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: Joe Rowell <joerowell4@gmail.com> Signed-off-by: Jimmy Lee <hirejimmylee@gmail.com> Signed-off-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> Signed-off-by: mayuyuace <qiming1.zhang@intel.com> Signed-off-by: peiyuanz <peiyuanz@inferact.ai> Signed-off-by: Lai, Yejing <yejing.lai@intel.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Signed-off-by: ErenAta16 <erena6466@gmail.com> Signed-off-by: Ievgen Bondarenko <ibondarenko@student.sierracollege.edu> Signed-off-by: Ievgen (Jack) Bondarenko <ibondarenko@student.sierracollege.edu> Signed-off-by: AlejandroParedesLT <alejandroparedeslatorre@gmail.com> Signed-off-by: Zhenhui Zhao <zhenhui.zhao@intel.com> Signed-off-by: vx120 <893600387@qq.com> Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: hcenteno <hugo.centeno@estudiantat.upc.edu> Signed-off-by: Liran Schour <lirans@il.ibm.com> Signed-off-by: omerpaz95 <omerpaz95@gmail.com> Signed-off-by: Alex <alex.tech.lab@outlook.com> Signed-off-by: Tanish Malekar <tanishmalekar32@gmail.com> Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com> Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Signed-off-by: zzt <zengzetang.zzt@antgroup.com> Signed-off-by: fai <fangzhouai@gmail.com> Signed-off-by: guybd <guy.boudoukh@intel.com> Signed-off-by: Guy Boudoukh <guy.boudoukh@intel.com> Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com> Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com> Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com> Signed-off-by: Karthik Kothuri <karthikkothuri2009@gmail.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com> Signed-off-by: Yan Ma <yan.ma@intel.com> Signed-off-by: zengxian <xiangdong.zeng@intel.com> Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com> Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com> Signed-off-by: Gavin Morris <gmorriscs@gmail.com> Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com> Signed-off-by: Pavani Majety <pmajety@nvidia.com> Signed-off-by: shanjiaz <zsjwpianpian@gmail.com> Signed-off-by: wenpengw-nv <wenpengw@nvidia.com> Signed-off-by: atalhens <sneh.lata@nutanix.com> Signed-off-by: Lang Zhao <lang.zhao@galileo.ai> Signed-off-by: emricksini-h <emrick.birivoutin@hcompany.ai> Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com> Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com> Signed-off-by: Matthias Gehre <matthias.gehre@amd.com> Signed-off-by: Saeid Rostami <srostami@amd.com> Signed-off-by: nemanjaudovic <nudovic@amd.com> Signed-off-by: Matt Woodson <mwoodson@redhat.com> Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Signed-off-by: Bartosz Stefaniak <bstefaniak@nvidia.com> Signed-off-by: Hoang Nguyen Tien <hoang.nguyentien.2601@gmail.com> Signed-off-by: xiangdong <40376367+zxd1997066@users.noreply.github.com> Signed-off-by: Artur Fierka <artur.fierka@intel.com> Signed-off-by: gangula-karthik <gkarthik923@gmail.com> Signed-off-by: Tomer Gilad <tgilad@nvidia.com> Signed-off-by: Itay Etelis <Itay.etelis@gmail.com> Signed-off-by: Adhithya Balakrishnan <adhithya.b2004@gmail.com> Signed-off-by: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com> Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com> Signed-off-by: rishitdholakia13 <rishit+github@cohere.com> Signed-off-by: Divakar Verma <divakar.verma@amd.com> Signed-off-by: Chaemin Lim <chaemin.lim@mangoboost.io> Signed-off-by: Edwin Lim <edwinlim0919@gmail.com> Signed-off-by: Joan Velja <joan.velja22@gmail.com> Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com> Signed-off-by: HDCharles <charlesdavidhernandez@gmail.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: Jane Xu <janeyx@meta.com> Signed-off-by: Shengqi Chen <i@harrychen.xyz> Signed-off-by: Mikhail Kostryukov <mike@triptrack.net> Signed-off-by: Giuseppe Grossi <ggrossi@amd.com> Signed-off-by: aarushjain29 <aarushi.jain2@amd.com> Signed-off-by: Turner Jabbour <doubleujabbour@gmail.com> Co-authored-by: Simon Mo <simon.mo@hey.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com> Co-authored-by: Ameen Patel <ameenp360@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: liuzhenwei <zhenweiliu@habana.ai> Co-authored-by: Robin <jdmjdm1998@163.com> Co-authored-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com> Co-authored-by: Sting Lin <sting.lin@cienet.com> Co-authored-by: Nils Matteson <nilsmatteson@icloud.com> Co-authored-by: Nils Matteson <nils@thaw.sh> Co-authored-by: Rishabh Saini <rishabhsaini01@gmail.com> Co-authored-by: Benjamin Chislett <bchislett@nvidia.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Nick Cao <ncao@redhat.com> Co-authored-by: Sahil Kadadekar <147995121+Sahil170595@users.noreply.github.com> Co-authored-by: vanshbhatia-amd <vansh.bhatia@amd.com> Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com> Co-authored-by: Jason Li <jasonlizhengjian@gmail.com> Co-authored-by: Juan Pérez de Algaba <124347725+jperezdealgaba@users.noreply.github.com> Co-authored-by: yzong-rh <yzong@redhat.com> Co-authored-by: AmeenP <ameen@primeintellect.ai> Co-authored-by: stefankoncarevic <skoncare@amd.com> Co-authored-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Rahul Vishwakarma <168823860+rahulssv-ibm@users.noreply.github.com> Co-authored-by: Martin Hickey <martin.hickey@ie.ibm.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Posedge_Lin <jacklin78911@gmail.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> Co-authored-by: Walter Beller-Morales <walterbm@users.noreply.github.com> Co-authored-by: Ting SUN <suntcrick@gmail.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Ethan Yang <ethany@amd.com> Co-authored-by: amd-ethany <amd-ethany@users.noreply.github.com> Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-authored-by: Zach Zhu <zzqshu@126.com> Co-authored-by: Nicklas Frahm <nicklas.frahm@gmail.com> Co-authored-by: Muhammad Fawaz <mfawaz182@gmail.com> Co-authored-by: Mahad Durrani <114791389+mahadrehmann@users.noreply.github.com> Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Co-authored-by: Ace Eldeib <aeldeib@coreweave.com> Co-authored-by: manayang <jackmanayang@gmail.com> Co-authored-by: manayang <manayang@tencent.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Chaojun Zhang <chaojun.zhang@intel.com> Co-authored-by: sungbin1015 <sbin@solbox.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg> Co-authored-by: aoright <102943475+aoright@users.noreply.github.com> Co-authored-by: 辰言 <oncwnuIWp30GguOyJ615Fqj8H-yc@git.weixin.qq.com> Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: rasmith <Randall.Smith@amd.com> Co-authored-by: Saddss <108515797+Saddss@users.noreply.github.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Saddss <28726669061@qq.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Vivek Sharma <Vivek.Sharma20@ibm.com> Co-authored-by: vivek sharma <vivsharm@redhat.com> Co-authored-by: Puneet Sharma <puneet.sharma21@ibm.com> Co-authored-by: depthfirst-app[bot] <184448029+depthfirst-app[bot]@users.noreply.github.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Co-authored-by: Robert Shaw <robertgshaw2-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Martin Vit <martin@voipmonitor.org> Co-authored-by: shawn <shawnyht@gmail.com> Co-authored-by: almayne <anna.mayne@arm.com> Co-authored-by: Ilya Markov <markovilya197@gmail.com> Co-authored-by: Markov Ilya <markovilya19@gmail.com> Co-authored-by: Chris Leonard <chleonar@redhat.com> Co-authored-by: Kaihang Jiang <88449510+kjiang249@users.noreply.github.com> Co-authored-by: djramic <djoramic@amd.com> Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Co-authored-by: Itay Etelis <itay.etelis@ibm.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: qizixi <22851944+zixi-qi@users.noreply.github.com> Co-authored-by: music-dino <111048524+music-dino@users.noreply.github.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com> Co-authored-by: JooHo Lee <96564470+BWAAEEEK@users.noreply.github.com> Co-authored-by: JooHo Lee <BWAAEEEK@users.noreply.github.com> Co-authored-by: Tiezhen WANG <38108242+xianbaoqian@users.noreply.github.com> Co-authored-by: wenjun liu <wenjun.liu@intel.com> Co-authored-by: jun,du <jun.du@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Kevin H. Luu <khluu000@gmail.com> Co-authored-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-authored-by: Tsvika Shapira <tsvikas@gmail.com> Co-authored-by: Charlie Fu <charlifu@amd.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: weishu <838677410@qq.com> Co-authored-by: Teemu Virolainen <teemu.virolainen@amd.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com> Co-authored-by: Zhewen Li <zhewenli@meta.com> Co-authored-by: Zhewen Li <zhewenli@inferact.ai> Co-authored-by: ZihaoMu <zmu@amd.com> Co-authored-by: Micah Williamson <micah.williamson@amd.com> Co-authored-by: ap9272 <ap9272@users.noreply.github.com> Co-authored-by: gnovack <novackgm@gmail.com> Co-authored-by: peizhang56 <pei.zhang@amd.com> Co-authored-by: Augusto Yao <augusto.yjh@antgroup.com> Co-authored-by: Chang Guo <changg@nvidia.com> Co-authored-by: Yan Xu <yancey.yx@alibaba-inc.com> Co-authored-by: Jingyi Yang <girasoleyang@gmail.com> Co-authored-by: GongLei-HW <1327185943@qq.com> Co-authored-by: Lei Gong <gonglei25@huawei.com> Co-authored-by: alberto <aperdomo@redhat.com> Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Co-authored-by: Mingfei Guo <52491257+guoriyue@users.noreply.github.com> Co-authored-by: Andrey Talman <atalman@fb.com> Co-authored-by: Andrey Talman <atalman@users.noreply.github.com> Co-authored-by: ViranjanPagar <vpagar@nvidia.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com> Co-authored-by: FAN YUCHEN <2994114386@qq.com> Co-authored-by: XuZhou <17717803682@163.com> Co-authored-by: Xu Zhou <xuzhou9417@163.com> Co-authored-by: Hoseung Kim <ghyutjik123@gmail.com> Co-authored-by: HuYiPeng <144002351+MynameFelix@users.noreply.github.com> Co-authored-by: Yipeng Hu <i26268@metax-tech.com> Co-authored-by: larryli2-amd <Larry.Li@amd.com> Co-authored-by: Ashwin Giridharan <ashwing@users.noreply.github.com> Co-authored-by: Joe Rowell <joerowell4@gmail.com> Co-authored-by: Jimmy Lee <58957694+thisisjimmyfb@users.noreply.github.com> Co-authored-by: Robert Shaw <robertgshaw2-redhat@ip-172-31-18-125.us-east-2.compute.internal> Co-authored-by: Qiming Zhang <qiming1.zhang@intel.com> Co-authored-by: peiyuanz <peiyuanz@inferact.ai> Co-authored-by: zhouzhou <zhouzhou@zhouzhoudeMacBook-Pro.local> Co-authored-by: Yejing Lai <yejing.lai@intel.com> Co-authored-by: ErenAta16 <erena6466@gmail.com> Co-authored-by: Ievgen Bondarenko <ibondarenko@student.sierracollege.edu> Co-authored-by: Alejandro Paredes La Torre <99832715+AlejandroParedesLT@users.noreply.github.com> Co-authored-by: zhao, zhenhui <zhenhui.zhao@intel.com> Co-authored-by: vx120 <57470515+vx120@users.noreply.github.com> Co-authored-by: crp0128 <191679376@qq.com> Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: yewentao256 <zhyanwentao@126.com> Co-authored-by: Hugo Centeno <133872718+hugo-cen@users.noreply.github.com> Co-authored-by: liranschour <liranschour@users.noreply.github.com> Co-authored-by: omerpaz95 <73347585+omerpaz95@users.noreply.github.com> Co-authored-by: AlexHuang <alex.tech.lab@outlook.com> Co-authored-by: Tanish Malekar <60835372+tanish-malekar@users.noreply.github.com> Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Hongxia Yang <hongxia.yang@amd.com> Co-authored-by: Jun Kang Chow <junkangchow@gmail.com> Co-authored-by: zzt <zengzetang.zzt@antgroup.com> Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Co-authored-by: guybd <guy.boudoukh@intel.com> Co-authored-by: Rehan Khan <Rehan.Khan7@ibm.com> Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com> Co-authored-by: Mohammad Miadh Angkad …
ningjingbengxiaohai
pushed a commit
to ningjingbengxiaohai/vllm-ascend
that referenced
this pull request
Jul 18, 2026
### What this PR does / why we need it? vllm upstream `e5588e49...85c09e98` (1/1 steps). #### Due to vllm-project/vllm#46998, modified `vllm_ascend/ops/gdn.py`, `vllm_ascend/patch/worker/patch_qwen3_5.py` - Upstream changed QwenGatedDeltaNetAttention.forward, Qwen3NextAttention.forward, and Qwen3NextDecoderLayer.forward to return tensors instead of writing to output buffers. Callers no longer pass output= kwarg. - vLLM Ascend Changes: Added output=None default parameter to AscendGatedDeltaNetAttention.forward and AscendQwen3NextAttention.forward with vllm_version_is("0.23.0") guards. Version-guarded AscendQwen3_5DecoderLayer.forward attention section. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@e5588e4 Signed-off-by: main2main-bot <main2main-bot@users.noreply.github.com> Co-authored-by: main2main-bot <main2main-bot@users.noreply.github.com>
ningjingbengxiaohai
pushed a commit
to ningjingbengxiaohai/vllm-ascend
that referenced
this pull request
Jul 18, 2026
### What this PR does / why we need it? This PR is a follow-up to vllm-project#11875. PR vllm-project#11875 introduced the vLLM v0.24.0 support. Those adaptation changes are treated as the baseline and are intentionally not repeated in this description. This PR removes the remaining vLLM v0.23.0 compatibility paths from `main`. Most changes only remove `v0.23.0` branches and keep the existing v0.24/main implementation. Those mechanical removals are not listed individually below. #### Release defaults and documentation Update the default `VLLM_TAG` in all maintained Dockerfiles from `v0.23.0` to `v0.24.0`. The main-branch support matrix, slash-command examples, and balance-scheduler design documents are updated accordingly. This prevents source-built images and contributor documentation from continuing to select the unsupported v0.23 release. #### Fused MoE weight layout boundary Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard and 310P unquantized Fused MoE implementations. Upstream vLLM PR vllm-project/vllm#44589 was merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is present in the verified-main revision. Therefore v0.24.0 and verified main do not share the same post-load weight-layout behavior: - v0.24.0 explicitly materializes the transposed weights as contiguous tensors before the NPU layout conversion; - verified main follows the post-PR #44589 path without forcing the same intermediate contiguous layout. The standard and 310P unit tests cover both version-specific layouts, the current MoE runner contract, shared-expert handling, and the 310P-specific communication method. #### Qwen3.5/Qwen3Next output contract Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches from `v0.23.0` to `v0.24.0`. Upstream vLLM PR vllm-project/vllm#46998 was merged as `300e3379`, after the v0.24.0 cut point and before the current verified-main revision. It changed the attention contract from writing into a caller-provided output buffer to returning the output tensor. #### Balance scheduler alignment `BalanceScheduler.schedule()` is a downstream copy of the upstream scheduler body because the balance admission logic cannot be implemented through a small wrapper. The copied body is therefore updated from the v0.23.0 implementation to the v0.24.0 implementation while preserving only the existing balance-scheduling deltas. Both supported upstream references now expose: ```python schedule(self, throttle_prefills: bool = False) ``` The old signature-introspection compatibility code is removed and the disabled path delegates directly to `super().schedule(throttle_prefills)`. The v0.24 scheduler alignment also preserves the corresponding upstream behavior for: - DP prefill throttling; - speculative-token and maximum-length accounting; - hybrid Mamba KV-cache hit handling; - resumed-request bookkeeping; - dynamic speculative decoding; - deferred KV-block freeing; - MRV1-only previous-step request tracking. The balance scheduler unit tests and English/Chinese design documents are updated to use v0.24.0 as the release reference. The drift test continues to verify that the copied scheduler body differs from the pinned upstream release only by the intended balance deltas. #### Deferred removal of owner-maintained patches The following compatibility patches and their unit tests are intentionally retained in this PR: - GLM47 zero-argument tool-call streaming parser; - MiniMax-M2 incremental tool-call parser; - MiniMax usage accounting; - `tool_choice=none` empty-`tool_calls` response cleanup. The first three remain behind the existing `vllm_version_is("0.23.0")` condition, with a TODO explaining that their owners will remove them in a follow-up. The `patch_tool_choice_none_content` registration is also left unchanged for the same ownership reason. These files are not required by the newly supported v0.24/main lanes, but deleting owner-maintained patches is intentionally outside the scope of this compatibility cleanup. #### Test boundaries The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were skipped on main. Since v0.23.0 is removed and the cases remain broken on both v0.24.0 and verified main, they are now explicitly skipped on both supported lanes instead of being unintentionally re-enabled. The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather than v0.23.0. MRV2 remains supported only by the verified-main lane, as established by PR vllm-project#11875. The HunyuanVL release helper names are updated from `_v023_*` to `_v024_*` because the bundled-processor compatibility path now targets v0.24.0. This is a naming correction only; the HunyuanVL adaptation itself belongs to PR vllm-project#11875. ### Does this PR introduce _any_ user-facing change? Yes. The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default. There is no additional API change for the supported v0.24.0 and verified-main lanes. ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@85c09e9 --------- Signed-off-by: zhao-stack <2020265299@qq.com> Signed-off-by: shenzhao <shenzhao9@huawei.com> Signed-off-by: MrZ20 <2609716663@qq.com> Co-authored-by: zhao-stack <2020265299@qq.com> Co-authored-by: shenzhao <shenzhao9@huawei.com>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
plasticchris
pushed a commit
to plasticchris/vllm
that referenced
this pull request
Jul 20, 2026
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
MengqingCao
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 24, 2026
### What this PR does / why we need it? Adapt vllm-ascend to vLLM main commits up to July 17. ### Changes | Files | Upstream vLLM change | vllm-ascend adaptation | |-------|---------------------|------------------------| | `vllm_ascend/_310p/ops/fla/idex.py`<br>`vllm_ascend/_310p/ops/fla/l2norm.py`<br>`vllm_ascend/ops/bailing_moe_linear_attn.py`<br>`vllm_ascend/ops/gdn.py`<br>`vllm_ascend/ops/triton/fla/chunk.py`<br>`vllm_ascend/patch/worker/patch_idex_310.py`<br>`vllm_ascend/patch/worker/patch_triton.py`<br>`tests/e2e/nightly/.../test_fused_recurrent_gated_delta_rule.py`<br>`tests/e2e/nightly/.../test_fused_sigmoid_gating_delta_rule.py`<br>`tests/ut/ops/test_gdn_attn_builder.py` | [vllm#48500](vllm-project/vllm#48500) relocated flash-linear-attention from `vllm.third_party.flash_linear_attention` to `vllm.model_executor.layers.fla` | Version-gated all FLA imports and monkey-patch sites with `vllm_version_is("0.25.1")` | | `vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/config_data.py` | [vllm#45939](vllm-project/vllm#45939) replaced SHA-256 rehashed grouped block hashes with chained fine-grained hashes (terminal hash identifies complete block) | Removed `_rehash_block_hash_group` and associated constants. `get_block_hashes` returns `block_hashes[idx + scale_factor - 1]` instead of computing compound hash. | | `vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/coordinator.py`<br>`vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/pool_scheduler.py`<br>`vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/pool_worker.py` | [vllm#45939](vllm-project/vllm#45939) added `hash_block_size` to block pool API; [vllm#46384](vllm-project/vllm#46384) renamed `prefix_match_unit` -> `hash_block_size`, moved hash resolution into individual managers | `ExternalCachedBlockPool` takes `hash_block_size`. `_find_longest_cache_hit` return version-gated. `block_hashes_for_spec` no-op on main. `prefix_match_unit` fallback on main. | | `vllm_ascend/core/recompute_scheduler.py`<br>`vllm_ascend/core/scheduler_profiling_chunk.py`<br>`vllm_ascend/core/single_type_kv_cache_manager.py` | [vllm#46384](vllm-project/vllm#46384) / [vllm#47782](vllm-project/vllm#47782) changed `get_computed_blocks` 3-tuple -> 2-tuple, `get_num_blocks_to_allocate` params, `find_longest_cache_hit` return | Version-gated unpacking with `cast`. `num_tokens_main_model` made optional. `find_longest_cache_hit` returns blocks-only on v0.25.1 vs `(blocks, hit_length)` on main. | | `vllm_ascend/patch/platform/patch_kv_cache_coordinator.py` | [vllm#46384](vllm-project/vllm#46384) / [vllm#47782](vllm-project/vllm#47782) restructured return contracts; added partial Mamba hash hit support on main | Added `enable_partial_hash_hits` (main-only). Added `_cache_hit_alignment_tokens`. `find_longest_cache_hit` tracks per-group lengths, uses `cdiv`. `find_longest_cache_hit_per_group` simplified to single-pass; returns per-group `(blocks, lengths)`. Fixed v0.25.1 `hit_length` over-counting: use physical `spec.block_size` (not `_get_effective_block_size()` which includes `compress_ratio`) when multiplying by physical block count — affected both `find_longest_cache_hit` and `find_longest_cache_hit_per_group`; without this, MLA models (DS-V2-Lite/V3/V4) got 4x inflated hit length -> scheduler skipped non-cached tokens -> segfault. | | `vllm_ascend/patch/platform/patch_mamba_manager.py` | [vllm#46384](vllm-project/vllm#46384) / [vllm#47782](vllm-project/vllm#47782) changed method signatures | Version-gated `find_longest_cache_hit` (delegates to super on main) and `get_num_blocks_to_allocate` (optional params). | | `vllm_ascend/patch/worker/patch_qwen3_5.py` | [vllm#47006](vllm-project/vllm#47006) changed Qwen3-Next sequence-parallel gather contracts | Added `_ascend_all_gather_hidden_and_residual` (main-only). Gated `Qwen3_5DecoderLayer.forward` to v0.25.1. | | `vllm_ascend/ops/vocab_parallel_embedding.py` | [vllm#48390](vllm-project/vllm#48390) changed LM head apply contract | Added `_apply_head` routing to `quant_method.apply` on v0.25.1 and `super()._apply_head` on main. | | `vllm_ascend/patch/worker/__init__.py`<br>`vllm_ascend/patch/worker/patch_v2/patch_eagle_speculator.py`<br>`vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py`<br>`vllm_ascend/worker/v2/spec_decode/eagle/speculator.py`<br>`vllm_ascend/worker/v2/spec_decode/dflash/speculator.py`<br>`vllm_ascend/worker/v2/spec_decode/dspark/speculator.py` | [vllm#48261](vllm-project/vllm#48261) / [vllm#48167](vllm-project/vllm#48167) unified `PrefillSpeculatorCudaGraphManager` + `DecodeSpeculatorCudaGraphManager` -> `SpeculatorCudaGraphManager`; `capture()` parameterless; `set_attn` new params; `dflash_causal` -> `_group_causal` | Gated MRV2 spec decode imports to main-only. Consolidated Eagle managers into `EagleAclGraphManager`. Simplified DFlash/DSpark `capture()`. Added `target_input_buffers`/`target_attn_groups` to `set_attn`. Renamed `dflash_causal` -> `_group_causal` in DSpark `build_draft_attn_metadatas` (DFlash was already updated; DSpark was missed). | | `tests/ut/ops/test_gdn_layerwise_kv.py`<br>`tests/ut/ops/a2/test_gdn_layerwise_kv.py` | [vllm#46998](vllm-project/vllm#46998) changed GDN `forward` from `(hidden_states, output)` -> `(hidden_states) -> Tensor` | Added `_run_gdn_forward` wrapper handling both calling conventions. | | `tests/e2e/conftest.py` | [vllm#48549](vllm-project/vllm#48549) removed `swap_space` from `LLM` | Removed from `VllmRunner` and `DPVllmRunner`. | | `tests/e2e/pull_request/one_card/model_runner_v2/test_basic.py` | [vllm#48261](vllm-project/vllm#48261) MRV2 spec decode main-only | Added `_SKIP_V025_MRV2_SPEC_DECODE` skip marker. | | `tests/ut/core/test_recompute_scheduler.py` | Upstream main's `_free_request` accesses `self.ec_connector` (new attribute) | Added `scheduler.ec_connector = None` to test setup — scheduler is constructed via `__new__` (bypasses `__init__`), so the attribute must be mocked explicitly. | | `tests/ut/distributed/ascend_store/test_config_data.py`<br>`tests/ut/distributed/ascend_store/test_coordinator.py`<br>`tests/ut/distributed/ascend_store/test_pool_worker.py` | [vllm#45939](vllm-project/vllm#45939) / [vllm#46384](vllm-project/vllm#46384) | Updated block hash assertions to terminal hash. Added `hash_block_size`. Version-gated mock returns. | | `tests/ut/ops/test_vocab_parallel_embedding.py` | [vllm#48390](vllm-project/vllm#48390) | Replaced `patch()` with context manager for `set_current_vllm_config`. | | `tests/ut/patch/platform/test_prefix_cache_cp_patches.py` | [vllm#47782](vllm-project/vllm#47782) | Removed obsolete `num_prompt_tokens` kwarg. | | `tests/ut/patch/worker/test_patch_qwen3_5_mtp.py` | [vllm#48429](vllm-project/vllm#48429) | Set `use_attn_reduce_scatter_for_moe = False` on mock layer. | | `tests/ut/test_compressed_prefix_cache.py` | [vllm#46384](vllm-project/vllm#46384) / [vllm#47782](vllm-project/vllm#47782) | Version-gated unpacking. Added `test_find_longest_cache_hit_per_group`. | | `vllm_ascend/patch/__init__.py` | - | Updated HunyuanVL comment block. | - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@54503ec --------- Signed-off-by: wjunLu <wjunlu217@gmail.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com> Co-authored-by: hfadzxy <starmoon_zhang@163.com>
Alex-stack-hub
pushed a commit
to 0moyi0-2024/vllm-ascend_tp
that referenced
this pull request
Jul 27, 2026
### What this PR does / why we need it? vllm upstream `e5588e49...85c09e98` (1/1 steps). #### Due to vllm-project/vllm#46998, modified `vllm_ascend/ops/gdn.py`, `vllm_ascend/patch/worker/patch_qwen3_5.py` - Upstream changed QwenGatedDeltaNetAttention.forward, Qwen3NextAttention.forward, and Qwen3NextDecoderLayer.forward to return tensors instead of writing to output buffers. Callers no longer pass output= kwarg. - vLLM Ascend Changes: Added output=None default parameter to AscendGatedDeltaNetAttention.forward and AscendQwen3NextAttention.forward with vllm_version_is("0.23.0") guards. Version-guarded AscendQwen3_5DecoderLayer.forward attention section. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@e5588e4 Signed-off-by: main2main-bot <main2main-bot@users.noreply.github.com> Co-authored-by: main2main-bot <main2main-bot@users.noreply.github.com>
Alex-stack-hub
pushed a commit
to 0moyi0-2024/vllm-ascend_tp
that referenced
this pull request
Jul 27, 2026
### What this PR does / why we need it? This PR is a follow-up to vllm-project#11875. PR vllm-project#11875 introduced the vLLM v0.24.0 support. Those adaptation changes are treated as the baseline and are intentionally not repeated in this description. This PR removes the remaining vLLM v0.23.0 compatibility paths from `main`. Most changes only remove `v0.23.0` branches and keep the existing v0.24/main implementation. Those mechanical removals are not listed individually below. #### Release defaults and documentation Update the default `VLLM_TAG` in all maintained Dockerfiles from `v0.23.0` to `v0.24.0`. The main-branch support matrix, slash-command examples, and balance-scheduler design documents are updated accordingly. This prevents source-built images and contributor documentation from continuing to select the unsupported v0.23 release. #### Fused MoE weight layout boundary Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard and 310P unquantized Fused MoE implementations. Upstream vLLM PR vllm-project/vllm#44589 was merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is present in the verified-main revision. Therefore v0.24.0 and verified main do not share the same post-load weight-layout behavior: - v0.24.0 explicitly materializes the transposed weights as contiguous tensors before the NPU layout conversion; - verified main follows the post-PR #44589 path without forcing the same intermediate contiguous layout. The standard and 310P unit tests cover both version-specific layouts, the current MoE runner contract, shared-expert handling, and the 310P-specific communication method. #### Qwen3.5/Qwen3Next output contract Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches from `v0.23.0` to `v0.24.0`. Upstream vLLM PR vllm-project/vllm#46998 was merged as `300e3379`, after the v0.24.0 cut point and before the current verified-main revision. It changed the attention contract from writing into a caller-provided output buffer to returning the output tensor. #### Balance scheduler alignment `BalanceScheduler.schedule()` is a downstream copy of the upstream scheduler body because the balance admission logic cannot be implemented through a small wrapper. The copied body is therefore updated from the v0.23.0 implementation to the v0.24.0 implementation while preserving only the existing balance-scheduling deltas. Both supported upstream references now expose: ```python schedule(self, throttle_prefills: bool = False) ``` The old signature-introspection compatibility code is removed and the disabled path delegates directly to `super().schedule(throttle_prefills)`. The v0.24 scheduler alignment also preserves the corresponding upstream behavior for: - DP prefill throttling; - speculative-token and maximum-length accounting; - hybrid Mamba KV-cache hit handling; - resumed-request bookkeeping; - dynamic speculative decoding; - deferred KV-block freeing; - MRV1-only previous-step request tracking. The balance scheduler unit tests and English/Chinese design documents are updated to use v0.24.0 as the release reference. The drift test continues to verify that the copied scheduler body differs from the pinned upstream release only by the intended balance deltas. #### Deferred removal of owner-maintained patches The following compatibility patches and their unit tests are intentionally retained in this PR: - GLM47 zero-argument tool-call streaming parser; - MiniMax-M2 incremental tool-call parser; - MiniMax usage accounting; - `tool_choice=none` empty-`tool_calls` response cleanup. The first three remain behind the existing `vllm_version_is("0.23.0")` condition, with a TODO explaining that their owners will remove them in a follow-up. The `patch_tool_choice_none_content` registration is also left unchanged for the same ownership reason. These files are not required by the newly supported v0.24/main lanes, but deleting owner-maintained patches is intentionally outside the scope of this compatibility cleanup. #### Test boundaries The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were skipped on main. Since v0.23.0 is removed and the cases remain broken on both v0.24.0 and verified main, they are now explicitly skipped on both supported lanes instead of being unintentionally re-enabled. The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather than v0.23.0. MRV2 remains supported only by the verified-main lane, as established by PR vllm-project#11875. The HunyuanVL release helper names are updated from `_v023_*` to `_v024_*` because the bundled-processor compatibility path now targets v0.24.0. This is a naming correction only; the HunyuanVL adaptation itself belongs to PR vllm-project#11875. ### Does this PR introduce _any_ user-facing change? Yes. The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default. There is no additional API change for the supported v0.24.0 and verified-main lanes. ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@85c09e9 --------- Signed-off-by: zhao-stack <2020265299@qq.com> Signed-off-by: shenzhao <shenzhao9@huawei.com> Signed-off-by: MrZ20 <2609716663@qq.com> Co-authored-by: zhao-stack <2020265299@qq.com> Co-authored-by: shenzhao <shenzhao9@huawei.com>
Closed
7 tasks
aditi-amd
pushed a commit
to aditi-amd/vllm
that referenced
this pull request
Aug 4, 2026
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Qwen3NextDecoderLayercurrently passes a preallocated output buffer into itsattention submodules, which write their result via slice assignment:
It prevents all reduce rmsnorm fusion
PR
main
Test Plan
Test Result
main
PR
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.