Repository navigation
vLLM: opt-in ReplaySSM decode for Gated DeltaNet (Qwen3.5 / Qwen3.6 / Qwen3-Next) - #1761
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
|
@codex security review |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bddac253cd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| decode_base_cpu = m.replayssm_decode_base_cpu | ||
| seq_lens_cpu = m.seq_lens_cpu_upper_bound | ||
| if decode_base_cpu is None or seq_lens_cpu is None: |
There was a problem hiding this comment.
When the exported patch is used with both async scheduling and speculative decoding enabled, seq_lens_cpu_upper_bound is not an exact accepted sequence length, but this guard accepts it and derives write_pos and is_flush from it. The ring can consequently read unwritten entries or flush at the wrong token, silently changing logits; the upstream implementation explicitly rejects scheduler_config.async_scheduling and speculative_config is not None for this reason. Include that condition here so ReplaySSM is refused and the caller can fall back to the normal decode path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not reachable on the vLLM this targets: in vLLM 0.31 VllmConfig.validate_mamba_cached_kernel sets use_kda_recoverssm whenever num_speculative_tokens > 0 and raises "RecoverSSM is only supported for Kimi-K3 KDA" for every other architecture, so a speculative config with use_replayssm never reaches the GDN builder for Qwen. The patched builder also disables itself when use_kda_recoverssm is set, and load_vllm passes no speculative config.
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review |
1 similar comment
|
@codex review |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
1 similar comment
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
CI triage: |
1 participant
Stacked on #1759. Adds vLLM ReplaySSM decode for Gated DeltaNet models (Qwen3.5, Qwen3.6, Qwen3-Next) to fast_inference / GRPO rollouts, opt-in:
Why a backport
Released vLLM has ReplaySSM for Mamba2 only. The Gated DeltaNet half, vllm-project/vllm#48792 ("ReplaySSM: cache SSM inputs for faster Gated DeltaNet standard decode"), was reviewed and then closed unmerged when its fork was deleted. This PR carries it onto released vLLM (0.31) at runtime:
unsloth_zoo/_vendored/vllm/gdn_replayssm_decode.py: the PR's Triton decode kernel, verbatim except for its launch-config lookup (inlined; released vLLM has no GDN entry). Apache-2.0 LICENSE, MANIFEST and NOTICE entry added the same way as the vendored fla tree, and mirrored in_licenses.py.unsloth_zoo/vllm_gdn_replayssm.py: the integration, ported from the same PR.d, keyk, log-decayg), both in the layer spec and in the model-class hooks that size the hybrid page.GDNAttentionMetadataBuilderderives the per-row ring cursor and flush flag fromreplayssm_decode_base_cpu, with persistent buffers for CUDA graphs. The metadata is a dataclass subclass, soupdate_block_table'sdataclasses.replacekeeps the fields.QwenGatedDeltaNetAttentiondecodes through the ring both in decode-only batches and for the decode rows of mixed prefill + decode batches. The second matters: the conventional update would otherwise advance a state the ring then replays.ModelConfig.supports_replayssmis widened for the five Qwen GDN architectures (same mechanism as vLLM: ReplaySSM on by default for Mamba2 hybrids #1759).Why opt-in
Unsloth
fast_generate, vLLM 0.31.0, GSM8K first 250 (greedy), forced 256-token decode, fresh process per arm, ReplaySSM against today's default (V2 runner):Accuracy holds and peak memory is unchanged. Speed does not justify a default: vLLM 0.31 runs Triton ReplaySSM only on Model Runner V1, and for Qwen3.5 that switch alone costs 23% at 64 sequences on A100 (V1 without ReplaySSM: 0.766x) against about 3% for Nemotron-H. The GDN ring kernel is also slower than the packed decode kernel on A100 below about 100 sequences (isolated kernel, A100: 0.67x at 32, 1.63x at 128; RTX PRO 6000: 1.02x to 1.80x at every size; L4: 1.19x to 1.93x). The GDN recurrence is 5% to 24% of a Qwen3.5-2B decode step and 3% to 14% for Qwen3.6-35B-A3B, which bounds the end-to-end gain. So auto mode leaves GDN models alone; it can become a default once vLLM runs ReplaySSM on the V2 runner.
Tests
tests/test_vllm_replayssm.py: the GDN model types are gated, and auto mode does not request them.tests/test_vendored_license_shipped.py: the vendored vLLM license and manifest are embedded verbatim.tests/test_relative_imports_resolve.py: passes; the kernel import is spelled absolutely.