[Bugfix][Model][SM70] Repair Qwen3.8 Flash Next correctness - #408
Merged
yangzhuxinyzx merged 2 commits intoAug 29, 2026
Merged
Conversation
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
This was referenced Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Repair the Qwen3.8-Flash-Next correctness gaps found while auditing current
1Cat public
mainagainst the live vLLM support PRs, without weakening theexisting V100/SM70 routes.
Base SHA:
62ad1e02693f4c857f3b7547cef1860ee54e8053.This PR:
preserving 1Cat's pinned-host CPU PLE offload;
state grid and stops at every state boundary, including the
(n - 1)tail;batch-invariant split-K reductions;
requantizes dense checkpoint weights at load time; the exact checkpoint
NVFP4 QPN-M1 expert route remains default-on and unchanged.
Output-quality decision
The historical online-QPN8 A/B kept 96/96 GSM8K answers correct but only 8/96
token sequences matched. One answer expanded from 1,653 to 3,626 tokens and
its repeated-4-gram ratio rose from 0.199 to 0.507. This does not justify a
precision-preserving default, so
VLLM_SM70_QWEN4_EXP_ONLINE_QPN8now defaultsto
0and requires explicit=1opt-in with a quality warning.The previous ~82.3 tokens/s result used online QPN8 and is not an accepted
no-loss baseline. The precision-preserving no-MTP control is about 67.6
tokens/s; reaching 80 tokens/s without QPN8 remains a separate optimization
target.
Test Plan
and QPN policy tests.
model gate with QPN8 unset, rather than restarting once per scenario.
Test Result
124 passed, 1 deselected, 18 warningsin 52.17 seconds for the focusedQwen4Exp/hybrid/prefix/QPN suite.
1 passedwith explicit online-QPN8 opt-in.7 passed, including the exact FP16-input/FP32-beta state test.mypy, SPDX, forbidden-import, configuration, and API checks passed.
git diff --checkpassed.safetensors files: zero extra and zero missing keys, so vLLM #54230 does not
affect this checkpoint.
No branch E2E result is claimed yet. GPUs 0-3 host the existing public MTP4
service, and GPUs 4-7 were held by a separate Nsight task at validation time;
this PR deliberately did not preempt either workload or repeatedly restart the
model.
Remaining gates and dependencies
KVBlockZeroerasynchronous H2D staginglifetime fix and should land before high-concurrency prefix-cache acceptance.
Mamba prefix-cache precopy. A bounds-only workaround is not included because
it can replace a crash with silent recurrent-state corruption.
cover deterministic generation, tool calls, PLE request-layout changes, and
prefix lengths around the real state boundaries.
Duplicate-work audit
runner-owned snapshots and request-ID remapping.
logprobs before its drafter, and this PR rejects Qwen4Exp legacy-runner use.
already calls
configure_quant_config(Qwen4ExpMTP).local runners by disabling generic slot mapping for QSA circular groups.
keeps PLE in host RAM.
AI assistance was used for upstream comparison, implementation, tests, and the
audit write-up. Every changed line and reported result was reviewed locally.