Repository navigation
perf(qwen-flash): accelerate prefill and MTP catch-up - #485
Merged
Merged
Conversation
Chiakai-Chang
pushed a commit
to Chiakai-Chang/gufo
that referenced
this pull request
Oct 10, 2026
…-org#485) # Conflicts: # src/cli/serve/continuation_cache.cpp # src/cli/serve/http_server.cpp # src/cli/serve/openai_chat.cpp # src/models/qwen/chat_template.cpp # src/models/qwen38_flash_next/engine.cpp # src/models/qwen38_flash_next/kernels/rocm/executor.cpp # src/models/qwen38_flash_next/kernels/rocm/executor.hpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flash-Next prefill was doing unnecessary activation writes, n-gram uploads and MTP hidden-state copies. This removes that work, improves GDN/expert/projection kernels, and uses the existing selector scratch more efficiently at long context. It also corrects MTP catch-up across the dense/sparse attention boundary. The optimized paths are the default, with no new switches or dependencies.
Production Nix, HTTP pp2048/tg128; previous published results versus the refreshed sweep:
MTP pp includes predictor catch-up and takes the higher mixed/repetitive rate. Repetitive MTP reaches 60.39 tok/s for one user and 162.98 summed decode tok/s at C8. README, benchmark tables/charts and quality evidence are updated. Loading and reference-engine measurements retain their original provenance.
Validation:
The historical C2 repetitive result did not reproduce: 84.52 tok/s in this sweep, 84.46 on current main and 84.40 in a focused repeat. The original sweep result is retained. The existing independent upstream vision-parity gap remains unresolved; these changes do not introduce a new regression. Detailed controls and provenance are in
docs/models/qwen3.8-flash-next/artifacts/prefill-long-context.json.