Make prefill shared-read completion explicit and default-on - #37958
Open
weireweire wants to merge 4 commits into
Open
weireweire wants to merge 4 commits into
weireweire wants to merge 4 commits into
Conversation
weireweire
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
Ying1123,
hebiao064,
hnyls2002,
ispobock and
merrymercy
as code owners
September 4, 2026 07:37
weireweire
force-pushed
the
refactor/default-prefill-shared-read-done
branch
3 times, most recently
from
September 7, 2026 09:02
12709ba to
df12976
Compare
weireweire
changed the title
weireweire
force-pushed
the
refactor/default-prefill-shared-read-done
branch
2 times, most recently
from
September 8, 2026 03:08
a62dbd6 to
95ecc9b
Compare
Root cause: prefill WAR eligibility and DSV4 snapshot readiness were split between runner-side gates and a mutable metadata flag. Eager and CUDA graph runners had to prepare first and later rediscover whether PRE_REPLAY was safe, while wrapper delegation was incomplete. Fix: centralize per-batch eligibility in one backend resolver, make preparation return SharedReadEnds explicitly, and pass that result through eager and prefill graph execution. Use default-false global and prefill-only coarse-barrier opt-outs, skip snapshot preparation under either opt-out, preserve capture-stable fallbacks, delegate through HybridAttnBackend, and remove the DSV4 readiness flag. Validation: compiled all changed Python files and ran pre-commit run --all-files --show-diff-on-failure.
weireweire
force-pushed
the
refactor/default-prefill-shared-read-done
branch
from
September 8, 2026 05:18
95ecc9b to
fb4f006
Compare
Contributor
Author
|
/tag-and-rerun-ci |
Root cause: the common prefill shared-read resolver accessed self.model_runner even though attention backends do not uniformly retain ModelRunner. The prefill CUDA graph load_batch return contract also changed without updating one existing caller test. Fix: pass the speculative algorithm explicitly from eager and prefill graph runners, update existing resolver tests, and unpack the load_batch boundary result in the existing CUDA graph test. Validation: compiled all changed Python files and ran pre-commit across the full repository.
Root cause: the PR became unmergeable after context-parallel prefill handling changed on main, preventing normal pull-request CI from being created. Fix: merge current main, retain its all-CP guard, and preserve the unified prefill shared-read boundary resolution. Update the existing replay mock to pass the now-required speculative algorithm. Validation: Python compileall passed for the modified runtime paths. PR-relative diff checks and formatting/lint hooks passed; the all-files registered-test taxonomy hook reports three unrelated NPU violations already present on current main.
Collaborator
|
/rerun-failed-ci |
1 similar comment
Collaborator
|
/rerun-failed-ci |
Collaborator
|
@weireweire could you fix thie conflicts? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance validation
A direct parent/head A/B used the same InferenceX c960 configuration: GB300, disaggregated 4P+4D (prefill DEP8, decode DEP16), DeepSeek-V4-Pro, DSpark K=6, concurrency 960, and a one-hour AIPerf agentic trace. Both runs completed with zero request errors or cancellations.
15d2cbcc)98713d66)This validates no performance regression. The approximately 2% aggregate throughput difference is treated as normal run-to-run variation, not as a performance-improvement claim for this refactor. Fixed five-minute windows alternated between the two runs and showed no persistent late-run degradation. The current PR head differs from the benchmarked revision only by a comment.
Validation
CI States
Latest PR Test (Base): 🚫 Run #34811199006
Latest PR Test (Extra): ❌ Run #34811198808
Latest PR Test (AMD ROCm 10): ❌ Run #34811198969