Repository navigation
Conversation
The reservation and the pool builders now read the same width from the parallel context, so the two cannot drift: initialize_model_parallel stamps attn_tp_size from the widths the attention groups are actually built at, which can differ from the configured leaves once initialize_dp_attention settles the effective attn_dp_size. Re-deriving the width from the configured leaves at the call site would miss that correction. Adds a CPU regression test for the DP-attention arithmetic. Co-authored-by: liubaichuan <liubaichuan@infini-ai.com>
The test used to spell the topology as dp_size=16 + enable_dp_attention. Resolution now leaves the deprecated flag false and override declarations do not re-run the deprecation mapping, so the published topology was tp16 with attention DP width one and the test asserted the unsharded width. Declare the attention DP width directly.
b261650 to
3afc785
Compare
|
Rebased onto current main. The code half of this PR is already there: #42651 landed the same attn_tp_size fix in the reservation while backing the draft pool to post-capture sizing, so after the rebase the code line matches main and what remains here is:
Re-verified locally after the rebase: the whole pool configurator suite passes, 52 tests plus 11 subtests, on a CPU-only run. |
|
Small status bump: with #42651 merged, this PR is now test-plus-comment only (the regression test pins the DP-attention budget arithmetic that the reservation fix relies on). The pr-gate failure above is just the missing @hanming-lu, you merged #42651; could you tag |
Motivation
Fixes #38202. Supersedes #38203.
Under DP attention the DFLASH/DSPARK draft KV pool is reserved at the raw
tp_sizewidth while the pool builders shard the draft KV heads byattn_tp_size, which is smaller byattn_dp_size. The reservation under-counts the draft pool by that factor and the draft worker OOMs out of the runtime slack, exactly as reported in the issue.#38203 identified the same bug and proposed deriving the width from the configured leaves (
tp_size,dp_size,enable_dp_attention) at the call site. That derivation no longer works on current main: resolution leaves the deprecatedenable_dp_attentionflag false, and thederive_attention_widthshelper it imported is gone. More fundamentally, re-deriving topology at the call site duplicates logic that can drift from what the runtime actually stamps;initialize_dp_attentionsettles the effectiveattn_dp_sizeduringinit_parallel_runtime, beforeModelRunner.__init__ever reaches this resolver.Modifications
_resolve_dflash_draft_cell_sizenow readsget_parallel().attn_tp_size, the same width the draft KV pool builders shard by. The call site runs afterinit_parallel_runtime(init_distributed_environment->initialize_model_parallel->initialize_dp_attention), so the stamped value already carries any DP-attention correction.test_draft_budget_prices_the_attention_sharded_width: publishes a tp16/attn-dp16 topology and asserts the draft KV heads are priced at the sharded width.attn_dp_size=16directly, since resolution leaves the deprecatedenable_dp_attentionflag false and override declarations do not re-run the deprecation mapping.Co-authored with @Baichuan7, who reported the issue and wrote the first fix.
Accuracy Tests
No model-output path is touched; this only changes how much KV budget the draft pool reserves.
Speed Tests and Profiling
No hot path is touched; the resolver runs once at startup.
Checklist
test_pool_configurator.py, 52 passed locally on CPU).CI States
Latest PR Test (Base): ❌ Run #37583413583
Latest PR Test (Extra): ❌ Run #37583413529
Latest PR Test (AMD ROCm 10): ❌ Run #37583413643