dcp: don't fail boot when B12X DCP collectives don't support the world size - #79
Conversation
|
Warning Review limit reached
Next review available in: 33 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…d size The B12X PCIe DCP channel exists only for world sizes 2/4/8, and the runtime dispatchers already fall back to NCCL collectives per call. The dedicated DCP warmup added in 'Optimize B12X DCP collectives and warmup' turned that graceful fallback into a boot failure: with TP6 + DCP3/DCP6 (GLM head66 configs that worked on the v13 stack) warmup_b12x_dcp_a2a raised 'B12X PCIe DCP query all-gather is unavailable for the configured attention geometry' and EngineCore died. - warmup_b12x_dcp_a2a: log once and skip when world size is not 2/4/8 (still raises for genuinely broken geometries at supported sizes). - MLAAttention: don't set dcp_b12x for unsupported DCP sizes, so neither warmup nor the per-step dispatch attempts the PCIe channel at all. Verified on GLM-5.2 NVFP4 TP6/DCP6/MTP3 (v13 wiki shape: GMU 0.957, max_model_len 128000, max_num_batched_tokens 2048): boots clean, KV cache 989k tokens, test.py c0/c3000 coherent (CJK 0), 73-75 tok/s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b75e729 to
59944f4
Compare
Problem
The B12X PCIe DCP channel exists only for world sizes 2/4/8, and the runtime dispatchers (
dcp_b12x_all_gather_heads,dcp_a2a_lse_reduce) already fall back to NCCL collectives per call. The dedicated DCP warmup added in "Optimize B12X DCP collectives and warmup" turned that graceful fallback into a boot failure: with TP6 + DCP3/DCP6 (GLM head66 configs that booted fine on the v13 stack)warmup_b12x_dcp_a2araisesand EngineCore dies. This regression is on the current base branch / v14 image — it is independent of PR #78.
Fix
warmup_b12x_dcp_a2a: log once and skip when the DCP world size is not 2/4/8 (still raises for genuinely broken geometries at supported sizes). This also covers the DSv4 module path inkernel_warmup.py.MLAAttention: don't setdcp_b12xfor unsupported DCP sizes, so neither warmup nor the per-step dispatch attempts the PCIe channel at all.Verification
GLM-5.2 NVFP4 TP6 on the v14 image (
vllm02f5b41-b12xe44cb77) with the v13 memory shape (GMU 0.957,max_model_len128000,max_num_batched_tokens2048), MTP3,MOE_MODE=a16:Without the fix, TP6 DCP6 dies at warmup with the RuntimeError above. Head66 virtual-TP padding (
attention heads 64 -> 66) and PCIe allreduce at world size 6 both work on this stack.New unit test:
test_warmup_skips_unsupported_world_size(no dist/GPU needed).Note: the same commit was originally pushed as the second commit of #78; it is split out here because it fixes a pre-existing base-branch regression and can merge independently. #78 will be rebased to contain only the hybrid-dispatch feature.
🤖 Generated with Claude Code