Skip to content

dcp: don't fail boot when B12X DCP collectives don't support the world size - #79

Merged
lukealonso merged 1 commit into
dev/eldritch-enlightenmentfrom
fable/dcp-warmup-ws-guard-20260707
Jul 7, 2026
Merged

lukealonso merged 1 commit into
dev/eldritch-enlightenmentfrom
fable/dcp-warmup-ws-guard-20260707

Conversation

@voipmonitor

Copy link
Copy Markdown

Problem

The B12X PCIe DCP channel exists only for world sizes 2/4/8, and the runtime dispatchers (dcp_b12x_all_gather_heads, dcp_a2a_lse_reduce) already fall back to NCCL collectives per call. The dedicated DCP warmup added in "Optimize B12X DCP collectives and warmup" turned that graceful fallback into a boot failure: with TP6 + DCP3/DCP6 (GLM head66 configs that booted fine on the v13 stack) warmup_b12x_dcp_a2a raises

RuntimeError: B12X PCIe DCP query all-gather is unavailable for the configured attention geometry

and EngineCore dies. This regression is on the current base branch / v14 image — it is independent of PR #78.

Fix

  • warmup_b12x_dcp_a2a: log once and skip when the DCP world size is not 2/4/8 (still raises for genuinely broken geometries at supported sizes). This also covers the DSv4 module path in kernel_warmup.py.
  • MLAAttention: don't set dcp_b12x for unsupported DCP sizes, so neither warmup nor the per-step dispatch attempts the PCIe channel at all.

Verification

GLM-5.2 NVFP4 TP6 on the v14 image (vllm02f5b41-b12xe44cb77) with the v13 memory shape (GMU 0.957, max_model_len 128000, max_num_batched_tokens 2048), MTP3, MOE_MODE=a16:

Mode KV cache tokens short-ctx tok/s 3k-ctx tok/s CJK
TP6 DCP1 168,384 131.1 106.7 0
TP6 DCP6 (fixed) 989,125 74.9 74.5 0

Without the fix, TP6 DCP6 dies at warmup with the RuntimeError above. Head66 virtual-TP padding (attention heads 64 -> 66) and PCIe allreduce at world size 6 both work on this stack.

New unit test: test_warmup_skips_unsupported_world_size (no dist/GPU needed).

Note: the same commit was originally pushed as the second commit of #78; it is split out here because it fixes a pre-existing base-branch regression and can merge independently. #78 will be rebased to contain only the hybrid-dispatch feature.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@voipmonitor, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 33 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: be4d500d-35e0-44ee-81ee-df50d9acc47f

📥 Commits

Reviewing files that changed from the base of the PR and between 872145f and 59944f4.

📒 Files selected for processing (3)
  • tests/distributed/test_dcp_a2a.py
  • vllm/model_executor/layers/attention/mla_attention.py
  • vllm/v1/attention/ops/dcp_alltoall.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fable/dcp-warmup-ws-guard-20260707

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@voipmonitor
voipmonitor changed the base branch from codex/modelopt-online-fp8-overlay-20260706 to dev/eldritch-enlightenment July 7, 2026 14:15
…d size

The B12X PCIe DCP channel exists only for world sizes 2/4/8, and the
runtime dispatchers already fall back to NCCL collectives per call. The
dedicated DCP warmup added in 'Optimize B12X DCP collectives and warmup'
turned that graceful fallback into a boot failure: with TP6 + DCP3/DCP6
(GLM head66 configs that worked on the v13 stack) warmup_b12x_dcp_a2a
raised 'B12X PCIe DCP query all-gather is unavailable for the configured
attention geometry' and EngineCore died.

- warmup_b12x_dcp_a2a: log once and skip when world size is not 2/4/8
  (still raises for genuinely broken geometries at supported sizes).
- MLAAttention: don't set dcp_b12x for unsupported DCP sizes, so neither
  warmup nor the per-step dispatch attempts the PCIe channel at all.

Verified on GLM-5.2 NVFP4 TP6/DCP6/MTP3 (v13 wiki shape: GMU 0.957,
max_model_len 128000, max_num_batched_tokens 2048): boots clean, KV cache
989k tokens, test.py c0/c3000 coherent (CJK 0), 73-75 tok/s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@voipmonitor
voipmonitor force-pushed the fable/dcp-warmup-ws-guard-20260707 branch from b75e729 to 59944f4 Compare July 7, 2026 19:43
@lukealonso
lukealonso merged commit c872b12 into dev/eldritch-enlightenment Jul 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants