Skip to content

[Kernel][SM70] Avoid NVFP4 MoE prefill input expansion - #390

Merged
yangzhuxinyzx merged 2 commits into
mainfrom
codex/v100-qwen38-nvfp4-indexed-prefill-20260828-160000
Aug 28, 2026
Merged

yangzhuxinyzx merged 2 commits into
mainfrom
codex/v100-qwen38-nvfp4-indexed-prefill-20260828-160000

Conversation

@yangzhuxinyzx

@yangzhuxinyzx yangzhuxinyzx commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Purpose

Stacked on #387. Remove the Qwen3.8 TP4 NVFP4 prefill input-expansion bottleneck without changing routing order, W2, activation, unpermute, decode, or output arithmetic.

The exact E512/K10 W13 route keeps the existing stable expert sort, derives original token-row indices in a lightweight metadata kernel, and uses TurboMind SM70 MatrixLayout.idxs instead of materializing [tokens * 10, 2560] FP16 rows.

Safety

  • Restricted to exact TP4 Qwen3.8 W13: E512/K10, K2560/N320, group size 16, at least 128 tokens.
  • VLLM_SM70_NVFP4_QWEN38_MOE_INDEXED_PREFILL=0 restores the old route.
  • Implicit default falls back with an older extension; explicit =1 fails closed when either operator is missing.
  • Draft remains stacked on [Kernel][SM70] Group QSA Page4 prefill #387 until grouped QSA lands.

Test Result

  • SM70 _C and _moe_C incremental builds and schema imports pass.
  • Focused suite: 15 passed, 5 skipped.
  • Ruff, clang-format, typos, and git diff --check pass.
  • Locked V100 8192-token operator chain: 6.026752 -> 4.235264 ms (1.423x, 1.791488 ms/layer/rank saved). All routing metadata and W13 output are bitwise equal.
  • One-start TP4 V2/no-MTP endpoint versus [Kernel][SM70] Group QSA Page4 prefill #387:
    • 32K: 5998.65 -> 6507.10 tok/s (+8.47%)
    • 64K: 5777.43 -> 6241.48 tok/s (+8.03%)
    • 131K: 5450.92 -> 5871.47 tok/s (+7.71%)
    • 8K: 6394.74 -> 6970.80 tok/s (+9.01%)
  • Arithmetic, Chinese, 32K, 64K, 131K, 8K, and profile token hashes match [Kernel][SM70] Group QSA Page4 prefill #387 exactly.

Full implementation and benchmark evidence are recorded in docs/design/sm70_v100_migration_control.md.

Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
@yangzhuxinyzx
yangzhuxinyzx changed the base branch from codex/v100-qwen38-grouped-page4-prefill-20260828-144759 to main August 28, 2026 08:44
@yangzhuxinyzx
yangzhuxinyzx marked this pull request as ready for review August 28, 2026 09:00
@yangzhuxinyzx
yangzhuxinyzx merged commit 3bfb572 into main Aug 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant