[Perf][SM70] Group Qwen3.8 AWQ concurrent decode

[Leonccaa]

Purpose

On the post-#477 SM70 AWQ path, Qwen3.8 C2-C8 decode still schedules the 512-expert dense grouped route even though each step has only 20-80 routed rows. This PR compacts the already-sorted routed rows into contiguous active-expert segments and uses those segments for both AWQ MoE W13 and W2.

The optimized route is intentionally narrow: Qwen3.8, TP4, AWQ W4A16 group-size 32, E512/K10, hidden size 2560, local intermediate size 160, and C2-C8. C1 and unsupported contracts retain the existing path. VLLM_SM70_AWQ_QWEN38_MOE_COMPACT_GROUPED_DECODE=0 is the rollback switch; an explicit =1 fails closed if the loaded extension lacks the required ops.

This aligns AWQ with the active-group execution behavior used by the adjacent SM70 NVFP4/MXFP4 paths; it does not attempt to align quantization formats or numerical behavior. A repository PR/issue search found no existing AWQ grouped-decode implementation with the same scope. In particular, draft #476 covers NVFP4 DFlash2 attention/speculative-decode concurrency rather than AWQ MoE active-expert dispatch, so it is complementary rather than duplicate work.

AI assistance: OpenAI Codex assisted with implementation, analysis, and test orchestration. Leon reviewed every changed line and the evidence below before submission.

Test Plan

.venv/bin/python -m pytest -q \
  tests/quantization/test_sm70_awq_compact_grouped_decode.py

.venv/bin/pre-commit run --files \
  benchmarks/benchmark_sm70_turbomind_exactness.py \
  csrc/ops.h \
  csrc/sm70_turbomind/ops/awq_sm70_gemm.cu \
  csrc/torch_bindings.cpp \
  tests/quantization/test_sm70_awq_compact_grouped_decode.py \
  vllm/_sm70_ops.py \
  vllm/envs.py \
  vllm/model_executor/layers/quantization/awq_sm70_moe.py \
  vllm/model_executor/warmup/awq_sm70_warmup.py

• Build the complete SM70 _C extension from this single commit in an isolated CUDA container.
• Compare compact grouped W13/SILU/W2 output with the per-expert dense reference for TP4 rank 0 and rank 3, C2/C4/C8, and both round-robin and single-hot-expert routing. The clean-main benchmark does not yet include the checkpoint TP-slicing arguments from the [Perf][SM70] Add indexed-A AWQ prefill for Qwen3.8 #464 series, so this check uses that TP-aware benchmark driver while loading vllm and _C.abi3.so from the clean commit snapshot.
• Apply the final PR patch unchanged (stable patch ID ee4e5db9bc20bf9ec9cbcdf2414ead5c6e5dab57) on top of pending [Bugfix][SM70] Enable Qwen3.8 AWQ in dual-compile lane #487, then run one frozen-prompt cell each for C4x64K and C8x16K on 4x V100 PCIe 32 GB: TP4, AWQ g32, FP16 activations/KV, MTP0, prefix cache off, FULL_AND_PIECEWISE, max_num_batched_tokens=8192, output 320, ignore_eos=false. The matched baseline is the same [Bugfix][SM70] Enable Qwen3.8 AWQ in dual-compile lane #487 stack without this patch; neither arm includes [Perf][SM70] Add indexed-A AWQ prefill for Qwen3.8 #464/perf(sm70): right-size AWQ MoE persistent scratch #465/[Perf][SM70] Bound Qwen3.8 AWQ W2 output scratch #470/[Perf][SM70] Compact Qwen3.8 AWQ scale metadata #473.
• Repeat the two-cell endpoint check on the [Perf][SM70] Add indexed-A AWQ prefill for Qwen3.8 #464/perf(sm70): right-size AWQ MoE persistent scratch #465/[Perf][SM70] Bound Qwen3.8 AWQ W2 output scratch #470/[Perf][SM70] Compact Qwen3.8 AWQ scale metadata #473 + [Bugfix][SM70] Enable Qwen3.8 AWQ in dual-compile lane #487 integration stack, again changing only the grouped-decode commit.
• Run a four-request greedy generation quality probe with finite logprobs, exact arithmetic/constraint checks, UTF-8 and repetition checks, while confirming C8/C4/C3/C2 route hits.
• Re-run the prior regression matrix once per cell: C1/C4/C8 x 1K/16K/64K/128K, one warmup followed by one scored run, output 256. Cells requiring more than 99% of measured KV capacity are recorded as capacity skips.

Test Result

• Directed pytest: 3 passed.
• Changed-file pre-commit: passed.
• Full clean-main SM70 extension build: passed (exit 0).
• Operator gate: passed within the configured FP16 bound, with no NaNs, sign failures, or bound failures.
  • TP4 rank 0: max absolute diff 2.44140625e-4, max nonzero ULP 22.
  • TP4 rank 3: max absolute diff 1.220703125e-4, max nonzero ULP 2.
• Four-request model quality probe: 4/4 passed; outputs included 156, 391, exact 4, and normal Chinese prose; every sampled logprob was finite. Logs confirmed C8/C4/C3/C2 grouped-route hits and contained no traceback, OOM, or CUDA error.

Frozen cell	#487 baseline pure ITL	This PR pure ITL	#487 baseline aggregate decode	This PR aggregate decode	Change
C4x64K	36.320 ms	33.492 ms	110.132 tok/s	119.431 tok/s	ITL -7.79%, throughput +8.44%
C8x16K	38.372 ms	36.807 ms	208.483 tok/s	217.349 tok/s	ITL -4.08%, throughput +4.25%

The exact-patch matched prefill/E2E changes were small: C4 prefill throughput -0.19%, TTFT +1.22%, wall -0.72%; C8 prefill throughput +0.68%, TTFT -1.14%, wall -1.76%. Mixed-phase ITL changed -1.45% for C4 and +10.91% for C8; this sparse mixed interval is scheduler-sensitive and is not used to attribute the MoE change. All 12 timed requests completed 320 output tokens with finish_reason=length.

The four-PR integration repeat independently confirmed the direction: C4 pure decode 107.845 -> 117.844 tok/s (+9.27%, ITL -8.49%) and C8 199.993 -> 213.113 tok/s (+6.56%, ITL -6.16%). Its 12 requests also completed all 320 output tokens.

The full regression matrix completed all nine runnable cells; C4x128K, C8x64K, and C8x128K were expected capacity skips. Every runnable request completed 256 output tokens with finish_reason=length, all warmup and scored stderr files were empty, and logs confirmed C2/C4/C8 grouped-route hits. A structured audit found no contract or result errors.

The matrix client's aggregate decode metric includes time spent waiting behind overlapping prefills, unlike the pure-decode table above. On that historical metric, the two long-concurrency cells were:

Regression cell	Earlier four-PR stack	Current post-#477 stack + this PR	Change
C4x64K	24.607 tok/s, 163.192 ms ITL	25.343 tok/s, 158.454 ms ITL	throughput +2.99%, ITL -2.90%
C8x16K	90.981 tok/s, 88.275 ms ITL	95.672 tok/s, 83.947 ms ITL	throughput +5.16%, ITL -4.90%

That historical comparison is a regression check across the accumulated post-#477 stack, not a grouped-decode-only attribution. The exact-patch pure-decode A/B above isolates this PR. The remaining mixed prefill/decode scheduling stall is outside this PR.

The source change has no direct dependency on #487: commit 41f0a8f builds and its operator gate passes on clean main. The real AWQ endpoint evidence applies that exact patch on top of #487 because post-#477 main still needs the pending LM-only admission and late PLE-IPC initialization fix to start this checkpoint through the intended dual-compile route. The endpoint validation commit 50efff1 and PR commit 41f0a8f have the same stable patch ID.

C8x64K is not claimed: 524,288 prompt tokens exceed the measured 505,574-token FP16 KV capacity before output tokens are included.

---

Essential Elements of an Effective PR Description Checklist

• The purpose and exact Qwen3.8 AWQ scope are documented.
• The test plan and commands are documented.
• Unit, build, operator, model-quality, matched E2E, and full-matrix results are documented.
• The rollback switch, ABI fallback, metric boundary, and remaining scheduler limitation are documented.

[Leonccaa]

Closing this for now after reviewing the implementation cost against the measured endpoint benefit.

The optimization is valid and both independent A/Bs confirm the direction, but the production-code surface is relatively large (new CUDA/Torch ops, ABI bindings, warmup, and runtime dispatch) for the isolated pure-decode gain:

• C4x64K: 110.132 -> 119.431 tok/s (+8.44%)
• C8x16K: 208.483 -> 217.349 tok/s (+4.25%)

On the historical matrix metric that includes mixed prefill/decode waiting, the corresponding gains are only +2.99% and +5.16%; the long-prefill scheduler stall remains unchanged.

The branch, exact-patch evidence, tests, rollback contract, and profiling artifacts are being retained. This can be reconsidered if the implementation can reuse existing metadata/operator surfaces more substantially, or if further profiling identifies a materially larger end-to-end opportunity.

Thanks to everyone reviewing the approach. This closure is a cost/maintenance decision, not a correctness failure.