[Kernel][SM70] Add native-g32 AWQ QPN M1 operator

[Leonccaa]

Purpose

Add a native SM70 operator for the Qwen3.8 TP4 / native-group-32 / E512 / top-k 10 AWQ single-token (M=1) routed-expert step: _C::awq_moe_qpn_m1_sm70_out, exposed as vllm._sm70_ops.awq_moe_qpn_m1_sm70_out, with an independent GPU test and a design/contract document.

On 4x V100 the current AWQ M=1 path runs W13 and W2 through the generic TurboMind GEMM: M padded to 8, N320 padded to 384, a 2-stage shared-memory pipeline and a serial cross-CTA split-K. Isolated profiling of the real layer-1 weights showed AWQ W13 at about 1.9x the time of the NVFP4 M=1 kernel while reading 4% fewer DRAM bytes, i.e. an execution-structure gap rather than a format cost. This operator applies the existing quadpair-N m8n8k4 M=1 dataflow (mxfp4_qpn_m1_sm70.cu, v100-skinny lineage, retained LICENSE.v100-skinny notice) to AWQ's existing prepared banks, and closes most of that gap at the operator level.

The operator is self-contained: it consumes the prepared banks that the AWQ loader already produces, needs no repack or extra weight copy, and can be benchmarked and validated on its own. It registers nothing in the model runtime and changes no default route.

What it does

• W13: selected experts in original router order, N32 tiles, 16-warp CTA-local FP32 split-K reduction, FP16 gate/up materialization, fused SwiGLU into a (10, 160) FP16 intermediate.
• W2: one warp per route, FP32 dot products, per-route FP16 materialization, then ordered FP32 router-weight accumulation into the (1, 2560) FP16 output. No atomic FP16 reduction.
• Dequantization keeps the current TurboMind U4-to-half conversion and the half_fma(q, scale, half(-zero * scale)) boundary; it does not substitute half((q - zero) * scale). Both existing metadata layouts are supported: 4-byte {FP16 scale, FP16 bias} and 3-byte {FP16 scale, u8 zero}.
• Validates dtype, exact shapes, device, contiguity, 16-byte alignment and output/input non-overlap; requires SM70. Invalid expert IDs contribute zero; duplicate valid IDs keep their separate router weights. Graph-capturable, no allocation, fake implementation registered for torch.compile.

The CTA-local reduction changes FP32 summation order relative to the legacy split-K route, so bitwise equality with the legacy path is not promised and neither path is treated as the mathematical reference. Weight decoding is bitwise identical to the legacy path on the sampled real weights (3.27M values); on layer-1 W13 outputs, 194 of 76,800 values differ in the last FP16 bit, max absolute difference about 3.05e-5.

Operator results

Real layer-1 AWQ weights, TP4 rank 0, M=1, top-k 10, K2560/N320 (W13) and K160/N2560 (W2). CUDA Graph replay of 48 calls, 7 samples, medians in ms per 48 calls. "Hot" reuses the same 10 experts; "rotating" cycles through 480 experts so the working set exceeds L2. The NVFP4 column is the existing native NVFP4 M=1 kernel on the same skeleton, as a reference, not a target.

W13 + SwiGLU, ms / 48 calls	hot 10 experts	rotating 480 experts
Current AWQ TurboMind route	1.0890	1.1354
This operator, 4-byte metadata	0.5854	0.6406
This operator, 3-byte metadata	0.6024	0.6578
Native NVFP4 M=1 kernel (reference)	0.5472	0.5831

W2 + weighted reduction, ms / 48 calls	hot 10 experts	rotating 480 experts
Current AWQ route (W2 + standalone reduce)	0.8072	0.8687
This operator, 4-byte metadata	0.4057	0.4514
This operator, 3-byte metadata	0.4015	0.4331
Native NVFP4 M=1 kernel (reference)	0.3737	0.4235

These are operator timings, not model throughput. Evidence root: qwen38-c1-qpn-awq-20260905/evidence-r1 (W13) and evidence-r2 (W2).

What is deliberately not in this PR

Runtime dispatch is not included. Selecting this operator changes model behavior (reduction order, hence sampled-token trajectories at near-ties) and has to be judged with its own full-model quality evidence, whereas the operator itself can be reviewed as a kernel against a fixed contract. An opt-in, default-off dispatch on top of this operator will be proposed separately; a full-model C1x64K measurement with that dispatch already exists and will be presented there.

Also not included: any change to prepared weight layout, loader, prefill, grouped decode, router, shared experts, attention or defaults.

Duplicate-work check

Open-PR searches on 2026-09-06 in 1CatAI/1Cat-vLLM for "AWQ QPN", "QPN M1", "awq_qpn", "SM70 AWQ single-token", "AWQ M1" and "Qwen3.8 AWQ decode" found no competing native AWQ M=1 operator. Hits were #509 (conversion-cache release), #349 (PP2/TP4 FP8 tactics), #431 and #410, which are unrelated. #510 tunes the NVFP4 M=1 kernel and does not implement an AWQ decoder. Base is main@755baae1d0; the commit applies cleanly.

Test plan and results

• Native SM70 build on V100: tests/kernels/test_sm70_awq_qpn_m1.py passed 2/2. It independently constructs both prepared layouts and the N32/K8 nibble order, checks one-hot boundary reads at K indices 0/31/32/2559 against exact expectations, an FP64 W2 dot reference with an explicit FP16 rounding allowance, changing CUDA Graph inputs, invalid and duplicate expert IDs, aliased/misaligned/wrong-dtype argument rejection, and the registered fake op under torch.compile. A CPU skip is not a GPU pass.
• Real-checkpoint harness on the same build: 256 native/prototype/dynamic-graph exact comparisons across both metadata layouts and all four TP partitions; 72 independent FP64 stage checks with no retained rounding-bound violations; 48 invalid-argument rejections. One tiny-subnormal case sits above an old relative-only smoke threshold (relative L2 0.00201759, max absolute 5.96e-8); it is recorded, not renamed a pass.
• Pre-commit on this commit (ruff, ruff-format, clang-format, typos, markdownlint, mypy, SPDX header, sign-off): all hooks passed. git diff --check: clean.

Provenance

Fork review draft: Leonccaa#10 (same commit, rebased onto main without changes).

AI assistance was used. OpenAI Codex implemented the operator, tests and evidence under Leon's direction; Claude Code performed an independent line-by-line review of the kernel, the rebase onto main, the pre-commit run and this submission. Human review is still required.

🤖 Generated with Claude Code

[Leonccaa]

Fresh exact-head SM70 regression (2026-09-06 UTC)

Rebuilt and tested standalone head 5b4135fd8bb1f89d0f090c17e7144c09d4f5ca37,
not just the old fork binary. The loaded _C SHA-256 is
40a58bdb0d6534fb82497873a659f39ac3fd4ed71536499674074a3a03219747.
Tracked runtime files were checked against the exact git archive. No product
code was changed for this regression.

On V100 / SM70 / CUDA 12.8:

• 2 portable GPU tests passed.
• 256 native/reference and dynamic CUDA Graph comparisons passed bitwise.
• 72 independent FP64 stage-pair rounding-bound checks: zero violations.
• 48 invalid-argument rejection checks passed.
• Coverage includes both metadata layouts, all four logical TP4 checkpoint
  partitions, changing inputs, invalid expert IDs, and duplicate expert routes
  with their separate weights. The four logical partitions in this operator
  test are not being described as a TP4 full-model execution.

The old relative-only gate still has its documented tiny-value exception:
maximum absolute error 5.960464477539063e-8, relative L2
0.0020175909157842398 versus the old 0.002 limit. That gate was not
changed or relabeled passed; it is distinct from the independent FP64
stage-bound checks. Native/reference equality is not equality to the old
TurboMind reduction trajectory.

A small real-checkpoint layer-1 / rank-0 graph timing check (12 ABBA pairs,
64 calls per graph) measured this standalone operator at 21.624 us for 4B
and 21.808 us for 3B. Its 3B reader is the scalar implementation preceding
#522. These warm single-layer timings do not replace the PR's differently
defined rotating-expert experiment, and are not whole-model speedup claims.

The final public stack also passed 209 adjacent tests in a complete native
SM70 environment (14 deprecation warnings). Initial CPU-environment failures
and a stale benchmark-helper mount failure were retained; correcting the
test environment/mount resolved them without product edits. This adjacent
result is stack coverage, not an independent rebuild of every intermediate
PR head. No fresh compute-sanitizer run is claimed.

Full-model runtime-selection/quality results belong to #521; this PR alone
does not select the operator in a model. Fixed-prefix logits, natural-EOS task
scores, and fixed-budget truncations remain separate acceptance dimensions.

AI assistance: Codex performed the source audit, isolated builds, test
execution, and preparation of this addendum at Leon's request. This is not
an independent maintainer review or upstream GPU CI result.

[yangzhuxinyzx]

本轮源码已通过后继 PR #521/#522 和集成 #526 纳入 main，优化没有丢失。核对 main@4366d9d5fe 与本 PR：QPN 核心差异仅为后继的 W13 合作元数据读取；原 GPU 回归测试内容相同。

本 PR 的 5b4135f 是独立提交，后继分支采用了另一份 kernel 提交 9ae1a29，因此 GitHub 没有自动将 #519 标成 merged；直接合并会产生 kernel/docs 的 add/add 冲突。本轮遵照“没合并的继续 open”保留该 PR，不关闭、不把新主线退回旧的标量读取实现。后续只需处理提交历史/重叠整合，不需要重新发明或重复叠加这个加速。