Skip to content

[Kernel][SM70] Add native-g32 AWQ QPN M1 operator - #10

Draft
Leonccaa wants to merge 1 commit into
perf/sm70-qwen38-awq-grouped-main-20260905from
perf/sm70-awq-qpn-kernel-20260905
Draft

Leonccaa wants to merge 1 commit into
perf/sm70-qwen38-awq-grouped-main-20260905from
perf/sm70-awq-qpn-kernel-20260905

Conversation

@Leonccaa

@Leonccaa Leonccaa commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Purpose and stack

Layer 1/3: native operator, binding, independent GPU tests and design contract.
Base is the existing grouped-decode fork draft #8, not a second copy of it.
This layer does not select the operator in any model runtime or enable a flag.

Review order: #8 (prerequisite) -> #10 (kernel) -> #12 (runtime) -> #11 (3B loads).

  • Add a model-specific SM70 Qwen3.8 TP4/native-g32/E512/top-k10 AWQ M=1
    W13/SwiGLU and W2/router-reduction operator using the existing QPN dataflow.
  • Consume current prepared banks and caller-owned output/intermediate buffers.
    Support scalar 3-byte and existing 4-byte metadata, without a new weight copy.
  • Preserve FP16 scale/bias dequantization, gate/up/SwiGLU boundaries,
    per-route FP16 materialization and original router accumulation order.
    CTA-local FP32 reduction order changes; legacy is not a mathematical oracle.
  • Validate shape, dtype, device, contiguity, alignment and non-overlap. Invalid
    expert IDs contribute zero and duplicate IDs keep separate route weights.
  • Register the native operator and fake implementation; no sidecar loader.
    The retained LICENSE.v100-skinny identifies the QPN dataflow provenance.

The subsequent layers are default-off runtime admission and independent
cooperative 3-byte W13 metadata loading. Four-byte full-model performance is
attributable to operator plus runtime, not to the third layer.

Scope and duplicate check

Open-PR searches in this fork, 1CatAI/1Cat-vLLM and vllm-project/vllm for
AWQ/QPN/SM70 found no competing native AWQ M1 operator. The QPN search hits
1Cat 1CatAI#349 (PP2/TP4 FP8 tactic stabilization) and 1CatAI#509 (conversion-cache release)
are different work. Merged 1CatAI#477 separates prefill/decode compilation; this
operator adds AWQ expert computation, not a replacement graph framework.
The separate QSA page4 fix used in both full-model validation arms is not
included here. There is no corresponding vllm-project/vllm issue submission.

Test plan and results

Changed-file pre-commit run --files ... and git diff --check: passed.
The three extra parenthesizations of existing NVFP4 fake lambdas in
_sm70_ops.py are required by the pinned formatter, with no semantic change.

On the native SM70 scalar build, the following portable test invocation passed
2/2 (the repository test directory was mounted as /tests):

/evidence/test-venv/bin/python -m pytest -q --confcutdir=/tests \
  /tests/test_sm70_awq_qpn_m1.py

It constructs both prepared layouts independently, checks one-hot boundary
reads and an FP64 W2 reference, changing graph inputs, invalid/duplicate IDs,
alias/alignment/dtype rejection, and the fake op. CPU skips are not GPU passes.

The archived real-checkpoint harness was also run:

/evidence/test-venv/bin/python /native-tools/probe_native.py \
  --manifest /output/native-manifest.json --output /output/probe
  • 256 native/prototype/dynamic-graph exact comparisons across both metadata
    layouts and all four logical TP partitions.
  • 72 independent FP64 stage checks, no retained rounding-bound violations;
    48 invalid argument rejections; invalid routes zero; fullgraph eager compile.
  • One tiny subnormal case remains above the old relative-only smoke threshold
    (relative L2 0.00201759, max absolute 5.96046e-8). It is recorded, not renamed
    a pass under that old rule. The independent reference includes an explicit
    FP16 subnormal rounding floor; it is not an assertion of token identity.
  • Scalar native core SHA-256:
    109e8e68852c701a638f05e5574f64f9d040f41497a6696a6320457285f0b434.
  • Evidence: qwen38-c1-qpn-awq-20260905/layer-gpu-scalar-r1, exit0/OOMfalse.

These checks establish the tested operator contract, not general full-model
quality. Runtime evidence and observed task-level changes belong to layer 2.

Review status

AI assistance: OpenAI Codex assisted with implementation, analysis, tests and
documentation. Leon directed the scope and requested submission to this fork.
This is a fork review draft; final human line-by-line review and test
ownership are not claimed complete. No upstream submission, default route
change or production deployment is made here.

Add the typed inference operator, prepared-bank decoder, FP32 reduction kernels and standalone GPU tests without changing model dispatch. Keep 3B scalar loading here so cooperative metadata access can be reviewed separately.

Assisted-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant