Conversation
Add the typed inference operator, prepared-bank decoder, FP32 reduction kernels and standalone GPU tests without changing model dispatch. Keep 3B scalar loading here so cooperative metadata access can be reviewed separately. Assisted-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
This was referenced Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose and stack
Layer 1/3: native operator, binding, independent GPU tests and design contract.
Base is the existing grouped-decode fork draft #8, not a second copy of it.
This layer does not select the operator in any model runtime or enable a flag.
Review order: #8 (prerequisite) -> #10 (kernel) -> #12 (runtime) -> #11 (3B loads).
W13/SwiGLU and W2/router-reduction operator using the existing QPN dataflow.
Support scalar 3-byte and existing 4-byte metadata, without a new weight copy.
per-route FP16 materialization and original router accumulation order.
CTA-local FP32 reduction order changes; legacy is not a mathematical oracle.
expert IDs contribute zero and duplicate IDs keep separate route weights.
The retained
LICENSE.v100-skinnyidentifies the QPN dataflow provenance.The subsequent layers are default-off runtime admission and independent
cooperative 3-byte W13 metadata loading. Four-byte full-model performance is
attributable to operator plus runtime, not to the third layer.
Scope and duplicate check
Open-PR searches in this fork,
1CatAI/1Cat-vLLMandvllm-project/vllmforAWQ/QPN/SM70 found no competing native AWQ M1 operator. The QPN search hits
1Cat 1CatAI#349 (PP2/TP4 FP8 tactic stabilization) and 1CatAI#509 (conversion-cache release)
are different work. Merged 1CatAI#477 separates prefill/decode compilation; this
operator adds AWQ expert computation, not a replacement graph framework.
The separate QSA page4 fix used in both full-model validation arms is not
included here. There is no corresponding
vllm-project/vllmissue submission.Test plan and results
Changed-file
pre-commit run --files ...andgit diff --check: passed.The three extra parenthesizations of existing NVFP4 fake lambdas in
_sm70_ops.pyare required by the pinned formatter, with no semantic change.On the native SM70 scalar build, the following portable test invocation passed
2/2 (the repository test directory was mounted as
/tests):It constructs both prepared layouts independently, checks one-hot boundary
reads and an FP64 W2 reference, changing graph inputs, invalid/duplicate IDs,
alias/alignment/dtype rejection, and the fake op. CPU skips are not GPU passes.
The archived real-checkpoint harness was also run:
layouts and all four logical TP partitions.
48 invalid argument rejections; invalid routes zero; fullgraph eager compile.
(relative L2 0.00201759, max absolute 5.96046e-8). It is recorded, not renamed
a pass under that old rule. The independent reference includes an explicit
FP16 subnormal rounding floor; it is not an assertion of token identity.
109e8e68852c701a638f05e5574f64f9d040f41497a6696a6320457285f0b434.qwen38-c1-qpn-awq-20260905/layer-gpu-scalar-r1, exit0/OOMfalse.These checks establish the tested operator contract, not general full-model
quality. Runtime evidence and observed task-level changes belong to layer 2.
Review status
AI assistance: OpenAI Codex assisted with implementation, analysis, tests and
documentation. Leon directed the scope and requested submission to this fork.
This is a fork review draft; final human line-by-line review and test
ownership are not claimed complete. No upstream submission, default route
change or production deployment is made here.