Conversation
…<= 16 For a 4-bit code n, (0x4300|n) as bf16 is exactly 128+n and (0x6400|n) as fp16 is exactly 1024+n -- bit-exact dequant with no int->float convert. On CDNA2 the shipped .to(a.dtype) is a software round-to-nearest-even plus NaN handling (76 of 244 inner-loop VALU ops), and bf16 VALU math does not exist, so the per-weight scale multiply is hoisted to a per-output rank-1 correction: s_n * (sum a_k v_k - (magic+ZP) sum a_k). Gated on BLOCK_M <= 16 (structural crossover: correction grows with BLOCK_M, saving does not), symmetric quant only, ROCm gfx90a only. Measured 2x MI210: gate_up 1.56x/1.52x at M=1/8; regressions at M=32 (q_proj 0.81x .. gate_up 0.96x) are why the gate exists.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
1 participant
Purpose — why this change exists
The W4A16 Triton kernel dequantizes with
(b - z).to(a.dtype) * scales. OnCDNA2 (gfx90a/MI210/MI250) that
.to()is not one instruction: there is nohardware bfloat16 convert, so it expands to a software round-to-nearest-even
plus NaN-handling sequence — 76 of the 244 inner-loop VALU ops. And
gfx90a has no bf16 VALU arithmetic at all (bf16 is an MFMA operand type
only), so the per-weight
* scalesforces a round trip out to fp32 andback. Together those two are the decode bottleneck.
The change
Magic-bias dequant: for a 4-bit code
nin 0..15,(0x4300 | n)reinterpreted as bfloat16 is exactly 128+n, and
(0x6400 | n)asfloat16 is exactly 1024+n — at those exponents the mantissa step is 1
and the largest value still fits, so nothing rounds. Write the code straight
into the float mantissa; the int→float convert disappears. Verified
bit-exact against a float64 CPU oracle.
Scale hoist: move the scale out of the per-weight loop entirely, as
exact algebra (valid because one BLOCK_K tile lies inside a single scale
group, asserted at the launcher):
Scale and zero-bias become per-output work (BLOCK_M×BLOCK_N per tile)
instead of per-weight (BLOCK_K×BLOCK_N per tile).
The gate is structural, not tuned:
_use_magic_biaskeys onBLOCK_M <= 16. The correction costs O(BLOCK_M×BLOCK_N) per K-tile whilethe removed work is O(BLOCK_K×BLOCK_N) and independent of BLOCK_M —
benefit/cost degrades monotonically, so a crossover is guaranteed. Keyed on
BLOCK_M rather than M so a future tile-ladder change cannot silently break
the bound. Symmetric quant only (asymmetric generalizes but is unverified);
ROCm gfx90a only (MI300/NVIDIA have hardware converts — unmeasured there,
deliberately ungated-for).
Measured (2× AMD MI210, production entry point)
The M=32 regressions are exactly why the gate exists.