Conversation
mergify
Bot
added
quantization
glm
rocm
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
…orm fp8 dtype
fwht128_quant_fp8 hardcodes float8_e4m3fn for the indexer query and a 448
absmax clamp inside the Triton kernel. The function is called
unconditionally from Glm5NextAttention.forward, including on ROCm, where
the query flows into rocm_fp8_mqa_logits:
- On gfx942 the AITER flydsl fp8_mqa_logits kernel takes the
convert_q_fn path for a float8_e4m3fn query and crashes at kernel
emission:
AttributeError: 'int' object has no attribute '_CAPIPtr'
(aiter/ops/flydsl/kernels/mqa_logits/fp8_mqa_logits.py, _fn_to_fnuz_i64)
Any GLM-5.3-Flash request with chunked prefill (e.g. 40K-token prompts)
kills the worker; the engine crash-loops.
- Even where the FN path would compile, 448 is the e4m3fn max; the
e4m3fnuz max differs, so rows whose absmax lands above the fnuz max
would encode NaN/inf.
The K cache side already quantizes with current_platform.fp8_dtype()
(indexer_k_quant_and_cache_triton IS_FNUZ; the AMD kpool ops module does
the same via its FP8_DTYPE constant) — the query side is the odd one out.
Fix: derive the output dtype from current_platform.fp8_dtype() and pass
the matching max as an FP8_MAX constexpr to the kernel. CUDA behavior is
unchanged (fp8_dtype() is e4m3fn there); the e4m3fnuz path gets correct
dtype and clamp bounds.
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: HzTTT <767067749@qq.com>
Note on the max convention: the K-side quant kernel (indexer_k_quant_and_cache
IS_FNUZ branch) clamps to the deepgemm-conventional 224; this Q-side fix
follows the sibling AMD ops module, which derives torch.finfo(fp8_dtype).max
(240 on current torch for e4m3fnuz). Each side applies its own scale, so the
two constants are independent; matching the AMD module keeps the two
fwht call sites identical.
Signed-off-by: HzTTT <767067749@qq.com>
1e396ee to
38548e0
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Hi @tjtanaa @dllehr-amd — another ROCm GLM-5.3-Flash fix: |
|
I'm encountering the same problem and wanted to ask if you have a pre-released Docker image. I'm using GPUStack for deployment, but I don't know how to package a vllm-openai-rocm image from scratch. |
2 participants
Purpose
fwht128_quant_fp8hardcodesfloat8_e4m3fnfor the GLM-5.3-Flash indexer query and a 448 absmax clamp inside the Triton kernel. The function is called unconditionally fromGlm5NextAttention.forward— including on ROCm, where the query flows intorocm_fp8_mqa_logits:fp8_mqa_logitskernel takes theconvert_q_fnpath for afloat8_e4m3fnquery and crashes at kernel emission:Any request with chunked prefill (e.g. a 40K-token prompt) kills the worker and the engine crash-loops. Reproduced standalone:
flydsl_fp8_mqa_logitswith an FN-dtype query crashes at compile on gfx942.The K-cache side already quantizes with
current_platform.fp8_dtype()(indexer_k_quant_and_cache_tritonIS_FNUZ; the AMD kpool ops module does the same via itsFP8_DTYPEconstant) — the query side is the odd one out.Changes
fwht128_quant_fp8allocates the output withcurrent_platform.fp8_dtype().FP8_MAXconstexpr instead of the hardcoded 448.fp8_dtype()is e4m3fn there).Duplicate-work check: open GLM-5.3-Flash PRs (#55358 refactor, #56176 MXFP4, #55423 DFlash2) do not touch the fp8 dtype or clamp. File-level overlap with #55358 is a single import line — that PR relocates
sparse_attn_indexer_kpool(the layer), while this patch touches onlyvllm/models/glm5next/nvidia/ops/kpool_compress.py(the ops module #55358 moves an import of, not the file itself).Test Plan
Tests 1-2 are CPU-only (dtype allocation spy + kernel source contract). Test 3 (numeric Hadamard comparison) runs where Triton is active.
Test Result
CPU-only tests: 2 passed locally; negative control (fix stashed):
test_fwht_kernel_clamp_uses_parameterized_maxfails on the hardcoded 448.On 8× MI308X (gfx942,
vllm/vllm-openai-rocm:nightly+ this patch), all three pass:The numeric test asserts (a) output dtype = platform fp8, (b) scales exactly match the ue8m0 power-of-two formula, (c) dequantized values recover the Hadamard reference within the fp8 relative-step bound (≤ 0.07), (d) no NaN and range ≤ fp8 max — a pre-fix e4m3fn allocation on ROCm fails (a) and a pre-fix 448 clamp violates (d) for fnuz.
End-to-end on 8× MI308X, GLM-5.3-Flash TP8 + MTP-3: pre-fix, every ≥16K-token request crashes the worker; post-fix, 40K/65K/128K long-context probes complete cleanly with correct output.
AI assistance was used for investigation, implementation, and this description.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.