Glm5Next (GLM-5.3-Flash) on gfx1030: load + hybrid dual cache + AWQ W4A16 HIP + KDA/DSA kernels (UNC-33..37, Later) by BlivionIaG · Pull Request #2 · opengfx1030/vllm-rdna

BlivionIaG · 2026-09-04T02:12:36Z

GLM-5.3-Flash (Glm5Next) inference on gfx1030 — UNC-33/34/35/36/37 (Later work, not dest)

First cut of zai-org/GLM-5.3-Flash (architecture class Glm5Next) support for the RDNA2
extras tree, on the later/glm53-flash-awq side branch. This PR targets the side branch,
not rdna_extras — per the work order, rdna_extras/rdna2_extras are untouched and no
dest extras are bumped.

Written blind. No GPU/toolchain was available to the author; everything here is
static-checked (py_compile, brace/paren balance, signature cross-checks) but
runtime-unverified. Torch/generic paths are the defaults; every HIP fast path is
env-gated OFF until it is runtime-verified on 4× V620. No tok/s claims in this PR.

Architecture as implemented (per real `config.json` + safetensors map + HF modeling)

45 layers, 3:1 hybrid — 34 KDA linear-attention layers + 11 DSA layers (3,7,…,43),
layer_types = linear_attention / deepseek_sparse_attention.
KDA = Kimi Delta Attention: 64 heads × 128, causal conv4 (separate q/k/v conv
weights), A_log/dt_bias, lower-bound sigmoid gate (-5.0), per-(head, k-dim)
log-decay, l2norm q/k, beta-sigmoid delta rule, gated RMSNorm output. State
[slots, H, 128, 128] fp32 → MambaSpec cache group.
DSA = MLA-NoPE: q_lora 1536 → q_b → 64×256; kv_lora 512 latent cached per token,
kv_b_proj expansion at attend time; qk/v 256, no RoPE anywhere in the LM;
kpool indexer (32h × 128, topk 2048, kpool 4, always_select_tail, pool compression
softmax(gate+ape), tail ≤ 3 → width 2051). Two AttentionSpec cache groups per DSA
layer: latent (512) + packed indexer rows (257 = k|gate|valid).
Dual cache split (KDA state pages ≠ DSA paged KV) is structural: mamba state group
vs two paged attention groups; chunked prefill / prefix-cache boundaries follow the
stock hybrid KV machinery.
MoE (layers 3–44): 288 routed + 1 shared, top-8, sigmoid scoring +
e_score_correction_bias, norm_topk_prob, routed_scaling_factor 2.5,
swiglu_limit clamping — stock FusedMoEFactory.
mHC (UNC-37, torch fallback = success): 4 residual streams carried through the LM
([num_tokens, 4*hidden]), Sinkhorn-Knopp comb projection (20 iters), pre/post/comb
per attention/FFN site; checkpoint flat names hc_attn_*/hc_ffn_* remapped to
attn_hc.{fn,base,scale}/ffn_hc.*.
MTP: DeepSeek-style (Glm5NextMTPModel): enorm/hnorm/eh_proj + one DSA+MoE
decoder block loaded from checkpoint layer 45. Note: layer 45 carries no hc_*
tensors in the safetensors map (ground truth over the design note) → the MTP block runs
plain residuals (use_hc=False).
Vision: lean-but-loadable ViT tower + minimal multimodal processor (Qwen2-VL-style
fields) so weights load and --language-model-only text serving works; mm input path is
not a deliverable of this branch.

Commits / UNC mapping

Commit	UNC	Content
`feat(glm5_next): …config classes, registry entries, env gates`	33	configs + `_CONFIG_REGISTRY` + `layer_types` patch + env gates
`feat(glm5_next): KDA linear-attention layer …`	33/35	KDA layer, torch decode + chunk-64 prefill fallback, capture-safe
`feat(glm5_next): DSA MLA-NoPE layer + GLM5_DSA_ATTN …`	33/36	DSA layer + kpool indexer math + torch-scan backend + enum
`feat(glm5_next): Glm5Next model, mHC, …MTP, vision`	33/37	model, mHC, MoE, IsHybrid dual-cache specs, MTP, vision, registry
`feat(awq): RDNA2 HIP W4A16 dispatch for AWQ routed MoE`	34	AWQ INT4 routed experts → `moe_gptq_gemm_rdna2` via load-time repack
`feat(rocm): GLM-5.3 DSA indexer + MLA-NoPE decode HIP kernels`	36	`glm5_dsa_indexer_rdna2`, `glm5_dsa_mla_decode_rdna2`
`feat(rocm): GLM-5.3 KDA decode + chunked-prefill HIP kernels`	35	`glm5_kda_decode_rdna2` + 5-kernel prefill chain
`feat(rocm): wire GLM-5.3 KDA/DSA ops into build + bindings`	35/36	ops.h, `TORCH_LIBRARY` schemas, CMake gfx1030 list, `_custom_ops`

Runtime gates (all default OFF)

Env	Effect
`VLLM_FORCE_RDNA2_W4A16_HIP=1`	AWQ routed MoE → `moe_gptq_gemm_rdna2` (gfx10x + op present + 4-bit + gs≥32); otherwise Marlin/WNA16 unchanged
`VLLM_GLM5_KDA_HIP=1`	KDA decode (+prefill chain available) on HIP; torch fallback otherwise
`VLLM_GLM5_DSA_HIP=1`	DSA indexer/MLA-decode on HIP; torch scan otherwise

Prohibitions honored

No gdn_decode_rdna2 / gdn_prefill_* (Qwen 16/48×128), no sparse_mla_rdna2
(V4 fp8_ds_mla 448+64, 4 heads/CTA), no indexer_paged_mqa_rdna2 (V4 C4A FP8), no Kimi
fused_kda_decode_kernel_rocm.cu tiles — all new GLM tiles.
RDNA_ATTN stays UNIFORM_SINGLE_TOKEN_DECODE; the GLM5DSA backend likewise refuses q>1
(MTP verify) and falls back rather than faking inside the kernel.
supported_kv_cache_dtypes stays auto/float16; no fused INT8 KV dispatch.
Custom ops use Tensor! out-args, caller-zeroed buffers; no blocking D2H / .item()
under capture (prefill scan refuses capture explicitly).
Marlin / AutoRound / EXL3 / mxfp4 / official-FP8 paths untouched; no dest extras bump.

Known limitations / follow-ups (for the review+run phase)

HIP kernels are first cuts — math fidelity to the HF reference is the design bar,
but none has seen a compiler or a GPU. Expect fixes from the first build/run on .176.
AWQ zero-offset convention: the method synthesizes GPTQv1-style zeros
(awq_zero - 1 for the kernel's +1 decode) — verify against the first real AWQ
checkpoint produced for this model (expected gs=128).
DSA HIP decode checks 64 heads → TP>1 falls back to the torch scan automatically
(exception caught); a TP-aware variant is a follow-up.
KDA spec-decode (MTP verify) raises in the torch fallback by design
(spec_sequence_masks unsupported) — serve without MTP until that lands.
mm processor is minimal (token-ID-space placeholders); image/video serving is out of
scope here.
DSA latent-cache + expand-at-attend keeps memory at 1 KB/token; the expanded-K/V
gather is the scan-path cost — the HIP path gathers caller-side as well.

Suggested first-run checklist (later, on `.176`)

# torch path first (all gates OFF), text-only:
python -m vllm.entrypoints.cli.main serve <glm53-checkpoint> \
  --dtype float16 --language-model-only --skip-mm-profiling --trust-remote-code \
  --max-model-len 8192 --max-num-seqs 4
# then per gate: VLLM_FORCE_RDNA2_W4A16_HIP=1 (AWQ ckpt), VLLM_GLM5_KDA_HIP=1, VLLM_GLM5_DSA_HIP=1

Base: later/glm53-flash-awq @ rdna_extras tip 9ba5d5f46 (supersedes the pinned
38595867 — includes the fa_rdna2 sliding-window fix and the GDN-prefill-default-off fix).

UNC-33 foundation for zai-org/GLM-5.3-Flash (Glm5Next) on the later/glm53-flash-awq side branch. - What: Glm5NextConfig/Glm5NextTextConfig/Glm5NextVisionConfig (checkpoint nests KDA geometry in linear_attn_config; the config derives flat fields), registered in configs/__init__.py and the _CONFIG_REGISTRY; layer_types validation patched for glm5_next (linear_attention + deepseek_sparse_attention); env gates declared: VLLM_FORCE_RDNA2_W4A16_HIP (UNC-34), VLLM_GLM5_KDA_HIP / VLLM_GLM5_DSA_HIP (UNC-35/36, first-cut kernels default OFF). - Why: the card must instantiate on ROCm gfx1030 before any kernel work lands; envs land first so every later commit is self-consistent. - Testing: written blind (no GPU here); python -m py_compile clean.

…33/35) - What: Glm5NextKDAAttention(GatedDeltaNetAttention) for the 34 KDA layers (64 heads x 128, causal conv4, A_log/dt_bias, separate q/k/v projections + q/k/v conv weights, f_a/f_b forget gate, b_proj beta, g_a/g_b output gate, gated RMSNorm). Torch fallback is the DEFAULT path: fused conv-shift decode recurrence and chunk-64 chunked prefill, exact math per chunk_kimi_delta_attention (per-(head,dim) log-decay, l2norm q/k, beta sigmoid, state [slots,H,128,128] fp32). Uses the stock GDN backend for metadata; cudagraph-capture-safe (no .item()/D2H; prefill refuses capture rather than faking). HIP hook (glm5_kda_decode_rdna2) is env-gated behind VLLM_GLM5_KDA_HIP and falls back on any mismatch. - Why: KDA state pages must split from DSA paged KV; torch/generic scan is the success bar until HIP is runtime-verified. - Testing: written blind; python -m py_compile clean.

…-33/36) - What: Glm5NextDSAAttention for the 11 DSA layers (q_lora 1536, kv_lora 512, qk/v 256, NO RoPE) with the kpool indexer (32h x 128, topk 2048, kpool 4, always_select_tail, exact pool-compression math incl. softmax(gate+ape) and append_visible_tail). Dual cache groups via two AttentionLayerBase sublayers: latent cache (FullAttentionSpec num_kv_heads=1 head_size=512 head_size_v=0) and packed indexer cache (head_size=257: k|gate|valid). New GLM5_DSA_ATTN backend does the paged torch scan (gather latents, kv_b_proj expand at attend time, softmax over selected slots); UNIFORM_SINGLE_TOKEN_DECODE only — q>1 (MTP verify) falls back, never faked. HIP hooks env-gated behind VLLM_GLM5_DSA_HIP with fp16 staging; torch scan is the default. - Why: DSA must not execute sparse_mla_rdna2 (V4 fp8_ds_mla tile); new tiles only, torch scan until HIP lands. - Testing: written blind; python -m py_compile clean.

…(UNC-33/37) - What: full Glm5NextForConditionalGeneration for GLM-5.3-Flash: 45-layer 3:1 hybrid decoder (KDA layers via layers.mamba.gdn.glm5_kda_linear_attn, DSA layers via layers.attention.glm5_dsa_attention), mHC torch fallback (hc_mult=4 streams carried through the LM, Sinkhorn-Knopp comb projection, hc_attn_*/hc_ffn_* checkpoint renames), dense MLP for layers 0-2 + FusedMoEFactory sigmoid router with e_score_correction_bias / norm_topk_prob / routed_scaling_factor / swiglu_limit for layers 3-44, IsHybrid/HasInnerState classmethods (kda_state_shape/dtype/copy_func) for the dual KDA-state vs DSA-KV cache split, lean vision tower + minimal multimodal processor for weight loadability with --language-model-only text serving, and DeepSeek-style MTP (Glm5NextMTPModel: enorm/hnorm/eh_proj + one DSA+MoE block from checkpoint layer 45; the MTP block runs plain residuals because layer 45 carries no hc_* tensors in the safetensors map — ground truth over the design note). - Why: UNC-33 load path + UNC-37 mHC torch fallback; MTP/visual ride-along per the card. - Testing: written blind; python -m py_compile clean.

- What: AutoAWQRDNA2MoEMethod loads AWQ INT4 routed-expert packs and converts them at process_weights_after_loading into the moe_gptq_gemm_rdna2 kernel contract (per-expert AWQ->standard nibble reorder + gptq_shuffle, GPTQv1 zero-offset synthesis, fp16 scales), then runs the two-call w1/w2 fused-MoE flow with moe_align_block_size (block_size_m 1/4). Hooked from AutoAWQConfig.get_quant_method behind envs.VLLM_FORCE_RDNA2_W4A16_HIP + on_gfx10x() + op availability + 4-bit/group_size>=32; without the gate the existing Marlin/WNA16 paths are byte-for-byte unchanged. - Why: GLM-5.3-Flash routed experts (288, top-8) AWQ INT4 must hit the extras HIP fdot2 kernels, not Triton; leftover BF16 parts stay dense; Marlin/AutoRound/EXL3/mxfp4/official-FP8 untouched. - Testing: written blind; python -m py_compile clean.

- What: first-cut gfx1030 kernels (wave32, fp16/fp32-accum, fdot2 where dots are plain, no launch_bounds/occupancy pins): glm5_dsa_indexer_rdna2 — first-key pre-pass + per-(query, pool) kpool-4 compression (softmax(gate+ape) with fp16 rounding parity), q.pool_key logits + relu(128^-0.5) + weighted head-sum, pool indices/validity out; topk stays caller-side. glm5_dsa_mla_decode_rdna2 — one wave per (batch, head), q resident in VGPRs (256 fp16 = 8/lane), LDS-cached scores, two-pass fp32 softmax, PV accumulate; supports gathered [B,S,256] or pre-expanded [B,S,64*256] k/v rows. - Why: new tiles — sparse_mla_rdna2 (V4 fp8_ds_mla 448+64) and indexer_paged_mqa_rdna2 (V4 C4A FP8) are banned for GLM; env-gated OFF (VLLM_GLM5_DSA_HIP) until runtime-verified, torch scan is the default. - Testing: written blind; no hipcc here; braces/parens balanced, launcher TORCH_CHECKs mirror the contract.

- What: first-cut gfx1030 kernels for GLM KDA (64h x 128, conv4, lower-bound gate) — NOT Qwen GDN tiles (16/48 heads) and NOT the Kimi fused_kda_decode tiles; GLM packing per DESIGN.md contract: glm5_kda_decode_rdna2 — one 1024-thread workgroup per (batch, head): conv shift + silu (fp16 round-trip parity) -> l2norm q,k -> per-(head,dim) lower-bound gate -> delta recurrence on ssm_state[slot,h,k,v] fp32 -> gated RMSNorm. glm5_kda_prefill_{prep,kkt,solve_wy,delta_h,o}_rdna2 — chunk-64 varlen chain (cu_seqlens/chunk_indices) reproducing chunk_kimi_delta_attention with per-(head,dim) g; kkt emits the sign-bridged P[i,j]=beta_i*sum_d k_i[d]k_j[d]exp(g_i[d]-g_j[d]) so the GDN-style solve computes (I+P)^-1 = the HF WY transform; u=T@(v*beta), w=T@(k*beta*exp(g)); delta_h per-dim g_last state pass; o applies q scale and emits raw o (gated norm caller-side). - Why: UNC-35 new HIP with graph-safe contracts (zeroed out-args, Tensor! writes, no D2H); env-gated OFF (VLLM_GLM5_KDA_HIP) until runtime-verified, torch fallback is the default. - Testing: written blind; no hipcc here; braces/parens balanced.

- What: ops.h declarations, TORCH_LIBRARY schemas (Tensor! out-args, Tensor? optionals for the delta_h chain) under #ifdef VLLM_ROCM_GFX1030, gfx1030 CMake source list entries and _custom_ops.py wrappers for: glm5_kda_decode_rdna2, glm5_kda_prefill_{prep,kkt,solve_wy,delta_h,o}_rdna2, glm5_dsa_indexer_rdna2, glm5_dsa_mla_decode_rdna2. - Why: the 8 new GLM-5.3 kernels must register into _rocm_C for the env-gated hooks in the KDA layer / GLM5DSA backend to find them. - Testing: written blind; schema/launcher argument order cross- checked against the .cu host launchers.

…page fault Root cause of Bug #2 (HIP kernel GPU memory fault): vLLM pre-allocates the activation buffer to max_num_batched_tokens * num_sequences (e.g. 2048 * 4 = 8192 for a 4-prompt probe), but the actual batch is much smaller (1024 here). Even though the storage is properly sized for the full shape, the padding pages are not committed (torch.empty maps virtual pages but doesn't commit physical pages until written). The HIP kernel reads from uncommitted pages and faults at a bogus GPU address. The slice fix in qwen3_next.py doesn't work because positions is also padded to 8192 (full_num_tokens = positions.shape[-1] = 8192). The defensive storage-vs-shape check in apply() doesn't trigger because the storage IS properly sized for the full shape. Fix: hard-cap M to max_num_batched_tokens (default 2048, overridable via VLLM_EXL3_MAX_BATCH) in apply() before any kernel call. This avoids the uncommitted padding pages. - What: 3-line hard cap + comment in exl3.py apply() - Why: fix Bug #2 — HIP kernel fault on uncommitted GPU pages - Tested: pending probe run Co-authored-by: Agent

vLLM allocates the activation buffer with torch.empty, which maps virtual pages but doesn't commit physical pages. The HIP kernel reads from these pages and faults at a bogus GPU address. The hard cap slices x to max_num_batched_tokens, but the first _max_batched rows also have uncommitted pages because the buffer was never written to. Fix: add x = (x * 1.0).contiguous() after the hard cap. The * 1.0 operation allocates a new tensor and writes the result into it, which commits the physical pages. The .contiguous() ensures the result is contiguous. - What: 1-line fix + comment in exl3.py apply() - Why: fix Bug #2 — HIP kernel fault on uncommitted GPU pages - Tested: pending probe run Co-authored-by: Agent

…page fault Root cause of Bug #2 (HIP kernel GPU memory fault): vLLM pre-allocates the activation buffer to max_num_batched_tokens * num_sequences (e.g. 2048 * 4 = 8192 for a 4-prompt probe), but the actual batch is much smaller (1024 here). Even though the storage is properly sized for the full shape, the padding pages are not committed (torch.empty maps virtual pages but doesn't commit physical pages until written). The HIP kernel reads from uncommitted pages and faults at a bogus GPU address. The slice fix in qwen3_next.py doesn't work because positions is also padded to 8192 (full_num_tokens = positions.shape[-1] = 8192). The defensive storage-vs-shape check in apply() doesn't trigger because the storage IS properly sized for the full shape. Fix: hard-cap M to max_num_batched_tokens (default 2048, overridable via VLLM_EXL3_MAX_BATCH) in apply() before any kernel call. This avoids the uncommitted padding pages. - What: 3-line hard cap + comment in exl3.py apply() - Why: fix Bug #2 — HIP kernel fault on uncommitted GPU pages - Tested: pending probe run

vLLM allocates the activation buffer with torch.empty, which maps virtual pages but doesn't commit physical pages. The HIP kernel reads from these pages and faults at a bogus GPU address. The hard cap slices x to max_num_batched_tokens, but the first _max_batched rows also have uncommitted pages because the buffer was never written to. Fix: add x = (x * 1.0).contiguous() after the hard cap. The * 1.0 operation allocates a new tensor and writes the result into it, which commits the physical pages. The .contiguous() ensures the result is contiguous. - What: 1-line fix + comment in exl3.py apply() - Why: fix Bug #2 — HIP kernel fault on uncommitted GPU pages - Tested: pending probe run

BlivionIaG added 8 commits September 4, 2026 04:09

GeorgeMA-Strong mentioned this pull request Sep 16, 2026

fix(rocm): restore Intel CPU PLE and V620 MTP2 startup #12

Closed

wsantos mentioned this pull request Sep 24, 2026

Fixed Piecewise #20

Merged

BlivionIaG mentioned this pull request Sep 27, 2026

perf(rocm): FA-RDNA2 causal tile skipping, in-place output, parallel GQA softmax #28

Merged

2 of 4 tasks

Navigation Menu

Repository navigation

Glm5Next (GLM-5.3-Flash) on gfx1030: load + hybrid dual cache + AWQ W4A16 HIP + KDA/DSA kernels (UNC-33..37, Later) - #2

Glm5Next (GLM-5.3-Flash) on gfx1030: load + hybrid dual cache + AWQ W4A16 HIP + KDA/DSA kernels (UNC-33..37, Later)#2
BlivionIaG wants to merge 8 commits into
later/glm53-flash-awqfrom
later/glm53-flash-awq-glm5next

BlivionIaG commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

BlivionIaG commented Sep 4, 2026

GLM-5.3-Flash (Glm5Next) inference on gfx1030 — UNC-33/34/35/36/37 (Later work, not dest)

Architecture as implemented (per real config.json + safetensors map + HF modeling)

Commits / UNC mapping

Runtime gates (all default OFF)

Prohibitions honored

Known limitations / follow-ups (for the review+run phase)

Suggested first-run checklist (later, on .176)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Architecture as implemented (per real `config.json` + safetensors map + HF modeling)

Suggested first-run checklist (later, on `.176`)