[CI][Test] GLM-5.3-Flash layer-pruned checkpoint tests (e2e, KV offload, NIXL PD) by JaredforReal · Pull Request #57537 · vllm-project/vllm

JaredforReal · 2026-09-18T09:04:44Z

Purpose

Give the GLM-5.3-Flash serving stack (hybrid KDA + DeepSeek-sparse-MLA with the kpool indexer / tail
scratch group, fp8 MoE, mHC, MTP, PD disaggregation, KV offload) CI coverage that does not need a
4-GPU node holding the 328 GB checkpoint.

The asset is a layer-pruned real checkpoint, JaredforReal/GLM-5.3-Flash-4L: original layers 0-3
(one 3x KDA + 1x DSA-MLA period, dense MLP on 0-2, MoE on 3) plus the MTP layer, with every width
unchanged (288 routed experts, 64 KDA heads, MLA/indexer dims, vision tower). Only whole layers were
removed, so kernel selection and the KV cache layout match the 300B model: the same
{MLA+indexer}, {kpool tail} and Mamba groups aliasing the MLA slot, and the same auto block size
(4352 / 2176 / 1152 tokens at TP1 / TP2 / TP4). 18.5 GiB, loads in ~8 s. Derivation and the exact config
edits are documented in the model card.

New tests (SM90/SM100 only: the DeepGEMM paged-MQA indexer and FlashMLA/FlashInfer sparse MLA do not run on L4):

test	GPUs	guards
`tests/models/glm5next/test_pruned_e2e.py`	1	KV groups = {MLA + indexer}, {kpool tail}, 3 Mamba groups aliasing the MLA tensor; greedy reproducibility across a block boundary with a prompt longer than `index_topk`; MTP drafter loads and agrees with target-only greedy. A strict xfail documents the `fused_recurrent_kda` gridDim.z overflow at 64 heads x 1024 seqs (#56973, fixed by #56974; the marker must be dropped once that lands).; prefill checkpoint pages: a prompt crossing a block boundary leaves, in every Mamba group, a cached checkpoint page equal to a cold prefill ending at that boundary (relative error <= 0.03 observed, >= 1 when a group's checkpoint goes to another group's page, the #59536 regression that the greedy repeat test does not notice)
`tests/v1/attention/test_gdn_metadata_builder.py::test_update_block_table_matches_fresh_build`	1	`update_block_table()` for a sibling Mamba group equals a fresh `build()` on that group's block table in every metadata field (recursively over nested dataclasses), so a per-group field added later cannot stay pointed at the source group; fails before #59536 on `checkpoint.state_indices`
`tests/v1/kv_offload/test_glm5next_pruned_e2e.py`	1	SimpleCPUOffloadConnector / OffloadingConnector round trip through `reset_prefix_cache`, asserting CPU hits and identical tokens. Fails on main today because the kpool tail group (block 4, not prefix-cacheable) is not skipped (`resolve_block_hashes` assert / `tokens_per_block=4 not divisible by tokens_per_hash`); SimpleCPUOffload passes with #56810.
`tests/v1/kv_connector/nixl_integration/run_glm5next_pd_test.sh` + `test_hybrid_pd_consistency.py`	2	NixlConnector 1P1D with `VLLM_SSM_CONV_STATE_LAYOUT=DS`; reuses `test_mamba_prefix_cache.py` (its ~9000-token prompt spans two 4352-token blocks) and checks P-only vs P->D top-1 logprobs within a tolerance. Hybrid PD recomputes the last prompt token on D (`_prefill_backoff`), so exact text equality as in `test_edge_cases.py` is not the right assertion for this model.

Buildkite: single-GPU jobs on h200_35gb (models_language.yaml, lm_eval.yaml), the PD job on 2x h100
(disaggregated.yaml); MIG slices cannot do cross-device NIXL and the existing PD sweeps run on L4. The
Pruned E2E step also depends on vllm/v1/attention/backends/gdn_attn.py, vllm/model_executor/layers/mamba/,
vllm/v1/worker/gpu/ and vllm/v1/core/: #58762 (cross-group metadata reuse) changed only those paths and
would not have triggered a step keyed on vllm/models/glm5next/ alone.

Known interactions: #55219 (draft) replaces KpoolTailSpec with a CircularBufferSpec tail, which
would change the spec names asserted in test_kv_cache_layout.

Why this is not duplicating an existing PR

Searched open PRs/issues for "GLM-5.3-Flash CI", "glm5next test", "pruned checkpoint", "GLM-5.3-Flash nixl"
(2026-09-18): no PR adds a reduced GLM-5.3-Flash checkpoint or e2e/PD/offload tests for it. Related but
different: #56974 (KDA grid.z fix; the kernel still launches grid.z = num_seqs * heads, but the TP1
startup at max_num_seqs=1024 no longer reaches it on current main, so test_default_max_num_seqs_tp1 is a
plain test now), #56810 (SimpleCPUOffload tail-group skip, the offload test's expected fix), #55219 (tail spec
rework), #59536 (the checkpoint-page regression the new test guards; merged 2026-10-01).

Test Plan

# 1 GPU (SM90+)
pytest -v -s tests/models/glm5next/test_pruned_e2e.py
pytest -v -s tests/v1/kv_offload/test_glm5next_pruned_e2e.py
pytest -v -s tests/v1/attention/test_gdn_metadata_builder.py -k update_block_table
# 2 GPUs
bash tests/v1/kv_connector/nixl_integration/run_glm5next_pd_test.sh

GLM5_PRUNED_MODEL / MODEL override the checkpoint path for local runs.

Test Result

GB300 x4, main 18553c5, enforce_eager:

run	result
`test_pruned_e2e.py` (TP1)	3 passed + 1 xfailed (strict) on main `18553c5`; 5 passed on main `b538d80` (2026-10-01, includes #59536). Checkpoint-page relative errors per layer (conv / recurrent): 4e-4 / 2e-3, 1e-2 / 9e-3, 2e-2 / 2e-2; with #59536's two files reverted the same test fails with 2.4 / 4.3, 1 / 1, 1 / 1
`test_gdn_metadata_builder.py -k update_block_table`	4 passed on main `b538d80`; the new equivalence test fails on the pre-#59536 tree at `checkpoint.state_indices`
`test_glm5next_pruned_e2e.py` on main	2 failed as designed (tail-group asserts above)
same, SimpleCPUOffloadConnector on the #56810 branch	passed (CPU prefix-cache hits > 0 after `reset_prefix_cache`, cold == warm tokens)
`run_glm5next_pd_test.sh` (1P1D, `--max-num-seqs 256`)	2 passed: D-side prefix hits cut NIXL transfer 26.8 MB -> 22.2 MB, outputs equal; P-only vs P->D first-4 max logprob delta 0.20 / 0.07 / 0.61 (short / mid / 6.5k-token) with P-only vs D-only 0.00 / 0.00 / 0.06

Notes from the local runs: all tests are text-only (limit_mm_per_prompt={"image":0,"video":0}) because
cold encoder profiling of the 24-block vision tower took ~25 min of cuDNN/CUTLASS JIT; the fp8 MoE was
forced to deep_gemm on this SM100 node because the FlashInfer TRT-LLM MoE module was re-JIT-compiled on
every engine start (not relevant on the H100/H200 CI targets).

AI assistance (Claude Code) was used for the analysis, the checkpoint builder and the test drafts; all
runs above were executed by the author.

🤖 Generated with Claude Code

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

JaredforReal · 2026-09-18T09:16:34Z

we shall wait for the bug fix PRs land in vllm main branch

github-actions · 2026-09-21T06:21:28Z

✅ @JaredforReal, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

mergify · 2026-09-21T06:28:33Z

Hi @JaredforReal, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

ivanium · 2026-09-21T23:50:23Z

Having more test cases is generally good to me.
Just wanted to make sure: do we still use the full model for e2e accuracy test?

mergify · 2026-09-30T06:54:31Z

Hi @JaredforReal, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Copilot

Copilot review overview

🟡 Changes recommended

Several assertions do not reliably exercise their intended regressions, and asynchronous offload completion is not handled safely.

Review effort: Balanced
Findings: 4

Open (4)

Validate group kinds and configured auto block size · New
Test CUDA graph capture with the promised strict xfail · New
Compare probabilities for identical selected tokens · New
Wait for pending offload stores before resetting the cache · New

What changed in this PR

Adds CI coverage for GLM-5.3-Flash using a layer-pruned checkpoint.

Changes:

Adds model, KV-offload, metadata, and NIXL PD regression tests.
Validates hybrid KV layout, checkpoint pages, MTP, and cache transfers.
Adds dedicated H100/H200 Buildkite jobs.

File	Description
`tests/models/glm5next/test_pruned_e2e.py`	Adds single-GPU model and KV-layout tests.
`tests/v1/attention/test_gdn_metadata_builder.py`	Compares reused metadata with fresh builds.
`tests/v1/kv_offload/test_glm5next_pruned_e2e.py`	Tests CPU KV-offload round trips.
`tests/v1/kv_connector/nixl_integration/test_hybrid_pd_consistency.py`	Checks PD logprob consistency.
`tests/v1/kv_connector/nixl_integration/run_glm5next_pd_test.sh`	Launches the two-GPU NIXL test.
`.buildkite/test_areas/models_language.yaml`	Adds pruned model E2E CI.
`.buildkite/test_areas/lm_eval.yaml`	Adds KV-offload CI.
`.buildkite/test_areas/disaggregated.yaml`	Adds NIXL PD CI.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    for _, layers, bs in groups:
+        if "KpoolTailSpec" not in layers:
+            assert bs in (block_size, 4)


+    """Startup and a generation at the default max_num_seqs of large GPUs
+    (1024 sequences x 64 local KDA heads once hit CUDA's gridDim.z limit in
+    the recurrent kernel's warmup, vllm-project/vllm#56973)."""
+    llm = _make_llm(max_num_seqs=1024)


+        delta_pd_single = max(abs(a - b) for a, b in zip(lp_p, lp_d))
+        delta_proxy = max(abs(a - b) for a, b in zip(lp_p[:4], lp_x[:4]))


+    sp = SamplingParams(temperature=0, max_tokens=8)
+
+    cold = llm.generate([prompt], sp, use_tqdm=False)[0].outputs[0].token_ids
+    assert llm.reset_prefix_cache()


ZJY0516 · 2026-10-05T09:31:56Z

Having more test cases is generally good to me. Just wanted to make sure: do we still use the full model for e2e accuracy test?

yes, we have e2e accuracy test in https://github.com/vllm-project/perf-eval

mergify · 2026-10-20T22:58:36Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JaredforReal.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Use JaredforReal/GLM-5.3-Flash-4L (original layers 0-3 + MTP, all widths unchanged) to exercise the GLM-5.3-Flash hybrid KV cache layout (KDA mamba groups aliasing the MLA slot, kpool indexer + tail scratch group), fp8 MoE, mHC and MTP on a single SM90+ GPU in seconds instead of a 4x node. - tests/models/glm5next/test_pruned_e2e.py: asserts the KV group / tensor aliasing layout, greedy reproducibility across a block boundary with a prompt longer than index_topk, MTP drafting, and a strict xfail for the fused_recurrent_kda grid.z overflow at 64 heads x 1024 seqs. - tests/v1/kv_offload/test_glm5next_pruned_e2e.py: CPU offload round trip for SimpleCPUOffloadConnector / OffloadingConnector; fails on main because the kpool tail group (block 4, non prefix-cacheable) is not skipped. - nixl_integration/run_glm5next_pd_test.sh + test_hybrid_pd_consistency.py: 1P1D NixlConnector with the DS conv-state layout, reusing the Mamba prefix-cache test and adding a logprob-tolerance P vs P->D check (hybrid PD recomputes the last prompt token on D, so exact match is not expected). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

Single-GPU e2e and KV-offload jobs on h200_35gb, 1P1D NixlConnector job on 2x h100. The DeepGEMM paged-MQA indexer and FlashMLA-sparse need SM90+, so these cannot join the existing L4 PD sweeps. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

…p's page The pruned-model e2e now prefills a prompt that crosses a block boundary and, for each MambaSpec group, compares the cached checkpoint page with the boundary state produced by a cold prefill ending exactly there (retired block under dense retention). With vllm-project#58762's cross-group metadata reuse and vllm-project#56960's checkpoints, groups 2..N used to write their checkpoint into group 1's page (vllm-project#59536); the existing greedy-repeat test passes on that bug, this one fails with a relative error of 1 or more on every page. Widen the Pruned E2E step's source dependencies to the GDN backend, the Mamba layers, the V2 worker and v1/core, which is where that regression came from, and type the connector extra-config dict for mypy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

Compare every field of the metadata an update_block_table() produces for a sibling group with a fresh build() on that group's block table, so a new per-group field cannot stay pointed at the source group; the existing tests only checked the fields their authors had in mind. Fails before vllm-project#59536 on checkpoint.state_indices and passes after it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

…lash incoai/GLM-5.3-Flash-DFlash2 taps five target boundaries; on the 4-layer checkpoint a callable hf_overrides, composed into the draft config, points them at the embeddings and layers 0-3 so the drafter's 5 x hidden fc input is met. The test checks what vllm-project#56983 fixed: the drafter loads against the hybrid target (aux capture through mHC, no expert parallelism copied onto the dense drafter), its five GQA layers get KV cache pages next to the MLA, indexer, tail and KDA groups, drafts are verified, and greedy output agrees with the target-only run on the leading tokens. Acceptance is not asserted: the drafter was trained for the 45-layer model. GLM5_DFLASH2_DRAFT overrides the drafter path for offline runs, like GLM5_PRUNED_MODEL. The Pruned E2E step also depends on the DFlash model files. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

- test_kv_cache_layout compared the spec name against layer names, so the block-size check never ran; every non-tail group must now use the auto block size and the tail group index_kpool. - The PD consistency test compared each run's selected-token logprob without checking the token; it now requires the first four tokens to match before comparing their logprobs. - The KV offload round trip asserted reset_prefix_cache() right after generate(); offload stores finish asynchronously, so step the engine until the reset succeeds (bounded), as the connector unit tests do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

JaredforReal · 2026-10-27T05:52:59Z

/ci run

github-actions · 2026-10-27T05:53:25Z

✅ Triggered Buildkite CI #97524 for commit 69a0b9544a09.

ℹ️ This PR is 10 commits behind upstream main. Merged upstream main into this branch as 69a0b9544a09 before starting CI. Pull the branch before pushing more changes.

…ry jobs The GLM5Next and V1 KV Offload jobs already collect tests/models/glm5next and tests/v1/kv_offload, so the two dedicated steps ran every test twice. Drop them and widen the GLM5Next job's dependencies and timeout instead. Track the kpool tail rename to CircularBufferSpec (vllm-project#57169), and shut down each test's engines afterwards: a failed test's traceback kept its LLM alive, so the following tests could not get their GPU memory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

Since vllm-project#57169 every KV cache group lives in one packed buffer: all layers share the block stride, the KDA state fills the MLA latent's slot behind the indexer keys, and the indexer keys and the tail ring both start at the block. The old check compared per-tensor offsets, which no longer holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

mergify · 2026-10-28T23:35:22Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JaredforReal.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify Bot added ci/build glm kv-connector labels Sep 18, 2026

JaredforReal marked this pull request as ready for review September 18, 2026 09:08

JaredforReal requested review from AndreasKaratzas, ApostaC, DarkLight1337, Harry-Chen, ivanium, khluu, orozery and ywang96 as code owners September 18, 2026 09:08

claude Bot reviewed Sep 18, 2026

View reviewed changes

ZJY0516 added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 21, 2026

Copilot AI balanced review requested due to automatic review settings October 1, 2026 17:31

JaredforReal requested a review from wzhao18 as a code owner October 1, 2026 17:31

Copilot started reviewing on behalf of JaredforReal October 1, 2026 17:37

View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

JaredforReal mentioned this pull request Oct 10, 2026

[Bugfix] Fuse GLM-5.3 KDA q|k|v convs into one conv1d parameter #60991

Merged

6 tasks done

mergify Bot added the needs-rebase label Oct 20, 2026

Loken-Lau mentioned this pull request Oct 24, 2026

[Bugfix][GLM] Preserve kpool history at unaligned prefill boundaries #63471

Open

6 tasks done

JaredforReal force-pushed the ci/glm53flash-4l-tests branch from 91f9b4a to 6e94a50 Compare October 26, 2026 09:14

JaredforReal and others added 6 commits October 26, 2026 09:15

[Test] Point the KDA grid.z xfail at the upstream issue and fix

7018c3f

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jared Wen <w13431838023@gmail.com>

JaredforReal force-pushed the ci/glm53flash-4l-tests branch from 6e94a50 to 80365ba Compare October 26, 2026 09:15

JaredforReal requested a review from hmellor as a code owner October 26, 2026 09:15

mergify Bot removed the needs-rebase label Oct 26, 2026

Merge branch 'main' into ci/glm53flash-4l-tests

546df14

Merge branch 'main' into ci/glm53flash-4l-tests

69a0b95

hmellor removed their request for review October 27, 2026 10:35

JaredforReal requested a review from a team October 27, 2026 15:47

JaredforReal mentioned this pull request Oct 27, 2026

[Bugfix][Attention] Pin the host tensors the SM90 sparse MLA plan copies JaredforReal/vllm#37

Open

6 tasks done

mergify Bot added needs-rebase nixl NIXL KV connector / multi-tiered KV offloading (KVCR) labels Oct 28, 2026

Navigation Menu

Repository navigation

[CI][Test] GLM-5.3-Flash layer-pruned checkpoint tests (e2e, KV offload, NIXL PD) - #57537

[CI][Test] GLM-5.3-Flash layer-pruned checkpoint tests (e2e, KV offload, NIXL PD)#57537
JaredforReal wants to merge 11 commits into
vllm-project:mainfrom
JaredforReal:ci/glm53flash-4l-tests

JaredforReal commented Sep 18, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

JaredforReal commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 21, 2026

Uh oh!

mergify Bot commented Sep 21, 2026

Uh oh!

ivanium commented Sep 21, 2026

Uh oh!

mergify Bot commented Sep 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ZJY0516 commented Oct 5, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Oct 20, 2026

Uh oh!

JaredforReal commented Oct 27, 2026

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

mergify Bot commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		delta_pd_single = max(abs(a - b) for a, b in zip(lp_p, lp_d))
		delta_proxy = max(abs(a - b) for a, b in zip(lp_p[:4], lp_x[:4]))

Conversation

JaredforReal commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Why this is not duplicating an existing PR

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

JaredforReal commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 21, 2026

Uh oh!

mergify Bot commented Sep 21, 2026

Uh oh!

ivanium commented Sep 21, 2026

Uh oh!

mergify Bot commented Sep 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

ZJY0516 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Oct 20, 2026

Uh oh!

JaredforReal commented Oct 27, 2026

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

mergify Bot commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JaredforReal commented Sep 18, 2026 •

edited

Loading

ZJY0516 commented Oct 5, 2026 •

edited

Loading