integration(glm53-flash): complete NVFP4 and prefix-cache stack - #589
Conversation
Register the model-specific nvfp4_ds_mla dtype, retain its uint8 storage, and allow only that packed dtype through the generic MLA NVFP4 guard. Select the 304-byte GLM_NEXT record in the B12X cache spec and pass the explicit no-RoPE, inline-per-token-scale recipe to decode and prefill. Size hybrid attention/Mamba pages from the backend-published 337-byte token row so the smaller packed record receives a compatible manager block. Align the semantic record region to the embedded C4 index-page boundary, and derive the pooled-index tail from the bound runtime record width instead of hard-coding FP8 geometry. Treat nvfp4_ds_mla as an opaque packed-byte MLA cache throughout forward and DCP workspace setup. This preserves its uint8 record instead of reinterpreting it as native FP8, while leaving ordinary FP8 and fp8_ds_mla behavior unchanged. Non-GLM_NEXT sparse MLA configurations still reject the new dtype. Depends on the matching B12X GLM_NEXT NVFP4 record implementation. Co-authored-by: David Young <davidseanyoung@gmail.com>
Rebuild the MLA cache spec at the resolved manager block size before padding the Mamba page. This includes the backend-aligned pooled-index tail and prevents a 2 KiB underallocation at the 3328-token GLM5Next block size. Signed-off-by: derek <derek.yates@live.com>
…glm53-flash-nvfp4-kv # Conflicts: # tests/v1/attention/test_b12x_sparse_mla_api.py
Initialize the full-CKV and record-width fields expected by the merged target branch so the NVFP4 and finalized-geometry bind tests exercise their intended contracts. Signed-off-by: derek <derek.yates@live.com>
Size the DCP full-CKV workspaces and raw-record gather from the selected GLM5Next cache format. This preserves the qualified 528-byte FP8 path while allowing the 304-byte NVFP4 layout through DCP4 full-context prefill. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: derek <derek.yates@live.com>
Select packed sparse MLA workspaces from the layer group cache spec so per-layer auto overrides do not inherit an unrelated global packed dtype. Add the focused resolved-spec regression and complete the new helper docstrings requested in review. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: derek <derek.yates@live.com>
The generic DSA backend consumes a layer-compact token-major cache view, while GLM5Next requires a block-outermost view for its pooled index tail. Declare those contracts independently so layout resolution cannot select a shape that produces invalid attention output. Assisted-by: OpenAI Codex Signed-off-by: derek <derek.yates@live.com>
Give the shared GLM-5.2/5.3 DSA architecture an explicit B12X backend contract for its 368-byte NVFP4-latent plus FP8-RoPE cache record. Keep GLM5Next on its independent 304-byte no-RoPE ABI and reject unrelated 576-wide sparse MLA models. Use the B12X 1.3 public cache writer and explicit GLM_NSA run recipe without environment-controlled layout selection. Assisted-by: OpenAI Codex Signed-off-by: derek <derek.yates@live.com>
Signed-off-by: derek <derek.yates@live.com> Assisted-by: OpenAI Codex
…pare/jj-glm-nvfp4 Signed-off-by: derek <derek.yates@live.com> # Conflicts: # tests/v1/attention/test_b12x_sparse_mla_api.py # vllm/v1/attention/backends/mla/b12x_mla_sparse.py
Assisted-by: OpenAI Codex Signed-off-by: derek <derek.yates@live.com>
Pass the resolved cache recipe into B12X caps, bind the concrete KV cache against it, select the FP8 or NVFP4 writer at construction, and stop resending recipe policy on every decode or prefill call. Assisted-by: OpenAI Codex Signed-off-by: derek <derek.yates@live.com>
…lm53-nvfp4-auto-geometry-review
…3-nvfp4-auto-geometry-review # Conflicts: # tests/v1/attention/test_b12x_sparse_mla_api.py
…uncate the load boundary
update_state_after_alloc scans the allocated block list for the first block that
is not null and has no hash, and treats that index as the start of the freshly
allocated region. That index drives keys_to_load (via start_chunk_idx),
dst_block_ids, group_sizes and block_indices, so getting it wrong loads the
wrong keys into the wrong destination blocks.
The scan started at index 0, inside the region the scheduler had already
reported as locally computed. Blocks there are computed by definition, so
scanning them can only do harm -- and for a recurrent group it does. A Mamba /
GDN group has no per-token KV at all: it carries a fixed-size state, which is
why get_sliding_window_size_in_chunks() returns 1 for MambaSpec ("Mamba depends
on a single state"). Most positions in its block table point at the shared null
sentinel, and the one real block holding the state is legitimately not
full-and-cached. The scan took that as the start of the fresh region, collapsed
the boundary below the computed mark, and the following assert fired.
Start the scan at the locally computed boundary instead. For a full-attention
group this is a no-op: every token has KV, every block below the boundary is
full-and-cached, so the old scan skipped them anyway. The existing assert now
holds by construction (nlcgb >= cdiv(nlct, tpb) implies nlct <= nlcgb * tpb) and
is kept as a guard for other causes.
Reproduced on a 4-node GB200 deployment (Qwen3.5-family hybrid GDN+GQA, 4 KV
cache groups, 3 recurrent, MTP): under sustained partial-hit load the leader
crashed after 47,142 requests, exit=1 with this assert while the other three
ranks exited 137 on gang teardown. Partial hits immediately preceding: 12768,
6384, 8512, 6384, 4256 offloaded tokens.
Reachable only when num_external_tokens > 0 AND num_locally_computed_tokens > 0,
i.e. a partial hit. Tests that flood to full eviction before re-sending produce
num_computed_tokens == 0, where the assert reads 0 <= X and cannot fire; the
same deployment ran clean and idle for 3h47m immediately before the load.
Independent of the drafter-annotation defect in vllm-project#52735/vllm-project#52771: the crash above
was observed with vllm-project#52771 applied, no group marked as a drafter, and the
volatile-tail pop therefore never running.
Signed-off-by: Yifan Jiang <yifanjiang1997@gmail.com>
… under spec decode Two deterministic defects on the spec-decode + hybrid-mamba + KV-connector path: 1. Offloading connector marked every KV group as eagle under DFlash/DSpark. `SchedulerOffloadConfig` fell back to `use_eagle()` when no group carried `is_eagle_group`, flagging all groups (including the target's mamba group) as volatile and excluding the trailing chunk of every decode step from offloading, so the store/lookup windows drift. Use the precise `use_eagle_preserves_target_kv_cache()` capability bit (eagle/eagle3/mtp only) for the fallback. 2. `SlidingWindowManager.find_longest_cache_hit()` assert-crashed the engine when the coordinator alignment was finer than the sliding-window group's block size (`alignment_tokens % block_size != 0`, e.g. `prefix_match_unit` finer than the draft model's SWA block). Fall back to block-aligned hits for that group instead of asserting. 3. Regression test for the SWA group fallback. Tests: - pytest tests/v1/core/test_mamba_align_chunk_split.py -q - pytest tests/v1/core/test_scheduler.py -q -k mamba_align Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: ouzq <ouzq@seu.edu.cn>
…che block SpeculativeConfig.use_eagle() is a stand-in for "spec decode that reads target hidden states" and returns True for dflash/dspark too. The scheduler used it in _mamba_block_aligned_split() to back last_cache_position off by one mamba block, but only eagle-family drafters (eagle/eagle3/mtp) pollute the target's last matching full-attention block with the lookahead KV write; DFlash/DSpark draft via block diffusion from their own KV cache and never write target blocks. The spurious back-off made every prompt shorter than two mamba blocks skip the final block-aligned chunk, so the mamba recurrent state never materialized on a block boundary and the next turn's prefix-cache lookup converged to 0 -> the whole context was recomputed on every reply. Add SpeculativeConfig.use_eagle_preserves_target_kv_cache() (eagle/eagle3/ mtp) and drive the scheduler's last-block drop and the KV cache manager's eagle handling with it. use_eagle() keeps its existing semantics (encoder shift, lookahead budget, num_prefill_lookahead); only the prefix-cache last-block drop is scoped precisely. Tests: - pytest tests/v1/core/test_mamba_align_chunk_split.py -q - pytest tests/v1/core/test_scheduler.py -q -k mamba_align Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: ouzq <ouzq@seu.edu.cn>
…LE spec decode Fixes vllm-project#52735. Three defects in the offloading connector scheduler combined to make the CPU offload tier store KV but never serve it when speculative decoding is enabled: 1. When an eagle-family speculative method is active but no KV-cache group is annotated as a drafter group — the normal situation for shared-group MTP models like Qwen3.5/3.8, whose drafter is a regular full-attention decoder layer merged into a target group — a fallback marked EVERY group as a drafter group. Drafter KV served stale can only lower speculative acceptance (the target verifies every draft), so fail toward serving: mark nothing and log once. 2. The volatile-trailing-chunk pop in _lookup_complete_chunks widened its query only for sliding-window groups. For full-attention drafter groups the pop shrank the confirmed boundary below what was queried, which can drop it under a coarser sibling group's chunk granularity (e.g. a mamba-align group) and zero the whole request's hit. Widen the query for every drafter group so the pop lands on the extra queried chunk. 3. The store-side trailing-chunk exclusion for drafter groups persisted after the request finished, permanently losing the final chunk of every drafter group per request. Once finished, no spec-token rejection can rewrite the tail, so the exclusion is lifted at finish. The lookup-side pop still prevents the (possibly volatile) final stored chunk from ever being served directly. Unit repro: with a full-attention + mamba-align hybrid and a manager that only hits stored keys, a re-sent prompt served 0 tokens from the offload tier with MTP on vs 16 with MTP off; all three regression tests fail on main and pass with this change. Full offloading connector suite passes. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Kam Basra <kameldipbasra@gmail.com>
…o-geometry-review
|
Warning Review limit reachedNext included review available in 37 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (21)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Scoped review of the cache-semantics part of The DFlash no-drop changes what the LP15 profile reuses; please back it with a live equivalence run
Two things the code alone does not settle:
One test break to fix before merge
This review was written by Claude Fable 5.1 (Anthropic) and posted by me. |
Integration qualification and merge statusStatus: the exact PR head is qualified as an ancestor of the R20 composed
Qualification covered:
GitHub reports that this PR conflicts with The performance values and immutable image identity are recorded in #590. |
91e4884
into
local-inference-lab:dev/jovian-judgement
6 participants
Purpose
Provide one conflict-resolved Jovian Judgement integration head for the
GLM-5.3-Flash cache-complete r16 build. This is the release-integration view of
the component work, not a replacement for its focused review units.
Included work
nvfp4_ds_mlarouting, packed cache ABI, page sizing, and DCPrecord handling for GLM-5.3-Flash.
f63961c105: do not truncate at an unhashed recurrent block;7ae4b8987c: preserve safe hybrid/SWA cache-hit alignment;a2e92b9482: retain DFlash/DSpark's final eligible prefix block;728b32ca2f: preserve shared-group MTP/EAGLE offload hits.The overlapping #549/#575 test changes are resolved in this branch. During
composition, a dropped closing assertion delimiter in the replayed native
prefix test was also restored before publication.
Why this is not duplicate work
#549, #575, and #574 remain the focused review units. Their independently
mergeable heads overlap in
test_b12x_sparse_mla_api.py, and the four nativeprefix fixes are also required by the qualified image. This PR provides the
deterministic, buildable aggregate that an image-integration bot can consume
without inventing a conflict resolution or omitting an unpublished r15-only
equivalent.
Dependencies and preflight
c0f7f6ccd7ff664df1192ae9f68e1d3bf244ecc8.0b8eedb6c9c268c62234eff54229f239123678f0; the focused local review delta isLMCache models: load serialized MXFP8 MTP modules #35 at
2579f617a27382e6210632a6698f314986ad3419.on-demand allocator experiments.
particular, verify the equivalents of vLLM fix(dflash): replicate draft KV under DCP #513 and fix(dcp): handle empty ranks and gather sparse MLA queries #560 if those modes are
exposed by the image.
Validation
git diff --checkpass.CUDA cuMem interposer, and upstream allocator policy passed.
with both
fp8_ds_mlaandnvfp4_ds_mla.was unavailable.
AI assistance
OpenAI Codex assisted with source audit, conflict resolution, regression
construction, and integration validation. The submitter reviewed the resulting
source and evidence.